I get very self-conscious when I'm talking to a mic kind of brings out the inner singer that I have but yeah I'm not going to bore you with that. The idea is that at the end of the presentation at the end of what we have today at the end of two hours you will We will be sending out or we will be releasing all the content that we have Including the virtual machines, the gitbook, the scripts, the code all of that We're going to make it public and you folks can take it down and Try it out in your own personal systems because some of these require internet connectivity Some of them require some amount of processing and some of the examples we are going to cover and the way the So we're going to call it like a talk slash workshop now with the way this is structured Some of it you folks can practice on your systems And if you have internet access you can there are a couple of sites We are going to show you which which allow you to pull information and other things as part of OSINT Right you can definitely try that as and when you see if you have access to internet here you could go ahead and try that out Okay Now the primary focus the primary intent of the workshop is to ensure that Everybody in the room is familiar with OSINT right that's the whole idea you all you heard you have some basic amount of Knowledge and coverage of the idea of OSINT right what we're going to try and show in the next Hour and a half two hours is how do you take the amount of data that is out there in terms of OSINT and Specifically when you're focusing on targets that you want to you know You're trying to attack in a building or attack chain your attack toolkit How do you take that data and you visualize it? Right and the visualization has multiple you know multiple pros the idea of visualization enables you to see what is Available at the moment you can compare it with something else that you're trying to figure out right a simple example would be sub-domains for example Right, how many if you've done bug bounty here in the room before bug bounty hunters Okay, anybody who's who's from a penetration testing background? All right, so the idea is that when you're trying to attack I mean trying to attack a target you would want to find out as much information about the target as possible Right the visual cue that I would like to paint for attendees normally is the idea of you trying to get into a house Okay, now before you enter the house you don't try and normally try and break into the house directly You would scout around the house you try to figure out when the occupants of the house are available Not available when they're out how many windows are there are the fences around I do they have dogs are their windows unlocked all of this Information right analogous to what we do in a penetration testing exercise We will be trying to enhance the amount of data that we have with open source information that we gather Right the idea of today's workshop is going to align with that and we are going to cover three scenarios where this is useful Right and the scenarios will will be in a way where you would be able to use data That is available to you that you've discovered online and mother's going to cover most of the automation and the Visualization bit right and how do you visualize this so that it is presentable to some with either within your team or to your management If you're going to go coming at it from a defense point of view Right like go back to your senior management folk and show them that hey, this is what is visible you like fancy graphs Here you go. These are pie charts that you can maybe talk to your other board of directors and see what shit is open on the internet Right, so that's what you're gonna do So Essentially, we're going to cover three scenarios today. Okay, the first scenario is What we would like to The way would like to put it if the first scenario allows us to detect in near real-time The first scenario allows to detect in near real-time if there are SSL certificates for your sub domains being released Hey, we are something called as a trans a certificate transparency log. How many of you are familiar with certificate transparency logs, right? We'll talk about that briefly right and it's really interesting because I'm gonna show you I'm going to show you a couple of demos in the form of web applications on the internet website on the internet that you Can go back and if you have access to the internet try it out on your Organization domains right and it's completely passive and then you will be surprised the amount of information that it leaks Okay, we're going to use in the first scenario a certificate transparency logs to figure out if any Certificates have been issued for sub domains Okay, that you're trying to target for a particular company the second scenario uses a Forward DNS set. It's an open data set by rapid seven We're going to use that to try and visualize what is the infrastructure for a particular target organization They're going to be attacking right all the DNS information that the organization has Available in the form of visual cues Okay, and the third scenario is going to be if you can detect or rather in a way How would you go about detecting and visualizing if somebody's created rogue domain names that looks similar and You know appear similar Verbally or contextually in in enunciation as well to your organization like you've seen those Fishing campaigns right they you fishing campaigns and scammers would want to be able to visualize the target domain as nearly as The original domain and it kind of adds to the whole value of the scamming operation You would want to have suppose somebody was trying to scam Gmail right and gmail users if you could register a domain with gm with two a's and I L that would be a far more right Viable opportunity for scammers to get hook into the people who they're trying to fish So that's a third scenario. We're going to again visualize that data Again as I said at the beginning we are going to give away all the entire documentation along with a virtual machine that we're going to release Right at the end of this And we're going to raise you know get on github and we're going to tweet about it using recon village or even our absolute account Right, so just follow us whenever the data is available. You should be able to clone that So yeah, I've been asked to put a add the disclaimer but that please Mostly reconnaissance is considered is almost always passive right the idea of recon is trying to be as passive as possible But sometimes you do touch the remote target And there are cases test cases that you'd run when finding information that you would sometimes touch the target And I would recommend I've seen this in the real world With the several years of pen testing experience that I have and being a part of the company apseco I've worked at Citrix before and other companies before that The idea is that ensure that you have permission Your local country laws can easily screw you up if somebody wants to get back to you they will get back to you Right, so the code and the other things that we're going to release in As part of this class. They're not production ready, right? But the you can go back modify them and use them see use it for your test cases and see if they become viable to you All right Yeah I'll let Madhu introduce the primary visualization platform that you're going to use the Elk stack how many of you familiar with elastic search Okay, oh good. He's going to introduce that to you and then we can move on to Yeah, so before that our primary trainer our primary speaker for this class Bharat He ran into visa issues like most people who couldn't make it to Defconn Blackhead this year So he says hi to everyone right back from India He's he's an active member of my team. I had the offensive security team at Apsaco He's an active member of the team and yeah, he's our primary go-to recon guy, right and his idea of Obtaining information and we've done tons of I'll probably give you a couple examples during the talk where we've used Very minuscule amount of information available to us to completely pawn an entire organization We have couple of test cases like that brilliant stories, right? So I'm going to talk about them as well, but yeah He's core interest lies in infrasack application security and a lot of reconnaissance work that he does and he's done a couple of talks as well So he just wanted us to bring this up. This is Madhu my colleague at Apsaco again He's another part of he's a member of my team, right and I see that my name isn't there. So I'm just going to say hi So Ria's here, right? So I'm going to be pitching in whenever the opportunity occurs Thank you so as Ria's introduced most of the stuff we will be trying to focus on recon and The primary focus of the workshop will be mainly data sets That is the one of the reason we choose kind of ELK stack I think most of you are already familiar with the ELK stack as you said so the ELK is primarily Allows three things as ELK elastic search lockstash Kibana So each piece has individual pleasant advantages and disadvantages Suppose if your organization have really good data sets like which they have normalized and put it in JSON Then you don't wanted to use a lockstash, right lockstash is kind of normalizing and make sure everything data Which you are getting and send it to a elastic search or S3 buckets or whichever the data source output you wanted to send and Coming to elastic search Elastic search is kind of large no SQL kind of data store Where you kind of store all the data which is coming from the lockstash or like from other data sources Which will store an index using elastic search and the final Output whatever we wanted to showcase is we will be using Kibana So Kibana is kind of front end using I think written in angular jays Which is fancy graphs and the visualization to showcase the data This will make quarry to elastic search and get the data to represent in a visual format And just to give again one more time So we will be going through multiple demonstrations rather than giving hands-on because we couldn't able to make sure Yeah, I'll give actualization software now and troubleshoot But we'll be releasing this VM whatever we are trying to showcase now and you can go back and try using the steps You can follow our like P towards like if you have any questions Yeah So I'll just quickly highly Showcase like how it looks like in the configuration because this is one of the primary ingredient to set up entire the visualization platform So if you see in terms of especially lockstash part So it has three types of formats one is input filter and output So input is kind of a plug-ins a very it supports. I think nearly 50 plus amount of leggings It can be taken data from system your system on servers like system monitoring as well as any S3 pockets are directly file inputs So in the scenarios, which we are going to cover further three will be taking currently as a file output But it can be taken from anywhere else, right? And the similarly filter is one of the key piece why people try to use lockstash For example, if you have a data set which contains IP addresses, right? Which you wanted to enrich data using suppose longitude latitude which country where is the region? So filters kind of allows you to do that. I do just say the GUIP database like my Max mind What it does is you say that this is the field I wanted to enrich I will go ahead and enrich that data and give you more granular results and The output is kind of decision-maker Whether you wanted to send it to elastic search if you prefer or your team has really good amount of big data stuff You can go ahead and send it to Apaches Park or like Hadoop clusters and it supports I think nearly 60 plus plugins output so that you can go ahead and take actions as well here as well, right? And this is again one of the piece which is like hanging around especially if ELK stack around So this is kind of a data shippers like if you are familiar with the network security Like people use it to send the network switches data like logs to a centralized system monitoring server, right? So beats is kind of a go lang binary which sits in wherever you wanted to send the data from sources and send it to a Elastic search or log stack, right? And it's very lightweight the good part about this thing They kind of integrated with Kafka like if you don't want to lose not even a single log It will go in and send it to a Kafka streaming so you can push and pull you like methodology stuff and The people if you are most interested in like a packet level understanding like especially in your agnations They have released a packet bit as well like it kind of look at DNS queries and network monitoring as well each packet level And We are just gonna skip this section like not going very deep into ELK stack If you wanted to go back and refer like more interested in area So we kind of made a quick get book to deep dive into ELK stack and again, it's in the reference section I think my internet is super slow so I just Update this anyway, whenever we update the URL is here at the end apricot.com slash book slash ELK workshop So if you wanted deep dive and use that stuff so The second phase is alerting While we are storing huge amount of data sets The key pieces is we can't keep looking at the data because it's a human job to make mistakes like you may not miss Like a look at everything you may miss out something so alerting is kind of enables like I rather than keep looking at Whatever you look interesting you can go ahead and send it to some other place based on certain a lot queries How many of you slack here? Who doesn't right? so slack is kind of Alloting we will be using here is alastel again. It's as title said we will be using most of the open source stuff Alastel it is an open source project by Alp. So where it Completely work with the LK stack to query the data using rest API and send back the data based on the queries Which you made to the LK stack alerting system, right? And the very important features it support Then you can see fine the text cool a little bigger Yeah, I think I felt the same It'll better so if you see this This is a very simple rule it looks like it is we will be using in the coming section in the scenario What it does is it will keep looking for the locks and if it matches any of your organization Certificate which is generated a TLS certificate using any let's encrypt or like your proprietary Certificate authorities will go ahead and send an alert, right? It's kind of keep monitoring for our new certificate updates and to track up inventory of certificates as well So this is a simple filter what we are doing is we are looking for the index to receive the logs and if it matches in a number of events based on the alert to go ahead and look for the Query which we will be showing in coming section and it will send a slack alert, right? Similarly, it can be many other but it's kind of more integrated with elastic search So we have chosen and if you have like a really good fancy other stuff It is feel free to use because the open source we have chosen this people might be using splank oxide another base They can go and use as well same things And it has different types of rule set supports like if you wanted to configure based on the rule sets and other types So it can send you to email SNS Jira to gates to rise into security team and talk to them developers So I'll go ahead and Move into hands-on section if you have any questions or if you see anything like missing a lot some stuff Please go ahead and ask right up a font fine back Looks fine So we have set it up a LP stack already in this virtual mission Pre-configured using an ansible playbook which we will be again use giving that as well Like if you wanted to go back and set it up. So I'll quickly show you the configuration So this is a elastic search configuration. So what we are trying to say here is we have created a new cluster saying that elastic stack and We created a one note because for the workshop. We just using one mission if you are storing like it Okay Lucky one I'll just zoom in So the cluster name if you wanted to run huge data sets It may be better to use clustering mode and with different nodes and This is like a very basic configuration And the one thing we are trying to put is most of you might find in shodan searches If you shodan like elastic search doesn't have by default authentication if you put 0.0.0 here Anyone can access your cluster out there. Basically your data is out on the internet so we can have hardened that part and by default it listens on 9 to 004 and That's it. So this is very high-level basic configuration to work with elastic stack and You can just make curl request to local host 9 to 00 you can see that the cluster is up and running and We can able to send and receive the data from the cluster API using that 9 to 004, right? And once we have suited up the cluster So we have a place and data store like a to receive or send it back So now we have to get the data, right? So getting the data part We will be using lock stash as I said like I'll quickly show the lock stash Configuration as well so that you will able to get idea of how the locks are coming in place and how it will exchange Any questions like till elastic search part because am I going too fast like because people will give me feedback saying that I go too fast So if I am like too fast, please let me know I'll make it slow down and Let it be clear So in this phase if you see Just make it so this is an inputs configuration So as I said lock stash has three phases one is input configuration and output and Once you have understand these basic stuff I think you can go ahead and not only the the data which we are showcasing you can explore any data It's at the end like a passing that data and making sure that getting some sense out of the data, right? Visualizations so here what we are doing is we are listening on port five zero double four for the lock stash So anyone can able to reach five zero double four of this much virtual machine Can able to send data to the beats sorry the lock stash So if you wanted to use SSL search like for this to make sure this is over TLS You can able to configure that and it is over TLS So once we receive we kind of apply filters So for this we are using two different scenarios The long sets one is our stream as Ria has mentioned. So which is kind of Mostly data coming from the certificate transparency locks. We are currently picking up from CRT. SH I'll just quickly show you how the locks looks like so whenever it matches saying that search stream in the tags Like we tag the data whenever it is sending it to lock stash It will go ahead and perform these filters currently There is no filter because we kind of directly getting Jason and if you have different kind of data sets This filters allows us to parse the data so you can go back and check the lock stash plug-ins of filter So it has huge amount of data Piltrations So if you look at for the same thing like scan cyber data the filters We will be using geo IP filter because scan cyber data has IP addresses coming or like a domain names So what we are trying to do is rather than going and passing directly IP addresses and domain names We are resolving them and we are trying to capture their longitude latitude as well as the domain Information as well and they can sending it back to elastic search so that we can able to make some data sense, right? And once everything is done So we kind of sending everything currently to elastic search Which is again currently running in localhost 9 to double zero So once the data is receiving and perform the filtration will won't stays in the elastic search So if you see the index of the index is automatically created based on every day So you will create different indexes every day to search back. So that is about mostly cluster configuration We'll just go back and look at the data sets so that we can able to see how it's going to be parsing into the Elgast app So this is the one which we are talking about the previously the CR data SH is the one which we are taking the locks at the certificate transparency locks We will be making query using the postgres. They have a postgres interface And if you wanted to run like huge amount of queries, they kind of block that you can request a different credentials They kind of give you like postgres guest admin credentials So you can have go and query and perform some research on top of like a certificate transference logs I Think I'll just do a quick check. I'll switch off my monitor to make sure my internet is working Yeah, it's working So yeah, so I'll just go ahead and make a simple quarry To showcase how the data looks like so that we will see how we can go ahead and pass the data If you can't see please let me know I try to make sure like I'm increasing the font always So if I miss out like please let me know So currently it supports different kind of queries if you use their CLI especially the postgres the interface It's kind of you sequel like interface to query the database But in web you kind of use like percentage to say that this is a wild card and regular expression type So I'll go ahead and give one of the name like That's not okay Like I wanted to give someone anyone wanted to participate and give one of the domain Dot com So percentage here is like a wild card. So it's kind of gets all the data And again, I think looks like my internet is super slow. Yeah So if you see in the identity field So this log that whenever it creates a certificate and as well as the background has like a subdomain For this like as I have created a certificate called a dot background. And if you say that blog.buckrow.com Which is again generated by DG cert like you can see the who is the certificate authority as well So you can go ahead and use these kind of data to find subdomains as well, right as a bug bounty hunter They want like mainly look at like if we have given a domain What are the subdomain so that I can find a more issues in terms of a possible attack surface, right? If you in terms of pen tester point of view, it will apply similarly Like if you have given a target, it may not be the same way to go, right? You might find other ways to let it into the target So this is one of the good place to look at because most of the people don't wanted to make their servers as Applications run in a HTTP. So they kind of generate a TLS certificate As per the regulations or standards, they have to be available the certification transparency logs So if you don't want it to be here as a defense point of view, so you have to run your pka public information for yourself Yeah, so I'll be showing the output the same output in a JSON format how it looks like So this is how it looks like but I'll try to make it like it looks simpler Just So I kind of ran jq the jq is like JSON passing stuff tool Yeah, if you see from one certificate basically it starts from the data and it has like the certificate link The information about extensions authority and each and every information related to the certificate Which is generated from the transparency log. So this is the log We wanted to pass it to the other elk stack. So we have already pretty good json So we kind of directly use that to send it to elastic search and lobsters So if you have seen like some other datasets like not for this scenario Maybe you can have use some other filters to make it properly parsed into So you cannot see which domains they have updated the certificates as well like Let's encrypt recently started supporting wildcard domains So people started using rather than disclosing their subdomains, right? If you're rather than going on making a HTTP a certificate for block.backload.com They kind of put starter.backload.com, but it has again its own negatives or disadvantages as well So this kind of disclosures some of the information which kind of gives the visibility for the attackers point of view as well So we are passing these logs again So as I said in the beginning We will be using beats beats is kind of data shipper like a which will help you to pass send those logs from one location to the elk stack So if you see here the configuration Yeah, so we kind of have a different kind of inputs We are tagged here saying that search stream so that the log will whenever it reads He'll go and make sure if it matches at stream then it will perform those filter patterns And similarly we are using scan-civo for the another data set. We will be looking at the next scenario So we will be using tags called a scan-civo So the output will be ending up in the local host 5044 which is the lockstash running place so once we have logs and They receive everything we kind of use kibana kibana is like it looks like this. It's a fancy dashboard Using angler.js. So what it does is whenever we made a query using call local host 9 to double zero So this is similarly it will talk to their elastic search API and gets the data back and try to put it in a visualized format like Graphs like similarly like this some other So I'll quickly cover how to use this kibana in in the coming section not now I'm gonna skip for this now because if you understand this it kind of help you to put the data in a proper format and Showcase you to answer some kind of questions, right? So we'll get started with our first scenario. So it's kind of now basically I get Involved into ylk 101 I covered so that you can able to get and use your own data sets if we have any apart from the scenarios So in the first scenario, we kind of looking at mainly the certificate transparency locks And we wanted to monitor for an organization if any new certificate generated for their domains or sub-domains, right? and for this scenario what we have done is The one thing as we stated in the the abstract saying that in the real time So we kind of not did for this the workshop because it's kind of taking off whenever To get that certificate into the log in the transference log whenever it generated. Let's indicate certificate So what we had done is we have already created that previously certificates And we took the data samples what we kind of show you the how to run your own server as well as how we can get Those data sets as well to the json, right? So you can go ahead and read more about like the details of certificate transparency log how it works and everything They have documented each and everything here so We will be mainly using The certificate transparency logs Using a service which we are running which is by Cali dog. I'll just show case so there is a Script we have created custom scripts What it does is it will go ahead and look for the certificates is J any generated by the service Which we are running and it will go ahead and write into a json format so that we can directly pass into the ELK star, right? So this is a very simple Python code. What it does is Should I zoom in? so what it does is there is a Docker container to start the services for the Python for the Cali dog server Which will keep listening for the certificates which are generated throughout the certificate transparency logs from CRT data search So once it writes, it's listening on the local host 4000 server the Docker container so it will go ahead and make a web socket call and keep writing those data into the json file Okay, so that json file looks like what we have seen similar The one which we have seen using jq. So this is the output from that service So if you wanted to go back and try basically what we have to do is you just run this search stream server Which is a Docker container and we have created a documentation in read me how to run that service So once you have a run the service, you can have started a server to listen for all the certificates Transparency logs then you kind of use the client to store and send it back to the lockstash You Mean to say different kind of transparency lock states like a places. No, no Yeah, to answer your question. So that looks for all the certificate transparency locks, which are generated means it's throughout the internet It's not for the only specific to the domain. That is what your question It's not for saying that only buckrow.com or Defcon at all. It's kind of look for the internet by like a completely essentially That log is what the search stream server So Yeah An interface to So we kind of set it up that is the reason the Cali dog server Docker container rather than depending on them What it does is it is not only look for CRT.sh It's kind of look at Google transparency log Facebook has similarly a certificate transparency log So it kind of different places to look if it one server down on sets kind of loud balancer It'll go and talk to the another one and get the logs If you have internal applications, not visible on the internet Corporate applications within an environment that are not visible to the internet If you obtain an essential data certificate They will return a domain into the market If you search for a large organization like Accenture or IBM, you will see a lot of internet domain that are not accessible from the internet Yeah Yes I'm They don't give you who is information the certificate Okay, when generating a certificate So there are two ways to generate with the let's encrypt one is DNS based challenge second one is HDDB based challenge So either you have to point your DNS record basically saying that a record is going to point to the this IP address So that is the one verification another way is DNS challenge. So there are two ways they will be asked you to upload a file Yeah, it's kind of Yeah So as I said like previously, it's not only CRT.sh It's kind of looks for different places the CaliDoc server which that offer container one So it is like Facebook also runs their own certificate transparency log program like if you can give same buckroot.com here So it's kind of good and perform the same CRT.sh Like you can see like all the subdomains which are related to buckroot which are generated and all the details as well Like it's not only one server like one people maintain So Google has their own as well similarly and I think we have put it in the references section Where you can go back and read more about it and use different places But in terms of two hours workshop, we couldn't have not showing all the stuff So it's kind of like once we have got the data like as you have seen like this search stream So sorry Which service Okay, so this will be keep looking for the certificates But the time it is taking when we did the demonstration like to get the certificate can have took like 15 to 20 minutes To receive the certificate to the logs But the Facebook has really interesting features where you can see that you have a domain You can go ahead and subscribe to that domain whenever you get new certificate for any subdomains of the domain Also, you'll get a mail alert so that you can make sure that there is a Our subdomains are captured and if someone creates in your organization without your Notification so then you can have knows that there is a certificate generated someone has created a Subdomain or something else. So if you go here Facebook, they have a subscription can go ahead and create a subscription Based on the domain they kind of go ahead and sense that the email Cool So the ones we have got the data so we kind of currently already sent the data to Kibana Which is sorry elastic search. So Kibana is a front-end which will go ahead and query that and get back the data, right? So if you see here, we kind of using Tags to differentiate currently Just refresh So we are how like a different data locks because we for the next scenario also we already imported because of the time Intention so you can see those logs coming in here whenever you get the alerts are like whichever generated from the satellite transfer in So long so you can use this The filters especially in Kibana to query those are to make it granular to whatever the results Which we wanted to see because it's looks for the entire world right internet So we don't want to look for entire internet This can be done in two ways the best way to do this may be at lockstash level You can put tags saying that if matches to our domain then only send it to elastic search So that you kind of reducing your amount of data sets and you can easy to make it parsing another Or if you miss out there, you can have used Kibana filters to do that as well So I'll go ahead and apply the tags saying that make sure I wanted to see Lost again. I'll increase the font For that so tags is I'm looking for such stream logs only So as we used in file with saying the tagging features So we kind of leverage that to use filtering So we using tagging to make sure only look for the certificate transfer in C logs now So whenever you apply filter like you see 11,000 hits now you can see 1,000 and 227, right? It's kind of filtered down to the particular logs, which you wanted to see in Kibana And if you see the example Jason document like that stores in a Jason format, it's kind of The entire Jason It's super slow because it's the elastic search and lockstrips runs on Java. It's like it's in that memory So if you see that Jason, it's kind of automatically passes the Jason for you fields So you can use like a key value pairs queries like whenever you wanted to make a piece as well So it will pass enter Jason, whichever the keychain data keychain and all the domain names Which are part of this like as I said, it looks for all the domains throughout the Internet Some of the domain came as part of while we are running the scan so also it looks for which kind of the Digital signature and who is the owner like let's encrypt or dizzy dizzy sir or whoever it is created And the source name as I said it kind of looks for different data sources So one is Google's so we are kind of getting from Google organ to the native logs like the log Which is coming from the certificate transparency log And this is a log entry looks like so this is like pretty similar to what we have seen in the Jason But it's kind of fast structured way so that we can able to call it I'll reduce it So now what is main interesting stuff here is once we have captured the data We have identified the data source now We have placed the data which is in our visualization platform The third the final thing is interesting to look to us is querying the data sets and making sure Taking certain actions on top of it, right? So as I explained in the beginning, we will be using a last alert So we kind of created a last alert rule, which I'll again show you So it looks similar to this so whenever you see a new certificate for this This looks for the particular file bit index because the index if you see in the Kibana The name of this index is five it so we are saying that if any logs receive in this Index and it matches the time frame or whatever the reasons which you want to apply the filter suppose you may want to look for a different kind of patterns, right and Then also you can use these queries the one which you made using filters in the Kibana This is same. I'm saying tax is third stream, which is the filter which you see here This one you can also apply here tags Is sir stream Sorry, it's pulling me stick The same query so this is uses Apache losing which is a back-end querying thing for the elastic search So we will be using these queries To make the alerting and currently we are looking for the IETF dot org On the wild card applies here like it looks for all different kind of subdomains rather than looking for the particular domain And whenever it matches we can have send a slack alert So rather than slack alert you kind of send go ahead and create a zero ticket to where the ops team are admin team are security team Saying that new certificate has been generated is this part of your the inventory or like a cmdb like configuration management database, right? So once it has done we kind of look for the patterns here So this is where we kind of make interesting in terms of data sets So we can go ahead and look for different kind of patterns Especially the primary reason is meant for the alerting feature We kind of took this scenario, but not only that we kind of go ahead and look for subdomains as well So it makes it more easier especially here So I'll go ahead and open already saved search I kind of created a search so that easy to show rather than building here So basically what we are doing is we are trying to take whatever the parameters Which we wanted to show in the data sets so that we can only look for the granular data sets So here I can go ahead and run that query saying that I Wanted to look for the sudden domain So if you see here the query we are looking for the jira.insecure.dns Right, so whenever you see any instance a certificate has been generated for this query we kind of go ahead and see that it has been issued a new certificate and This is the DNS record and the other information if you want to drill down You can just go ahead and click this icon and I will give you complete JSON output of that particular certificate And you can see that it is generated by which time and what are the different resources regarding to that, right? So this is one way and I can also Use like wildcard to look at all the subdomains related to the organizations, right? This kind of gives you more Different results based on the use case So The main focus of the certificate transfer in Zilox or to look at here is To monitor for any new certificates has been generated or like to look for unknown like generation of your organization members, right? But we kind of moving to the second scenario now. Yes So you mean to say similar URL patterns are like a subdomains Different domain itself you mean this Yeah, like ABC.com means like ABC E.com like that. Oh Okay, okay, gotcha. So these kind of things I think you can't get from the certificate transfer in Zilox until unless they kind of use same CSR and using all domains Whatever they have as a single certificate They are generating like their own and if you wanted to look at those areas Maybe you wanted to get a inventory from your DNS like who are the unnamed Registar like a name cheaper the grantee so you can kind of use those data source and already make queries Already in the last alert. So whenever you make for those domains, you cannot get a lot So you have to make the inventory and put it in already alerting system So inventory has to be there like otherwise we kind of looking for the entire Internet, right? That's the only thing The similar domains you mean to say that that is the third scenario I'll be going to cover Good. So the intention is like mainly to take it forward in terms of making sure you are looking as a defensive way And as I was offensive way, so I kind of mostly done with the demonstration I'll just quickly show you Internet wide because we are lacking looking at currently only organization level if you wanted to look at Skate the how it kind of helps a certificate transparency logs Okay Oriaz is giving swag and stuff for the questions and cool Please give feedback like if you see any useful your features from the data sets It might be good to let us know like we kind of use and try to put it back in the documentation So we kind of written a simple CT log utilities using Python script What it does is you kind of go ahead and you the CA by third Okay, I just need to PIP Env install Python shell Okay, so what we are doing is we are trying to get the certificates based on the entire CA search with the Internet scale. So what it does is you can go ahead and Sorry, honey. Yeah Yeah, so Super slow. I'll just quickly show the how it looks like it looks like this. So what it does is So it look at it looks at Internet scale Like if you see this certificate authority had generated these many search at Internet scale and like a C-Panel Komodo has these many search Generated in the Internet scale and it's not only useful in that way And you can also use for our donation level like similar scripts Which is you can say that dot Buckrock.com says that let's encrypt has this generated these many certificates for that buckrock.com, right? You can also look at like different ways to use these data sets and Get some information out of your organizations using certificate transparency logs So before moving to the second scenario, is there any questions regarding this? Yes. Oh, yes History to be know because the certificate transparency logs is meant to look further. Oh, you mean to say this? Okay. Yeah, you can have used sub finder. There are so many sub domain finding tools I think if you I'll just you can go to our blog that app cycle.com. There is a really good blog about Pentester What is that ocean? Pentester guide to subdomains That is like more interesting in terms of like finding subdomains Yeah, I'll do in one cell get the results still. Yeah So people use that So Level up And we have created a cheat sheet as well I think fancy For finding the same thing Yeah, sub finder is the one which is written in Golang You kind of use the binary and you can specify that which one you wanted to use and we'll go ahead and find based on the all Ways like it can use like virus total. It can use like the passive DNS and also you can give your shodan keys API keys it supports and we'll go ahead and find Yeah Yeah, this is the one sub finder So you can just say that like whatever the thing which you wanted to look it has like a different Configuration which you wanted to specify using brute force dictionary. So it kind of does that cool My internet is low I think cool So we will be looking at the scenario to know it's mainly focusing on monitoring part rather than looking at alerting and other things and We will be looking at mainly to focus on two things One is as he said The cloudflare if you use it's kind of use a list of subdomains for the domain which are given But the people who ever use a cloudflare they may not really reveal their own IP address, right? So many people struggle like if you got a pen test or like the finding they get they are behind a cloudflare I don't even know what is IP address, right? So the DNA sets especially by this forward DNA sets by rapid seven There is a project called sonar by scans that I work which is so what it does is it's kind of collect entire internet wide scans especially for the different kind of records like a IPv6 a and NS records and other each text records MS records So what we have done is we kind of took a trimmed version of data set because of the demonstration Otherwise, it's like 300 plus GB of data sets. They have which we couldn't be able to fit in the VMs So we will be using the trimmed version of the data sets to kind of how you can use those data sets To kind of look at your external posture of organization in terms of security in terms of like the infrastructure which you have out there You want to add anything? So, yeah, I think it's they always release Monthly, I think once that same data set the problem with them is like they kind of release the entire data set again Rather than releasing the diff which has been done So it's kind of difficult to do that and if you wanted to know more like digging to deep I think They have done a pretty good research in and put it in a documentation We have referenced where they kind of explain more how they collect these data sets and the scans as well Okay, so what we have done is again for this scenario. We kind of already This is already captured data sets which are available at Rapids 7 scans that have a website if you go here I mean of you use internet here So I'll just quickly show the data sets I think it's already Downloaded and locally set it up in the VM But if you wanted to go back and look at the website is here You can just Google for Rapids 7 the FDNS data sets. It'll be there So this has a different types of records which are like a a the IPv4 IPv6 text C name as well as any records So we have captured those records and kept in the VM like if you see here in the So this is how it looks like the data set So what it contains is it's kind of timestamp and the name of the record which may be Domain name or the IP address and value and the type of record or maybe NS, TXT or SOA The records which are any different types of records and they have also segregated based on the type of record If you want like you wanted to only look at a records so they kind of set that data set as well and So it's already by default in JSON So what we have done is similarly again sending it to lockstash and elastic search and get into the Kibana, right? So once we have this data set the primary reason to look at this data set is huge as it Similar to certificate transparency log. So it's not only for the specific domain. It's kind of looks at internet scale So it has like the whatever the data which is available in the internet so once we have parsed the data set using the So extra feature as I said, we will be using GUIP filter So because of the value which is there. So we also added a The GUIP data pass for IPv6 as well. So what it kind of does is for IPv6 also, it will go ahead and give you the details so If I go ahead and change this filter to the scancivo I'll just increase the font One second So this is the data set which we have taken a trimmed version which has like 10,000 records But if you wanted to pass like entire data set, you can also pass a pass It's kind of support YLK stack at scale. So you can use like terabytes of data as well to pass So once we have done that data passing So it's kind of gives you complete visibility of the the information like as I said, it's kind of go ahead and get the Information about the GUIP that you're gonna see which location where the data set has the IP address has been Triggered like Singapore and the IP information each and every detail has been come because of the lock stash enrichment and People also use alternatively like a flu nd. It is similar to lock stash if you wanted to familiar with and write out So once we have this data set we kind of mainly focusing on trying to showcase those visualizations So that rather than looking at a single time Like if you wanted to keep looking it as a fancish sock or not dashboards whenever something happens and kind of builds those Visualization to represent is there any new domain has arrived or subdomains has been captured So I'll quickly showcase that the dashboard like it looks like this So rendering slowly So it's kind of looks like this So what we have I'll just quickly go back and show how I have built visualization So that you can try out for other data sets as well and based on your use case But to highlight the main things here is we kind of look for different kind of records Which we wanted to look at like is there any air records or like the air records are like a TXT based on the use case And we kind of map with the the location which it is generated from using a GUIP map Which is again available in the Kipana So that is one of the reason we picked and also we can have plot those data in a the graph Which kind of shows like which type of records has been coming so many for these organizations, right? so At the end again the entire data will be available as JSON format So that if you wanted to go back and rail down in terms of subdomain ways are like more granular way of the particular JSON record So if you see this for example rather than going and looking for the everything So I just wanted to go ahead and look for the particular domain So if you see here whenever I search the filter for the name, I can that are G So it will go ahead and apply those same query to the entire visualizations So this can be run in whatever the alerting as well If you wanted to go back and send the data which is coming out there So here if I go ahead and on most over seeing that this is a record I can go ahead and click that apply filter so it will apply that filter here And the data which is coming is only specific to the airicards which are created for your organizations, right? So whenever you see here the counting bridges basically there is a new airicard added for your DNS entry based from the proper DNS logs And I cannot use you which country they are creating if your organization is like across the world like it has like a different places people using the data centers and The apart from the main thing is here We kind of look at subdomains for those as well Like if anyone using same airicard to create multiple subdomains like I cannot use Jira and And CRM here cricket INT VIP so these all may be pointing to the same IP as well So you kind of use correlations capability of Kibana to make sure and look at the certain patterns as well using these kind of visualizations, right? So I'll switch back to one more level and try to show you how these visualizations has been created Just to give you overview so that you can go back and try for different data sets Especially if you are looking for the subdomains, maybe you are like using service like subfinder That will give you at the end Jason can go back and use these kind of features to look at the posture, right? So the data is by default available in the search patterns So once it has done the searches we kind of use visualization to represent those data Which ever has been produced. So Kibana has very friendly interface, but I think it's very difficult if you wanted to try out first time But if you like look some videos tutorials are two times It's a very easy to showcase the data which is available and most of the data science team And if you heard about Mozilla Mozilla entire defensive platform runs on Yale Gestalt like enter their SOC team and black hat I think they've gone there a sock also will use these kind of visualization because the amount of the data I can support is like the scale ability So we will be using different kind of graphs to showcase like we have created a Like already existing graphs as well. Like if you see here, I'll just show the existing one Like the scans I will based on the country. So it's kind of ask you with based on which type of events you wanted to create the graph Like we are trying to say that terms means like whichever the field you wanted to select to showcase there So we wanted to select the keyboard type means like a which type of regard it has been created So whenever you have applied you can give how many values it has to represent in this pie chart So we'll go ahead and apply this field. So if you rather 10 you can go ahead and apply 5 so it kind of reflects that data in near real time So this is the one way to showcase and people has build like complex visualizations if you see their tutorials and Things so where it kind of gives you more granular visibility Like if you wanted to apply based on the type of record and if it matches to certain IP address Then it will show the visualization. So it kind of gives you The especially business people management. So, okay, this many of the organization posture has been mapped to this IP So there are a bunch of other visualizations as well I think we kind of already created a pretty good dashboards for this other scenarios as well So what we have done is rather than going on teaching the Kibana We kind of put references to the Kibana tutorials if you wanted to try out and another thing What we have done is this entire dashboard can be exported as a single JSON file If you wanted to upload for your own dot asset can go ahead and import that JSON file It will be here if you go to the last management section and here saved objects, I can just click on the import and Use this scan cyber Kibana will automatically render that a dashboard based on this data sets for any of the scan Cyber stuff. I'll just skip because it's already there So this is the main one of the use case But people has extensively used this data logs to showcase like a different use cases Especially not only with the forward DNS They kind of use forward DNS data set with their own internal passive DNS So if you use your own internal passive DNS, you kind of use these logs with my correlating with the forward DNS data set To make sure that the alert is not false positive or some kind of thing so that they can ensure that The result is the true value Any questions before moving to the third scenario or the next section Yeah So for example if you have acquisition like Google has recently acquired like suppose What is that coming drop cam or something like long ago? Sorry So you can have go back and look at you have to write some kind of custom programming still because we don't know whether it is Acquired or not if you have that information We kind of use the correlation engine which is again in lockstash and Kibana to ensure that if it is related to this Then it can show in the certain organization level as well So this can the amount of the information which can be showcased It is very powerful if you have kind of data sets which are available to use that information Yes using lockstash. Yeah, so that is the reason we just quickly went through in the beginning But if you wanted to know more about like how we kind of use different data sets We have referenced the way how we can go back and read more about it This is Any feedback to improve in terms of like the data set like is there any other ways you people use like to look at like external infrastructure posture Like what kind of data might be useful to showcase here. Yes Okay You mean to say who is that I said You mean to say who we okay. Okay. Okay. Okay. Got you Yeah, yeah, so what he's trying saying is makes sense Actually, what we can do is like if you have found a domain or something we can go back and query the who is information for the domain We'll get back the who is the information like contact technical information and the phone number or email ID If you use that again You kind of use some other plugin to enrich that data and you can go back and showcase a reverse point to that. Is there any More data available. It's kind of going like a nested But it's kind of you more drill down information like how my I'll take a source like if you go and click and showcase But yeah, it can be done I think but it has to be put some work But that will be great to see like to go back Yeah, she will sometime who is users they kind of protect their who is regarding formation as well like Yes, yeah, but yeah that one is good like actually we can have and rich Anything else before moving on So as I said like in the beginning if you kind of use people The older ways like they have migrated from the servers to cloud play They kind of not able to get IP or the forward DNS data set may have historical data If you kind of download full 320 GB of data set you can able to see the world IP addresses map to that domains as well Yeah, we can have used in one of our engagement pen testing the kind of behind cloud fair We kind of used our Forward DNS data set to query that particular IP which is the whole life is till they point to the same IP So they kind of use useful information while doing testing Cool So that is one of the most thing So then we will move on to the section the third scenario where I think he's asked previously Is there any way to see the fishing domains like similarly domains? So this is a scenario which we wanted to focus on that area because I think Facebook has started extensive research on this They have a similar project On the similar areas. They are also looking at CT logs based on if you see any similar kind of DNS twisted a domain They kind of report back to the the team like to make sure that the Facebook This is not the Facebook domain or something Otherwise people go ahead and click that and use a password the gut Fishing has been an easily using those kind of domains so For this scenario, we can have using a data source by think all against certificate transparency log to look at bunch of stuff But it may not to be but in this scenario We will be using but I'll showcase where you can go back and try out other ways like a different log set as well So what we have doing is we kind of again loading the certificate transparency log So we mainly using a library called DNS twist. It's a Python base. I'll just see if it works Yeah, I think my internet is back So it's a Python Project in GitHub like what it does is it's kind of looks for the similar any domains available in the internet So if you say like buckroot.com as we saw it from the beginning So it figures out is there any similar domain which is available in the internet like But crowd g o o like that are any I'll just go ahead and run for the one of mine Yes I'll zoom in Yeah, make sense So the input is if an L list, what is the domain you said Microsoft Microsoft Oh, it already did that. So I'll just parse it using JQ fancy tool So you can see it has different kind of subdomains. Sorry different kind of domains which are similar to Microsoft domain If you see here users like what is this language like which country Okay Maybe yeah, so what people does is like they kind of use different kind of ways to represent That's all in browser. They don't go and see URL like coffee paste and see the text also, right? So it looks Microsoft, but we'll see over on top of that So this is the data which can be come from the DNS twist So what it does is the given a domain for your organization. It's kind of finds related domains different ways if you see this one this is Homoglyph the one I think it should see like insertion like if you see my cross-off Cross-off right like a different possible and the combinations It'll try and something which is very nice if you say omission as well Like if you just remove one kind of letter or something from the domain name to make look similar so and also it gives The information with all the list of domains possible related to that So you kind of use this inventory to go back and look at is there any similar phishing domain has been created And someone is using this to mimic basically our organization or like used to do some kind of spam Or like especially this is kind of data sets people look in fraud detection Teams if you have like fraud detection teams in your organizations Like especially large e-commerce companies They kind of heavily depends on these kind of things because they kind of use to go and check out this using this link photos you can now know like So Yeah So This one I think yeah the command Actually the same people I think kind of built a fancy website as well DNS twist dot report Twister report Yeah, this one So the same what we are said command. It's kind of you see a option here Mike if I say Microsoft comm He'll go ahead and look for the domains available And it'll says this is your version of it's kind of gives to IP information again You can go back and send it to your lockstack to enrich the data where it is has been created and all stuff And not only does that you can as iPhone or flag It's kind of shows undisolved the stuff which is able to resolve or not resolve for the domain which has been there Right, so this is the data set which we were playing with the third scenario Especially looking at the phishing campaigns has been created for your organization some by someone And the Facebook has similar project I think we have put it in the references section where they have working on similar areas to look for anyone has been created as detection for them So similarly we will be using for this as well like the certificate transparency logs To look is there anyone for our organization has been created a certificate For one of our similar looking domain like phishing domain by DNS twist So first way what we have to do is we kind of get the data sets using the DNS twits So this script does that automatically so we kind of put this script in our code when we are distribute So the output of this the DNS twist file will be sent it back to again lockstack and vice versa to ELK and Lackibana so What we have to do is extra feature apart from the first scenario and the second scenario We kind of get all the certificate transparency logs using the Certificate transparency log that script and the service so once we have Get the all certificate transparency log the pattern which we wanted to look is not only for the other organizations So we kind of look for all different kind of possible ways to look for the similar domains So if you look at one example, we have created a thing. So in the first scenario, I think we have used this So if you see insecure DNS, we kind of create this is owns by this this domain owned by us the insecure DNS So if you see insecure DNS, there is a zero at insecure.dns.com Like the domain which has been created. I'll just zoom in This one. So similarly what we have done is we kind of bought another domain looking similar to the insecure DNS But here we are just trying to remove the D basically insecure DNS So if you go ahead and run the similar query for the thing So if you see here payment.insecurens.com But if you see there is a D missing here insecure DNS But someone had generated a certificate for our payment server, which is publicly available They may be using trying to legitimately sending to some other people or customers who ever uses this organization, right? So using this kind of log data sets which are available publicly and looking for This kind of things may give more visibility in terms of what kind of things has been happening to your organization in terms of external The posture, right? So this is the one way to look We are still working on making sure that the DNS twist records data set will be coming back to lockstash to enrich And I think as someone mentioned we may work on that as well like looking who is information and getting back into the Logstash to make sure go back and drill down. So the what we are trying to do here is once it has been received We kind of creating a last alerts against some patterns using the last alert rules it So if it matches any of our DNS test records for the certain domain for our organization So we kind of go ahead and send alert to the the upstream saying that do we own this domain? So if this is yes, it's kind of our own domain So if it is not part of our inventory, we may want it would go ahead and take action saying that Sending an email to our one of the customers team marketing saying that if you see anyone sees the domain name in coming Similar to this area, please do not click or use that at least it kind of helps saying that disclosing point of view, right? Is there any questions in terms of this scenario? Are like the data sets any feedback? So Facebook is working on similar, sorry So how many of you run fishing campaigns at least for your organization like not you like at least your teams, yeah So the problem with those things it only works for your organization level, but what about your customers, right? So that is where we wanted to look at transparency logs so that we can see at Internet scale and look for send it back But yeah, that is really good. If you wanted to block it local things corporate level. Yeah, it's really good approach Mm-hmm. Yeah virus total or something like that. You mean to say oh Okay, you mean to say me the people who has done that Okay, so what if it is legitimate domain there someone our nation is using Classification maybe yeah, yeah, that is for sure This could simply be Yeah Advertise Exactly So I think that is More automation kicks in place like if you use phantom just like things like you kind of look for these domains and take a picture of the home page And compare with your like google has releasing auto ml if you know So what they are trying to do is still in beta. I think so you can upload both in all the images which are there So it kind of classifies based on the category So maybe you can kind of use these scenarios and go ahead and take all the phantom Yes, actually firefox support now headless can go ahead and run that script in the Automated way so you take all screenshots where you are phishing domains and compares with your main domain If it matches using auto ml kind of go and report that it's kind of automation Cool, uh, which one? Oh, okay. Yeah It's like it's kind of interesting to see so much stuff around this area because it's kind of more security work, but it's kind of more has to be put in terms of Uh, uh, thinking wise like how people innovatively will come up with the phishing campaigns are like new ways to attack Yeah, yeah, so historical data you mean to say like old data set and the new one you mean to say Yeah, it is possible. So the last alert, uh, as I I think I'll just go back to that slide just to showcase Yeah, so what it does is it has a different way to look at this thing. So if you see one of them thing is like The change feature is looks from the previous data set the timestamp And if anything apart from that something looks newer than the previous anomalies, it's kind of make an alert So it will look at the whole data set you can define which timestamp to which timestamp to look like I can look for past six months and if it doesn't match from the past six months similar area of patterns will go and and alert also as well So it can the problem with especially using open source solutions. It's kind of need to be built so much The fine tuning but once you have place fine tuning it's kind of works at scale and customization But uh, purchasing product is like, yeah, you just go and talk to the team Yes Oh, you mean to say for getting that data Yeah Oh, okay. Okay. So he you are saying that use google docs to find subdomains or insensitive data available using like index.off or file extensions those things Yeah Yeah Yeah Yeah, I think this is the one now. I got internet. I'll just go back and show So this is one of the popular blog which we have written and across internet buzzing So this is I think you if you are more interested in especially finding subdomain immigration We kind of wrote a blog post like The how it kind of can perform subdomain immigration way for different ways So the one way is kind of looking at techniques using google locking as he mentioned So what we can do is you can use different kind of site operators to look at like what are the different types of Subdomain like store uk Jobs and different ways like you can get this and people has written scripts like recon ng one of the framework for the recon so they also use similar techniques to do that as well and The people also use google, but there are so many other search engines out there Which can also give results like being it's really good if you wanted to look at subdomain immigration Yeah Kind of see you can get like juicy information like slack jira What do you want? The second thing is like we have kind of focused on virus total As people keep updating uploading files and information people also capture the metadata nothing but like a file owner information The file the downloaded urls like a cell reference if you use some kind of url So these all has been captured as a metadata part. So this kind of gives subdomains as well like Like you can see like for wikipedia wiki media You can have seen huge amount of subdomains which are from this and the third thing is we use Dumpster dns dumpster when it kind of gives the different subdomains and put it in a geographical map saying that which is map Like how we saw in the kibana previously it kind of enrich the data information And as someone mentioned, I think I think he mentioned saying that sublister is again one of the things like similar to subfinder So it can go ahead and look at bayadu Ahu google being different search engines as well as the passive dns And also it looks ssl certificate, which is certificate transparency logs again And I kind of give you complete information What are the domains we have come from those as well? And like the certificate transparency logs this we have looked in the scenario one as well Like this is another way to look at subdomains if you wanted more interested And as I said they kind of give you postgres interface to query This is the script and if you go to our blog post we have kind of given the script to make and use for your applications And the another thing is dns recon This is again kind of brute force and look at different ways to go back and find subdomains And it uses bunch of other ways I think so many other like it's kind of huge workshop itself finding subdomains right Yeah I just wanted to highlight this part like because if you seen in the second scenario we kind of using fdns set So if we have compared different data sets and fdns has a huge amount of data set which has given subdomains Because it kind of stores entire historical internet level data sets rather than looking at certain tool It gives more visibility across the data set. So that is one of the reason we picked the data set for fdns And the thing is like we have created a cheat sheet if you wanted to go back and try out feel free to do We will be here as well like if you want to talk to us and ask any questions So moving on So this is the one scenario which we wanted to cover for the last part and what we have seen is mainly focusing on looking for the phishing domains And as so many gave feedback we kind of update any of the things which we can try to use like who is data sets and compare it and correlate To showcase any data which can be possibly legitimate or not And the thing is we kind of not able to put like the time to make sure everyone has to do here like for the workshop But as we said like we kind of make this update the github with each step by step screenshot as well We kind of tweet from the recon village or our apsicle account You can like get entire the github and the vm go ahead and feel free to try whatever the stuff and please add your own data sets If you find and try out like how you can correlate and get more value out of this And if you have any questions, please let us know Then we kind of showcase the references part before concluding the workshop so that you can go back and try other ways to look at The same data sets to take it forward Any questions Yeah currently by default it doesn't give but we have to just use some other tool again to make But sometimes people wanted to not to show their information as well in who is so they kind of use who is god or something to protect stuff Yeah Yes Yeah, you mean to say if they given a domain and Like by name or like a kind of name I mean to say I'm thinking like do you give the domain as a input for the attacker or like the tester who is going to test Or like the company name Company name. Oh, okay Yeah Yeah Maybe one thing is like we can go back and refer to the site google or some other things once we got we kind of correlate that information Like is there any referral URLs from this site to going back to that like if you some people like Shopify or something like they kind of use app dot Shopify some different URL to give them shops Externally, but the main domain is completely different from them But if you see any of the referrals like a store go to click on the store or something He'll go back to the whole domain There are multiple. Yeah, it's kind of Yeah But then you run into the case where I'm using my personal work email to stand on the site Yeah, what is that? First of all Who's For the domain The information may not be company But from that set So if you are internal it will be makes easier you just look at passive dns traffic Uh, like if you are internal organization member and as an internal organization You wanted to look or test your organization poster Then you kind of look for passive dns where it will before going out of your network like a gateway So you can look for all passive dns traffic and profile them Maybe it is one of the best way to look because whatever they have to look They can go through the same dns which you have organization. So it will give you complete entries Uh, but Coming to the external poster, maybe you have to do some kind of work In terms of different techniques and ways to look at but yeah It's a little tougher to automate until unless you put the manual steps in place and document it Then Yeah Yeah good So, uh, that is the things I'll just quickly cover whatever the references we have put like so that you can go back and look at them Uh certificate transparency page like you can go and look at uh the more details about what how the way Capture and what are the data information they give away and uh the places to look if you wanted to get the certificate transparency logs and uh There is a blog post we have written like a as a kind of what do you call three different formats like a three three types Like three types now dark side bright side Only okay So there is a blog post we have written about more like detailed what are the things information it discloses and about stuff The project which we were talking when we are doing scenario one like the callidoc security This is the one which kind of automatically run your own server for listening all the certificate transparency logs And the rapids of an open data set uh is uh scans that i o So it not only has the forward dns data set. It also has a bunch of other data sets which you wanted to look and may be interested Uh, if you see this project shona Think yeah, uh it went away This one Yeah, this one. So the project owner has the uh data sets like uh, they kind of give you different ways to download As I said, it's kind of huge if you wanted to download at internet scale and people kind of use like Some huge servers to parse and process Uh feel free to go ahead and download and try those stuff and uh That is mostly about us and uh, if you have any questions Please uh, you have cars Please feel free to take our card or like we to us like happy to answer And we will be sending this or tweeting about this workshop the documentation and vm. Uh, mostly tomorrow Uh by tomorrow Thank you so much and uh, have a great day


Let me take a deep breath. We often hear the phrase self-made, but I always believe that somehow something or someone gave you a chance. I want you to think when that moment was or when that opportunity was for you. I'm Jan Kavin-Bowis. I'm a product designer at Automatic. I'm also an illustrator. I'm also a mom to a very, very active little girl. She's currently 17 months old. She'll be two on New Year's Day next year. I'm really excited to be here. I have been at Automatic for almost four years. I feel very fortunate to be working with the smartest, most passionate, and most talented people in this industry. But how exactly did I get here? So let's hop into this time machine for a moment. So I was born and raised in the Philippines. I lived in the central part of the country in an island called Cebu. Ever since I was little, growing up, I've always been into drawing and illustration and cartooning. I think I was influenced by my dad, who is a great illustrator. But I don't think I'm half as good as him. But my dad is a computer engineer. So growing up, I somehow was molded into becoming an engineer or probably working a job in the medical field. But my parents weren't exactly tiger parents. They're kind of like at least borderline tiger parents. And when I graduated high school, I had to go to college. I applied at a university. And I picked nursing as my chosen major. And unfortunately, the test results came out the next day. And I didn't quite make it. Of course, my parents were disappointed. But I was like, yes, I didn't pass. Because I didn't want to be a nurse. But of course, it's a great job. But it just wasn't what my heart desired. And so I applied at the, I still needed to go to college. So I applied at the same university at their engineering campus, which was at a different location. And the applications were almost due. And the night before its due date, my dad said, you know what, you got to pick your major. And I had no idea what to do. So he said, you know what, I'm just going to write you down for computer engineering. And I said, OK, I didn't really have much of a choice. I had to just accue us to whatever he said. And I wasn't fond of my major. I took up computer engineering. But I loved computers still. I loved tinkering with computers. I loved drawing on MS Paint, the internet, that sort of stuff. And when I was in college in the early 2000s, I had an idea. I wanted a place online where I could share my illustrations. And there weren't that many options for me. There was GeoCities and some other site builder. I can't even remember its name. I picked GeoCities. And I remember on the home page, I remember explicitly that I wrote, I want this to be a place where I could share my illustrations online in a shared effort where I could improve my own skills and help other people improve theirs. But then I wanted to name this doghouse as a tribute to a family dog that passed away. But this name was already taken. So I just couldn't come up with a better alternative. So I just named it doghouse, D-I-W-E-G. And I wasn't happy with the templates that GeoCities had to offer. I didn't know any code. I didn't know how to customize anything. I didn't know where to ask for help. So I couldn't take this idea any further. And so it basically died. And college took over my life anyway.  And that got me really excited. And the course package offered HTML, CSS, and MySQL and that sort of stuff. But I couldn't really be bothered by any of those because all I could think about was animation. And I was like, oh, finally, I'll be able to learn what I've been wanting to all these years. But it turned out that the animation that they meant was actually the HTML marquee tag. So I was really frustrated. It wasn't what I expected. But I already paid the down payment. So I'm just going to go ahead with it. But I fell in love with it by accident. I learned the basics of building a website. I felt really empowered. But it didn't really dawn on me that I could actually make a career out of it. And it didn't help that one of my college professors, he said that these web designers, they don't really have, they're pretty much worthless. You should learn other languages like Java, C++, assembly. So I still kept on going. I chose to do that because it made me really happy. I continued doing it because out of pure enjoyment, really. And then my last year in college, I bought my very first domain. At that time in the Philippines, owning a credit card wasn't a really commonplace. So at first, I didn't know where to get one. But then fortunately, I found a small hosting company just in the city where I could make in-person payments. So every month, I would commute and make payments for my domain. And then eventually, I graduated college. I applied for jobs. But in my city, there weren't really that many jobs available. And it didn't help that I didn't really have any experience. I didn't have anything to show in my portfolio. So nobody wanted to hire me pretty much. And then a lot of my classmates in college, they took on jobs at call centers because hiring for those was really hot, even up to this day. And I actually consider that. But then it wasn't really what I wanted to do. It wasn't what my heart desired. So I thought maybe for a little while, I'll work for free or for really cheap. And I posted ads on Craigslist so I could get more projects. And as you can see, there's very affordable web design services, or $100 web design services, or really cheap web design services. And I had a few different alternatives for the post-itals to somehow make it a little bit more attractive. And then I found my first project. This was the first site I ever built not for myself, but for someone else. This was for somebody. It was a site that I built for free for someone. It was for somebody in India. And then my second project was a school project for somebody's finals. It was really frustrating a lot of times because my rates were super, super low, but then people still ended up not paying me. At the very beginning of the project, there would be promises of payment. But then towards the end, nothing happened. But I kept applying for jobs. I blasted out my resume. I kept showing up at interviews. And then eventually, I found a job as a junior web designer at a design and development agency. I think part of the reason why they hired me was because I became that annoying applicant that they kept calling their office until it got an answer. And I have to say that this company, they gave me my wings because I learned a lot from them. I learned a lot by doing. And during the day, I'd show up for work. And at night, I would focus on tutorials and learn as much as I can. And back then, when Flash was popular, I would try to decompile this Swift files to somehow try to figure out how these animations have been put together. They always didn't come out perfect when you decompile them, but it somehow helped me still. And in 2007, that's when I first heard of WordPress. A lot of my designer and developer friends, that's what they were talking about. And a lot of the blogs that I followed, the design blogs that I followed, they were powered by WordPress. So I wanted to be one of the cool kids. So I tried learning it. But then when I tried learning it, it was a little too intimidating for me. Lots of PHP. My brain wasn't wired that way. And being my impatient self, I really wanted to roll something out. I really wanted to be part of the cool kids.  It was a static HTML file site, a typical looking WordPress site with a list of posts and a sidebar. And then when you click on one of the posts, it actually links to just a static HTML file. And then it got really popular, actually. For some reason, people didn't pay attention.  But eventually, I was able to convert it to a real WordPress site with the help of an old coworker. And one day, this site got featured in what used to be a popular design and development blog. I think they're still around today, called Six Revisions. But they credited somebody else. And that made me really upset. Obviously, somebody stole my site. And so I complained to the owner. And I wrote them, you know what? Like, this is my design. Obviously, somebody stole it. And the owner, Jacob, he was really nice. And he was really accommodating. And we were able to sort things out. And he asked me, would you be interested in writing a tutorial for us? At first, I was a little bit hesitant. Because first of all, it's something that I've never done before. Second, there's a lot of design blogs out there. Smashing Magazine, Web Designer Wall. So why would anyone want to listen to me? But then I wanted to stay true to what the doghouse was intentionally for. So I said, OK, I'll write a tutorial for you. So this was the first ever tutorial that I wrote how to create a web 2.0 web design and Photoshop. Surprisingly, it got really popular. It ended up on the front page of Dig. And everyone familiar with what Dig is? It's a popular, well, it used to be a popular content or news aggregator. And the Dig effect, that's when your article or your site gets picked up by Dig. And Dig brings you so much traffic that it just crashes your site.  So somehow it got viral before viral was even a thing. But alongside its popularity, it amassed a ton of negative feedback. Probably like 98% of the comments were negative. And of course, this was the first ever tutorial that I wrote. I held it close to my heart. And then people just ripped it apart with no mercy. But then I still wanted to stay true to what the doghouse was for. So I decided to keep writing even though it was really discouraging at first. And so aside from tutorials, I also put together like texture packs to give away, like icon sets for free. And this was one of my most popular icon sets. It's the old social media, bio crown icon set. And to me, it was a really good feeling to be able to not only help designers, but help non-designers as well, especially non-designers. Because it allows them to create little customizations without the need to hire someone. And I always believe that the more you give, the more you'll get back. Blogging opened up a ton of opportunities for me. It opened up a couple of speaking engagements in the US, publication features. And the one publication that I'm really proud of was this web designer magazine where I was featured in the same page as Matt. But in 2008, I felt like I hit a wall. I wasn't growing anymore in my career. I felt like I needed a change. And there were no more companies to apply it in my city. It's either I had to go to the capital of Manila, but that wasn't really an option for me, or I had to leave the country. I had a job opportunity in Australia, but that didn't quite work out. So I decided to use freelance full time. It was a risky decision for me, because you know you don't get that steady flow of income. And so I was scared at first. But then blogging allowed me to create a pretty decent client base. So I didn't have to scour the web for jobs and projects, at least for a little while. And then one day I got an email from someone. He was one of the co-founders of SendGrid. Are you familiar with the company SendGrid? No. So they're an email delivery service company based out of Boulder, Colorado. They have offices in San Francisco and Irvine, and they're recently actually IPO'd. And this email, he told me that he found my site through one of the design blogs that I was featured in. And he said that we're looking for a designer. And he really liked my work, you know? And we chatted for a little bit. I vaguely remember the details of our conversation, but he asked me where I was located. So I said, oh, I'm in the Philippines. And he said, oh, would you like to move to California? And I said, okay. And I never anticipated this big of a change in my life. I've been to the US before, mainly just for conferences, but I never really thought of moving here at all. And I worked for SendGrid for about a year to a year and a half before I actually moved to California because the whole visa application process is such a long and arduous process. And of course, I had a lot of doubts and fears moving to a new country. Everything was new. I didn't have that many friends. Well, actually I didn't have any friends at all. I didn't have any family. The only family I had was in a different state. I didn't even have like a credit score. I couldn't get an apartment. Every time they ran my credit, it would not even return a number, not even a zero. It wasn't all. So it was really tough for me. But for the most part, everything just happened in a blur. And then in 2014, I found, I was actively looking for a job again and I found automatic and I applied and after a couple months of going through the trial, I was happy to find out that I actually got hired and that was really thankful for that. And now I've been in the company for almost four years and after everything that I've gone through, I feel like I'm finally back home. And so looking back, I've learned a lot of things in this journey, but the one thing that truly resonated with me was that, if you're about to do something that especially is something that's totally unfamiliar to you, then of course it's gonna be, you're gonna be scared, there's a lots of doubts and fears, but just do it as Nike says it. Don't be afraid to put yourself outside your comfort zone and online people can be quite vile and just tend to spew stuff out, but just ignore it really. Over time, I feel like my skin just grew thicker and as Dory says it, just keep swimming. So I wouldn't be here today if I wasn't given that chance. WordPress gave me a voice, a voice that allowed me to reach a vast amount of audience that otherwise wouldn't have been possible. Blogging allowed me to create a platform to be able to showcase my work. Don't downplay certain things like blogging because everyone owns a blog now, but those things can lead to positive outcomes. So I wouldn't be here today, I wouldn't probably have even met my husband and wouldn't have had my daughter if I hadn't started writing. Thank you. Anyone have any questions? Yes. I think it's when you know you, okay. What would you say most bloggers, what discourages most bloggers or makes you stop blogging? I think a lot of that has to do with not getting an audience because you keep writing and writing and then you get like a couple likes here and there and then sometimes there's zero likes so that can be really discouraging. But my advice is just keep producing quality content, content that people will wanna read because for me I blogged at least once or twice a week and then every Friday I would give out free textures for designers, so that really was helpful. I don't know if I caught it or if you said in there, what year was it you started blogging in relation to the year 2007 when the world fell off the face of the earth and all went to Twitter and Facebook? Yeah, that's really where I'm curious, when did you start with relation to that year? I started in 2007, I created that fake blog. I had a couple blog posts and then when a couple months later that's when I started blogging for other more popular sites other than my own and then I would just cross post it that helped me bring a lot of traffic to my site. Yeah, does that answer your question? Any more questions? He has a question for you. So Jan. Yes. A couple of comments right here. Oh, hi. And just keep writing. Yeah, WordPress also changed my life. So I've been sitting here somewhere under the freeway or something. But anyway, what do you think you consider the highlights of your career from being a designer to working for Automatic? What was that again? What would be the highlights of your career? I think definitely working at Automatic was a really important thing that happened in my life and that magazine feature with Matt as well was a big thing for me. Eventually I got like several publications that I wouldn't even read them anymore, but yeah. And also the other thing that I wanna ask you are you also active on social media like Instagram and Facebook? Are you getting a lot of traffic from that and they shouldn't do your blogs as well? Yes, I am active on Instagram and Twitter. Well, I used to be on Twitter, but now it's just Instagram like posting pictures of my daughter. I have to admit though that I used to blog a lot, but it kind of just tapered off over time. I think that's when after my daughter was born, it was just like a really big change in my life. So I couldn't get around to writing more stuff, but yeah. Anybody else? Anything else you wanna say? No, thank you all for showing up. Yep, thank you. Good job, James.