Hi everyone, my name is Ashwant, I represent Walk-In. So Walk-In is a business consulting company that provide business solutions as well as tech solutions for retail businesses. So over the last one and a half year, we've been working with Cafe Coffee Day. I'm sure all of you must be knowing or have been to CCD at any point of time. Needless to say, CCD has more than 1,500 stores across India with all of these stores and at least 200 odd cities in India. So when we started working with CCD, they had a, so Walk-In started working with CCD about one half year back and CCD, we were mainly focusing on doing data analytics for them. So they came back and asked us questions like, who are my customers? And we said we don't know because we don't have any data for it. So Walk-In sort of looked at this problem and said, how do we solve the problem of collecting data for CCD so that they can better enhance the customer experience? Also, CCD did not have any single channel to communicate with the customers. The customers also did not have a channel to communicate with CCD. So we looked at this problem and we sort of went back to the Blackboard and we sort of came out with an experiment. So what we did was we went to one CCD and we said, for all the transactions that take place at the CCD, let the customer give their mobile number. Now, how this helped is for the customer, the incentive is every time he gives the mobile number, he gets a discount. And we have done this experiment over two months and we have found out that lot of customers who have given us their mobile number, they're actually transacting much higher value than the customers who have not given their mobile number. Also, these customers are more frequently visiting the cafes compared to the customers who have not given their mobile number. So this was clearly a loyalty problem that CCD has wherein we have established that CCD, we can leverage CCD's customer base to engage loyalty with the customers. That's how walk-in platform sort of, we envision the walk-in platform, which we built for CCD. And the platform basically has three elements. One is the payments. Payments are all the transactions that the customer does at a CCD, you know, goes through this platform. Then another is loyalty. We provide a loyalty benefits for almost all the transactions that happen through this platform. Also, we sort of, you know, enable personalization. So when I mean personalization, it, based on your spend patterns, we sort of tailor the offers that any customer gets. Also, any recommendations that we can provide to the customers. So one of the manifestations of the walk-in platform is the CCD mobile app itself. This is a customer's app, which anyone can just download from Play Store and install it on their mobile.  So downloads are more than 2 million and transactions are around 2 to 5 million. And even we have a healthy Play Store rating of 4.2. So the scale at which the app grew was very fast because it's only been like eight months and we have grown almost into a million plus downloads and transactions. So the marketing was, which was done here was mostly, you know, in-store marketing or whatever marketing, there was no other external marketing agency which was doing the marketing. So over the talk, I want to talk about different learnings that we have had building this platform and any challenges that we face. So the first and foremost point that I want to bring about is ease of use. So whenever you're building any app which you want to build for scale, you want to establish that this app is very user-friendly, otherwise the users are not going to come back. So here we looked at various options for, you know, how does the customer communicate to the barista? So in CCC's case, one of the main important places where customer needs to communicate with the barista is at the CCD counter here. He might not have internet, so we need to rely on other communication methods. We looked at Wi-Fi, Bluetooth, Sound, and then QR code. So Wi-Fi, Bluetooth has problems like users have to switch on the hardware, and Sound was still much more not so proven technology. And QR code has been used a lot by other people. And we've also found that with QR code, there are less chances of error. So we finally resorted to choosing QR code. So how does the user experience actually happen at a store?  And the barista asked me, do you want to pay using the app, so? And I say yes. I take out my mobile, and you know, I generate a QR code in my mobile, and I show it to the barista. The barista then scans the QR code, and voila, payment is done. Immediately after that, you can go back to your place, and the barista prepares the order and fetches it at your table. So here, it looks simple, right? So basically, you don't need to do anything, you just need to produce a QR code, and the payment is done. So here, some of the important aspects we took when designing the app was, one was user choice. So as you can see, there are like two options there. One is used beans for beverages, another is paid with wallet. So beans is a loyalty point for CCD. So whenever you're transacting at any cafe, then you get some loyalty benefits, and those are accumulated as beans, and wallet is just a regular wallet where you can add money. The reason for providing these options is that we wanted to give the user choice over what options he wants to pick. Also, we wanted to support multi-mode payments, wherein let's say if you have balance in these two, and yet you don't have sufficient balance to pay the bill, you can still use cash and finish your payment. Another important thing here was we wanted to ensure that the app is sticky. So if I'm just going to use cash, then why will I bother using the app? So that's where the loyalty benefits actually jump in, wherein you still need to use the app because you want to gain loyalty points. So this was very helpful because a lot of customers came back and told us that as a customer, I just want to accumulate beans, and only after a certain point I want to spend them. We've also ensured that the payment flows in the app are redundant. So when I say redundant, I mean to say that there are multiple ways and you can end up at a payment flow. For example, in the app, as you can see, there's an offer which is magical bruise offer where you can just click on the offer, add the offer, and then choose your payment modes, and then create a QR code. Or you just want to spend by cash, but just accumulate beans.  So through this, we have ensured that as a customer, you have your favorite way to get to payments. So we sort of provide the customer both user convenience, as well as ability to pick their own payment modes based on what their circumstances, whether they want to earn beans or redeem offer or anything. So I got my user experience locked down. Now, how do I ensure that my app is secure? So because we have QR code as one of the important channel to communicate to the barista, we want to ensure that QR code is as secure as possible. The first and foremost thing that we did was we disabled the ability to take screenshots on your QR code. So we wanted the QR code only to be generated from the app. And we've also added signing to the QR code because we wanted, we didn't want anyone to share the QR code outside the app. So it'll be very difficult for you to create the QR code outside the app. Another interesting thing that we did was we added timestamp which enabled us to debug any payment errors that have happened. So even in case of fraud, you can readily identify using timestamp if there is a QR code that was not generated from the app. You can subtract the transaction time minus the QR code creation time and you can figure it out pretty decently. Another thing that we did to the QR code, we added a number to every QR code that is generated per user. Now what this enabled is whenever you generate a QR code this number is synced back with our server. So in a way that we are invalidating all the QR codes that are generated for this customer. So at a given point, only one QR code could be generated per customer at any time. Also if you're building any systems, not only payment and your factoring scale, you need to factor fraud. So the CCD app for us said the traction grew really fast and we sort of burned the fingers with a lot of gamers who try to download the app and use the loyalty points, et cetera. So here important thing is device and user restrictions. So if you're building any app, you need to factor in how many users can in a device how many users can install an app or in multiple devices how many one user can install the app and how many ever devices. So these restrictions need to be put in place if you're building your app. Also because it was a brick and mortar company, we had this problem that a lot of barista end up using the QR code for getting beans on behalf of the customer. So you either end up with really good customers all over or you end up with a lot of baristas. So you need to always keep note of your outliers. So these things are something that we learned on the way and these things I would advise anyone who is building their app to take care. Also in a brick and mortar, a lot of things can go on, a lot of things can go wrong on the ground. Although you got your user experience, right? You got your security, right? You know, many things can still go wrong. And for us, one of the most important thing was or one of the things that went wrong was, you know, many times the process offline. So how do you do the payments? So we first tried to minimize, mitigate the liability. So what we did was we delayed the amount of beans that was given. So after a transaction, after a delay, you will get your beans. That kind of reduced the risk. Also, we looked at other modes. So one of the mode we looked and also worked on was goes as follows. Where in a customer walks into the store, he takes a, he makes his order and then the barista says, sir, the pause is offline. And the barista prints the bill and the bill will be printed with a QR code. So the customer now have to open his app, scan the QR code, do the payment on his mobile because he has internet. And finally, once the bill, once he does the payment, he again needs to generate a QR code which will then need to be shown to the barista to validate the payment. Now this flow, although it solves a problem, it went back wrongly on the user experience because there are two points of failure. One is scanning the QR code and another is showing the QR code. So we are also working on solving this problem with other ways wherein you can probably order from your app instead of ordering directly at the barista. Often neglected, the barista training is very important because in a FNB or in any retail business, the attrition rate is pretty high. So you can't expect to always held training sessions because your baristas come and go or your employees come and go. So here we first try to solve this problem by introducing videos into the POS software itself. So we then figured out that that was not possible because POS hardware did not have enough storage. So we supplemented the barista training by providing, creating WhatsApp groups and providing videos in them. They've also added, created, printed material so that the barista can, which is available at every store so that the barista, if he has any doubts, can look at and figure out. So when making the training material, one thing we kept in mind was, you need to be very, very specific because all these baristas are not so tech savvy as most of us. So you need to drill down the point to the minute details. And another thing that we did was business process re-engineering. So in a case like CCD, it's very important to provide accountability at all levels. So what we did was, we tried to change the process, business processes to accommodate the use of app also. So what we did was, whenever any customer complains in the app or gives a one-star rating, mail will be sent to the area manager of that cafe and then area manager will now need to be accountable for the customer. So he either needs to resolve the issue and then contact the customer again or he resolve the issue and we can get back to the customer. So here we have ensured that at all levels, the employees are aware as to what is happening and why are we getting bad ratings if there are any. Also, I mentioned that much of the marketing for this was word of mouth. So in order to bring that, we also had to incentivize baristas to market the app effectively. So we gave any incentive to barista who markets the app and transacts using that. That even the barista feels that this app is not a new thing but it's part of my work. Another thing which you should keep in mind. So I guess that's pretty much what I had in mind. So overall, we have had a great success story and we've been effective in building the app and also creating a good user base. Probably over the next year, we are looking at how to engage the users now that they are onboarded in the app. With that, I'll leave the audience to questions. So if you have any questions, just. Thank you, Ashwant. Can we take some questions quickly? Hi, Ashwant. Can you hear me? Can you hear me? Yeah. Okay. First of all, great talk using the CCD example. Thank you. My first part of question is on your data analysis and second is on the process of the payment. Okay. So data analysis is, you mentioned that people who gave the mobile numbers voluntarily were more frequent than people who didn't. How do you know that for sure? So we sort of guessed that number because we knew from the people in the cafe as to how many users frequent and when they are coming. So it was sort of a guess because we didn't have real data validating that and it was also very local to one particular cafe where we ran this event. Okay, so mostly visual analysis? Yeah, mostly visual analysis. But over, after this app, we sort of know it for certain, yes, people who are coming again and again, I mean, who are using the app are coming again and again. So this, thanks. So second process of your payment, right? I want to understand it more clearly. So the bill is, you place your order at the counter. The bill is generated and you know the total. So the barista guy tells you the amount and you create a QR code on the app and the person then scans the QR code. So how is the payment made through the wallet or the customer still swipes the card? So is this the pause offline scenario or is the first? First online scenario. Can you repeat the question? The question is on how the actual payment happened. So the customer placed the order, the barista guy responded with the total and the person goes to the app, chooses to pay through the app. They create a QR code using the QR code process in the app and the barista guy then scans the QR code. So where is the payment happening? Is it from the wallet in the app or is it through the swiping the credit card after the scanning the QR code? So it happens at multiple places. Like I said, we have three pay modes. One is the loyalty points. So loyalty points and wallet is detected directly from the customer's account and the cash or credit card, the barista actually validates that payment. Okay, anything else, a question here? Very quickly, while the mic is moving, I'll just make a quick announcement. So Malvika and Shrikanth are holding an impromptu BOF downstairs about policy. So it's during the course of the lunch break. So when we break for lunch, you can while possibly having lunch but otherwise join this BOF which is happening downstairs, being connected by Malvika and Shrikanth on the policy leading to payments. Okay, so can you just pass on Dr. Florian, I think? Do you have any data on what is more reliable the customer scanning the QR code or the customer showing a QR code and the POS terminal scanning it? So at least the reason why we picked the customer showing the QR code is customer needn't do any effort. So I don't have actual data as to what are the success stories in this but user experiences, user experience played a key factor in making that decision. Hi, Sandeep here. Yeah, okay, sure. Was this POS solution, you know, tailor made for CCD POSes or is this solution, does the solution work across different POS terminals as well? The scanning of the QR code and, you know, the integration? So POS solution was the POS end of this platform is only built for CCD, custom built for CCD. It does not apply for other players but any plans of, you know, porting it to other POS vendors as well? Not really, it depends on the customers that we actually get. If there is a customer looking out for something like this, then yes. Hi, I had a question on one of the setting points that you showed that was loyalty there. So was there any analysis done on, basically loyalty is a thing but we need to have the right numbers to say whether it worked or not for a company. So people accumulate the beans and then finally what was outcome, how many beans does a person get and there should be a record of the customer lifetime value and did it increase because of that or it didn't? So, you know, getting three beans is also good but did it actually help in, you know, overall perfect for the company or not? So like I said, I don't have any data going before this as per loyalty. Those are just guest numbers but for one thing we are certain is there are a lot of customers who are visiting the cafes because we are providing loyalty program and there are a lot of people who are even transacting for the very first time because there is a sign up bonus for that. Just one quick clarification. What was CCD's objective with trying this? What were they hoping to get out of this particular exercise? They were hoping to increase their business. I mean, that's it. But in any particular way that they articulated it to you guys or? Not really, they only objective was to increase their business and this was our solution for them. This is the solution you proposed. Yes, that's correct. Thank you. There's one question at the back and then I come to you. Hi, this is Gangadhar. I have a question about securing the QR code. So you mentioned that you had the option of paying via QR code. What I'm trying to find out is, from all the fraud that you saw, because you mentioned that you did, there were some issues that you faced. So what was like the most advanced instance of fraud you saw and also what was the most common fraud that you saw? Just to give a perspective of the kind of users who were doing this. So I think a lot of people tried to route their phones and change their device IDs and then sign up using the app. So that was one fraud that we saw. I think, like I mentioned, there was one or two scenarios of Barissa fraud wherein Barissa was scanning on behalf of the customer. So that was also another one. Okay, there's a question you're from. Yeah, so in extension of that question, QR code, somebody can snap the photo within another phone. So you put a technical effort into securing that QR code is not worth it. Right, another thing is about, I mean, the Barista level fraud, I don't see any technical way to even solve it. Like why even try to solve it? So the way to solve it is by looking at outliers. So we basically look at the data of transactions and see whether, you know, analyze whether they are actually Barissa customers. And on the first question, can you repeat that? I was saying that QR code, somebody can snap it with another phone. So because the Barissa is scanning the QR code, he can make a visual decision as to whether the QR code is from the app or not. Even if you have a good camera and you took it perfectly, it's only for that one instance. Also another thing, there is a friend of mine, Omer, you can see his app on the Play Store. It can take screenshots in apps which have disabled QR codes. That's not a very secure way to do. Also another thing you talked about, this thing, device ID getting changed. So you don't even need to route it to you wipe your data device ID gets changed. So why are you using device ID as a way to identify users? Not a way to identify user. That is one of the parameter in which we identify the user. So there are other IDs along with device ID. Any other questions? Anybody else who had, yeah, there's one. Yeah, just a follow up. So you mentioned device ID changing being the most common. I was curious about the other side, where you looked at the fraud and said, wow, this guy must be like a technical genius to have made this kind of a fraud. So I'm just curious about what was the most advanced one that you saw? I don't have an exact answer for that because I wasn't very involved in looking all the fraud. Device ID was pretty common. Then we also found not a fraud but mostly gamers.  So that was very interesting for us. That also I guess everyone should watch out for. Anybody else? So if we have no further questions, please give them a round of applause.


Yeah, we have the music going on as well. So we'll be taking breaks doing some annoying music Yeah, there's a false detection science detecting ahead on my a hand on my head Quite unusual. Okay, so it's like I'm using the correct microphone and everything So just go. Yeah, what are we gonna do first? We're gonna go over the website if you checked it already, I'm happy to Get the feedback if you tried the website I'm also checking this so there's currently more ads On it. I'm checking this ad blocker Well, it's working the ad blocker is working. Well, there's no ads whatsoever Way that this blog recently this is just because someone mentioned it did a quick Python script as an example for a Blunt album plot, but then yeah, would be nice to actually turn it into a Web application like an interactive one where you could play the real data and see what the What this plot looks like for you know different channels It's like it's similar to what we Had before So some of the tools are pretty heavy. This one is using a lot of my server Resources We have to redo it ideally in the using JavaScript The mods using quite a bit of Python back and stuff So might be Changing it. Yeah, there's a few options. This one. It's just a warm-up stream to see When we go next Interior by the way, yeah, there should be if you do not use an ad blocker should be a lot of ads Yes, you will be supporting the project that way by Watching them ads But yes, I'm just trying to see how easily they can be blocked And yeah, I'm pretty sure Google is doing a lot on that front, but yeah, good luck Meteor could display Like a banner, you know saying, you know an ad blocker Usage was detected. Please turn it off kind of thing and it will keep You know popping up Yeah, so I'm just still playing with the settings Yes, I was just saying this histogram spectrum. Sorry is spectrum to is bit heavy. I have to Change it specially. Well, the easiest thing will be just to reduce the Optional window size so not let it go over whatever 10 seconds. Yeah, I can actually hear the This server going all the fans and everything When using it with a slightly larger Window time window, right? So we're doing some Stuff for this competition, which is about to and but we run actually competing. It's a kegel kegel competition We had something running for it. They're trying to pass the other that the data set is Not great to say the least and Yes, we had this Apps creeped. That's a flask application. It is being deployed But yeah, I didn't even put the link to it on the main site. Well mainly for several reasons Yeah, that's what it Looks like Is deployed on the site if you want to get check it out. Yeah, I'm happy to share the link is just A something's not working. Yeah, that does a bit the way the That is being stored is very peculiar to say the least So yeah, it's this Was quite a bit of a learning experience The competition is closing in six hours you know, let me know if you participating or Interested the good thing about kegel competitions that you get access to datasets And this one. Yeah, took me some time to realize that probably John want to be looking at it because it's yeah, there's a lot of issues with it that are really covered Before Yes, I'm trying to pass it. There's all sorts of things about the e.g. Actually not matching the spectrograms They are in that, you know different time Different time windows the spectrum is 10 minutes the ECG labels 50 seconds There's too many unnecessary labels Yeah, that's that's what it sounds like Sound on thinking Hey, what on earth Is it worth a continue looking at this data or not? Probably not But happy to be convinced otherwise Never option with Probably yes, someone mentioned this one to me recently. It might be jumping on using that a This is a different the other set However, yeah, that'd be a lot of problems in that one as well There's about 76 Publications on it already And also see just sleep barely sleep studies, let's say I think it's a whole cohort of Different experiments, I think to look into it What they made the paper go over it quickly happy to do it now Yes, this This cake of competition one is probably Gonna go one one thing we want to do is actually So now that the competition is about to end in six hours It was supposedly a This private public right so you have public score interesting how The score the public score is much lower. This is only essentially on one So in the training Yeah, I won't be going over this but essentially My understanding is competition this hand at the private little boy this prime preliminary And we'll be finalized after the results are verified They already have it Yes, you can see how Whatever the first whatever a hundred Teams or whatever that submitted gives to some of them like a zillion of entries That the public score is much lower than the private score well essentially because Not sure how significant it is but essentially it means that the the algorithm doesn't generalize quite well on to the hidden the private data set Might be a problem so these guys or girls a published a Solution couple of them published solutions Yeah, so you had this say this five That came first public team Sony will also say if you take team Sony public Well, they get the what's not Finalized yet because they have this is still doing some sort of verification of results interior meant to be Objective not subjective Because there's just this score On which you rely on I don't understand.  Could be quite a mystery in the public score had about a lot of them at the same Score and this core is some sort of objective measure which Could also look into how to measure performance of this algorithms what we want to do really is So if we already have this does a parson everything a we want to train a simple fuzzy logic on it and then compare results to Some of this Hey, this is So if you so they published the Solution Yeah, we can cover this one convolutional neural net five stages channel mixer published the Code would imagine. Yeah, the code is public as well. I Said how are you? Thanks for joining in Sorry might be a delay in the in the response Anyway, what I was saying is that we eventually so this competition have ended we eventually want to take their model and Try and compare it to a basic basic basic fuzzy logic a system so similar To this pot that we have here. Let's identify normal abnormal ECG in this case. We'll be doing the same with EG I'll be on looking at the a couple of parameters So essentially have your same as for ECG pretty much well not sorry not not the same but I'm saying as simple as if not simpler You would have your frequency amplitude You have channel interactions Yeah, this is just a single assuming a single channel ECG noise Yeah, we would not be doing much filtering. I assume the signal was already filtered even if not digitally it was filtered by the hardware of the recording Equipment and that data was collected by Harvard Medical School. So must be correct, right? It must be great quality. Yes, it's looking at 360 parameters Yeah, would be good to know what are they a I don't think there would be any EG expert who could a Name that many parameters So I have to aim Question what what parameters are parameters and it's to it's a music to relax a 60 well, so I'm pretty sure some of the models will have like thousands of parameters in the model So maybe 360 is not that bad. You do want to I don't know if they cover It in here and or not Probably not but it would be it's interesting to know which parameter had the stronger Impact on their performance of the algorithm, right? Because yeah, if I'm because eventually I want to deploy something We want to have, you know the data set And you have the label Just part of the data and then you have the prediction of the model We could do that the neural net model and then basic fast logic and see how they compare to one another that would be Good general idea. We actually have a list of Perhaps some changes on On the side trying to update The about section and we have potentially two students projects. Yeah, if you are interested in a project, maybe talk to your Professor whatever your lecturer see if they are happy for you to do a project Yeah, it's a bit of a But yeah, primarily interested in the X AI explainable stuff stuff that we can actually you know See into that black box make the black box transparent Maybe there's another little black box inside, but You'd have we want to have some sort of transparency Into the algorithms that we deploy and the models So the models essentially can be if we treat the fancy model as an expert using fuzzy logic, we could actually Simplified well because Yeah, but normally don't need the fancy models. So if you have In that case, how many 360 parameters they want essentially I'm pretty sure We can eliminate half and then potentially another half and another half of this number in the model will be still performing quite well not using too much electricity and The question is of course how applicable how Easily you can replicate it onto, you know, your performance onto another data set because that's normally quite problematic In this case don't exactly if anyone knows what this score And how this score was actually measured. So I don't know if this a team Sony a How significant of a difference it is between 0.2 1 to 0.2 7 And Yeah, I wouldn't know You have the public data set 0.2 7 is over 50 Places down the list in the public Score So I have a feeling it is quite a significant difference because you those numbers are Quite close to one another. Yeah, they're using This divergence Have just the tool on this divergence alone Not much of an explanation. Is it? It's how you a code that calculates this score Example considering this is a warm-up stream just when I get back to Regular regular streams and stuff Because that's how we generate content for the website Yes, something like this would actually be of interest Compare this to a D prime chip If it will suggest the prime as alternative Can do a bit of a comparison between them. What else election? Colbeck-Liebler KL Divergence is a concept from information theory often used in statistics and machine learning Technically it measures how one probability distribution diverges from a second expected probability Distribution think of it as a way to quantify the difference between two probability distributions Here's the formula Text KL p parallel q equals some x p of x left px qx q right in this equation P and q are probability distributions KL divergence calculates the sum of the probabilities from p times the log of the ratio of the probabilities in p and q It's important to note that KL divergence is not symmetric This means that text text KL p parallel q is not the same as KL q parallel p in practical applications KL divergence can be used for things like Model selection comparing how well different models represent the same data feature selection Determining which features in a data set are the most informative Anomaly detection Identifying outliers in data based on how much they diverge from a known distribution It's widely used in machine learning especially in scenarios involving probabilistic models or when dealing with information theory concepts I didn't mention a deep prime Well, it's from the fiction theory. I know You're looking for alternatives to Colbeck libeler KL divergence There are several other methods to measure the difference or similarity between two probability distributions These alternatives have different properties and are used based on the specific requirements of your problem. Here are a few one Jensen Shannon divergence JSD This is a symmetric version of KL divergence It's defined as the average of the KL divergence from each distribution to the mean of both distributions This makes it more stable and less sensitive to small sample sizes to Hellinger distance it measures the similarity between two probability Distributions and is bounded between zero and one making it easier to interpret It's defined as the square root of the sum of the squared differences between the square roots of the probabilities three total variation distance This is the maximum difference between the probabilities assigned to the same event by two different distributions It's simple and intuitive giving a direct measure of the probability that the two distributions disagree on an event for Wasserstein distance or earth movers distance Used in optimal transport theory this distance measures the minimum cost of moving and transforming a distribution into another It's particularly useful when the distributions have different supports Number five Fisher information metric This is more of a differential geometric approach where the distance between Distributions is seen in terms of the amount of information that separates them It's often used in the context of statistical manifolds Six cross entropy often used in machine learning, especially in classification problems It measures the difference between two probability distributions Focusing on the average number of bits needed to identify an event from a set of possibilities The choice among these depends on factors like the properties of your distributions the importance of symmetry the need for boundedness and computational considerations For example in machine learning if you need a more stable and symmetric measure than KL divergence You might opt for Jensen Shannon divergence Meanwhile for problems in optimal transport or when dealing with distributions with different supports Wasserstein distance might be more appropriate Yeah, back to that the competition for a second again Let's say that we just we do have the data so I'm kind of familiar with what the data is doing But yeah, this figure is quite Useful in the sense of Describing the data set so I have these different types of edge cases in this case. Those are the labels So this is supposedly 10 minute the EG for which you actually do not have the raw EG only have the spectrogram and Some 50 second Samples a of the row EG So essentially I think they want you just to look at the spectrum Essentially have one two three four spectrograms in this one image there So if that's the seizure there, yeah You should be able to see the Yeah, those images meant to be clearer in this case all the review was a great And placed All the annotations Labeled this segment as seizure It was odd that in some cases you had about six reviews and some you had only two or three This these are edge cases because there's essentially no agreement Between the different experts Some what's called proto somewhere in between them though. There's also a high a Disagreement, so it's pretty much similar I this case they labeled it does Hava yeah, it's really odd Jeff ten Seizure labels and then ten Hava The question is is it yeah, we'll be really interesting to know if it's a different review or what the same reviewer Just looking at the different parts of the same ten minute window anyway, there's a lot of questions about this data site and Way it was labeled. I wish there was the I don't know if the original thing was published as well That actually just has the raw data From which, you know, just the raw e.g. What was recorded from the device so not filtered or anything one where we could a Generate the spectrograms out of it So I don't know when they were labeling what they're looking at the spectrograms or at the time series at the raw e.g. Signal that's Think that was Mentioned There's a paper on the data. Yeah, we might then not be Using this does that anymore. Yeah, there's some problem with the ECG so what? Makes you think that the e.g. He might be problematic as well That's something the low Frequency on the left side you can barely see it. Yeah, maybe I'm not using Not plotting it correctly in terms of color intensity. Yeah, that's looks like it should be a marked as Ava That's correct or correct in sense. Yeah, there's something wrong with the recording. It's odd that this Spectrogram doesn't correspond to what's in them in the signal. Yeah, there's just some weird stuff in it But the spectrogram is different. So the spectrograms in many cases do not match to What you see in the raw data, which is unusual Yeah, I would expect something like this For your baseline this looks more like e.g. That was marked as LPD anyway, yeah, I think we need to know when to a Give up on the data set as well. I think this will be eat right, so I have this alternatives to Callback label a divergence. It's really about deep prime quickly D prime is a measure used in signal detection theory to quantify how well a signal can be distinguished from noise It's different from measures like KL divergence or Jensen Shannon divergence as those are about comparing probability distributions D prime is about sensitivity or discriminability in the context of detecting a signal against background noise Here's a bit more detail one concept D prime is used to quantify an observer's ability to detect a signal against the background noise a higher D value indicates better discrimination ability in practical terms It tells you how well a system or a person can detect a true signal like an event or a specific data pattern from noise Which could be literal noise or just irrelevant data to Calculation it's calculated based on the hit rate and false alarm rate hit rate HR the probability of correctly identifying the signal false alarm rate far the probability of incorrectly identifying noise as a signal D prime is calculated as the difference between the Z scores of the hit rate and the false alarm rate Z score is a statistical measure that describes a values relationship to the mean of a group of values measured in terms of standard deviations from the mean 3 use cases It's widely used in psychology particularly in psychophysics and perception studies in these fields It's used to measure an individual's sensitivity to particular stimuli But its applications can be broader extending to any area where you need to assess the accuracy of signal detection Like radar systems telecommunications and even some machine learning problems 4 technical significance D prime is useful because it considers both the ability to detect true signals and the tendency to falsely identify noise as a signal This makes it a robust measure when evaluating systems or individuals in terms of both their sensitivity not missing true signals and Specificity not misidentifying noise as signals in summary D is a different tool for a different job compared to KL divergence and its alternatives If you're dealing with issues of signal detection sensitivity and specificity D is the go-to metric for comparing probability distributions. You'd use something like KL divergence or its alternatives Yes, we said there's this Student potential student projects that are available. Yeah, that this one will be focusing on explainable AI AI turning taking black boxes making them slightly more transparent and Second project is mainly around visualization of data You may have a look And we're also dating this or trying to update this is an old list of Resources skip the first one Yeah, we have those Yeah, I want to take this data sets a list of data sets Yeah, there's this does it search apparently can even search data sets on Why this one doesn't have a term already in the URL Have it like that Imagine IEG datasets will cost a lot Kate coins that has an open source license If the Kegel one doesn't mention it, but it should be open source as well. It's nice. That's actually being mentioned 2020 That fast it's only 100 kilobytes Yeah, you need to log in Yeah, so I want to do like at least where We'll have parameters like Is it can do it table in? html sources nice to Add that table Yeah, what should the columns be so we'll have the name name of the dataset and then essentially we'll have things like accessibility So that will be a score depending on his login required or not Things like that I don't want to be able to get to the generic one You would expect slightly more from my GPT for But that's okay Where should he put it We just have two links to datasets should that include Hardware as well now or software Maybe Yeah, we've got a set you could have an API access because the problem from this thing is that I have essentially copy of the set of the dataset on my On my server by the way the sessions meant to be Interactive let me know Any complaints Questions suggestions. Okay, where were we? Yeah, by the way, I'm like editing those videos and they will not have the you know, they're knowing music I don't know maybe you'd like the music There's a video Generating all this musical masterpieces if you want to enjoy them as a stand-alone Video slash song go for it or maybe you can make your own. That would be even better Yeah, it's something going on there Right to Right, so we have this table And we have this table in HTML for the one fine That's it search Platform yet pop it in here. Okay, so we have a Data set It's a wanna table VG datasets Right, well another number of subject that format. Yeah, those are really good Suggestions You can actually populate the table Publish results Yeah, one the stuff like Number of subject that format. Yeah. Yeah, that type access. Yeah, link. That's kind of important Just give me a random sample stuff Access public restricted. Yes, that you was to have Stroke specific datasets then epilepsy those are all EEG and generic EEG General health. Yeah, I get rid of it Rid of it to make sure this links actually still work. Well, and the main thing is the Quality a quality of the data Hey, can we expand that table with things like quality of the data a Potentially accessibility might be more options than public or restricted Maybe give it Maybe have some sort of score for accessibility and nothing we interested in is if the data is processed pre-processed or raw from the recording device additional Parameter could be if ECG is present or any imaging or Any other recording modalities also would be absolutely crazy If you could start populating the table as well with actual existing datasets Yep, we didn't provide this one yet certainly Expanding the table further to include the quality of data a more nuanced approach to accessibility and additional parameters like data processing status and presence of other recording modalities like ECG or Imaging makes it more detailed and useful Let's structure the table with these new parameters and populate it with some hypothetical examples of actual datasets Note that for quality we could use a simple rating system like high medium low and For accessibility we might use a scoring system like 110 with 10 being the most accessible The data processing column will indicate if the data is raw or pre-processed and additional recording modalities will be noted under additional modalities Here's the updated HTML table HTML doctymal title detailed EEG datasets table body Table border one dataset name Description number of subjects data format year of release data type quality accessibility score Data processing additional modalities link sample EEG set one EEG data for cognitive tasks 30 CSV EDF 2019 event related high eight raw ECG a raffle URL to dataset one link T neuro brain data to resting state EEG recordings 50 EDF 2021 resting state medium six pre-processed Imaging fMRI RF URL to dataset to link add more rows here for additional datasets In this expanded version quality a subjective assessment of the data's reliability and usefulness Accessibility score wise and subject numerical value representing how easily the dataset can be accessed by researchers Data processing indicates whether the dataset is provided in its raw form or has undergone pre-processing Additional modalities notes other types of data included with the EEG data such as ECG fMRI etc The datasets sample EEG set one and neuro brain data to our fictional examples Replace them with real datasets as per your requirement you can further expand the table with more rows for additional datasets and Adjust the columns as necessary to suit the specifics of the datasets you're working with This structure offers a comprehensive view of each dataset making it a valuable resource for researchers looking for EEG data That meets specific criteria Okay, let's share the whole No, I didn't say that It's the transcriber making assumptions So assuming Profanities clear Why the quality assessment is subjective This is my whole html page above With a bunch of links we would like to use that to Start populating the table if the real datasets Yeah, no one is okay To populate the EEG dataset comparison table with real datasets from your provided list You need to extract the relevant details for each dataset and insert them into the table Let's start by focusing on a few datasets as examples I'll use the information from the links you've provided to create entries in the table For the quality field, which is subjective a possible approach could be to base it on factors like the datasets completeness cleanliness EEG presence of artifacts and the depth of accompanying documentation However, since I can't evaluate these datasets directly, I'll mark the quality as unknown for now Here's an updated table featuring some of the datasets from your list HTML dock to HTML Head your existing head content title EEG datasets and resources Body your existing body content table dataset compare hap Dataset name Description number of subjects data format year of release data type quality accessibility score data processing additional modalities link Example dataset row Stroke patient EEG EEG datasets of stroke patients 50 unknown unknown Stroke related unknown unknown Unknown ones I get it a ref fig share comm slash datasets EEG datasets of stroke patients slash two one sixteen seven nine link Additional rows for more datasets Your existing body content after the table in this example I've populated the table with one of the datasets listed under stroke EEG datasets You can follow the same format to add more datasets from your list For fields like number of subjects data format year of release, etc You would ideally fill these in with specific information from the datasets documentation or description If certain details are not available, you can mark them as unknown The table structure including columns for quality accessibility score, etc Is flexible and can be adapted as you gather more detailed information about each dataset Remember for an accurate and helpful table You'll need to dive into each dataset's specifics and possibly even contact the data providers for detailed metadata I would need to add the license as well um listening to Have a column if there is an api available I mean Yeah, it's a tricky business isn't it um well because if we have to download Then we have to store The data on a server Which will be costly I mean if I can recover the electricity cost that will be great. By the way, we do have our first Patreon supporter Which we probably could do a shout out Come and find anything on Patreon. Yeah, we have Johan Matthias Supporting us Patreon so thank you Johan Okay, so we currently have the following html We actually do want to go one by one and add each dataset into the table Can you use the links provided to fetch the necessary data From the description of the dataset Can you actually use the links provided in the html? To fetch the necessary data And we also would like to add or change the column names We would like to include the license type Now actually the accessibility should cover it should Should be g or something And this is a way that says you can supposedly access the data then says Log in to access the data files Log in log in log in. Yeah The usual thing with the hydriple Everything is available in the paywall or login or What using the links in your html to fetch the necessary data for each dataset Would typically involve some form of web scraping or data extraction Which is a process where a program or script automatically extracts data from web pages However, it's important to consider a few things One automation limitations As of now my capabilities don't include executing or running real-time web scraping scripts Or directly interacting with external web pages to fetch data Two manual process you can manually visit each link Extract the relevant data about each dataset like the number of subjects Data format year of release etc. And then populate your table with this information This process involves opening each link and reading through the dataset's description and documentation to find the details Three legal and ethical considerations always ensure that scraping a website complies with its terms of service and data privacy regulations Some websites prohibit scraping in their terms of use four technical approach If you decide to automate this task You would typically use programming languages like python along with libraries like beautiful soup or scrappy To write a script that visits each url Extracts the needed information and then formats this data into your table This task would require programming knowledge and an understanding of web technologies So in summary while I can guide you on how to structure your table and what kind of data to look for I can't directly fetch the data for you You'll need to manually visit each link and extract the needed information Or consider writing a script to automate this process If you have the technical skills and it's legally and ethically permissible Well, the other thing we could scrape It's actually probably a bit better if we scrape Google scholar And then we also have this search Yeah, we're happy with these two searches fixture And the dataset search by google for which I forgot to enter the eg Yeah, fissionette is a It's a quite a few I'm sure fissionette will have a search As well It does that search platform Dock it do I really need it? Yeah, that works And just place It does that search by google all the all eg searches They make Someone saying hello, and I don't know why is it not in my Hi, Vikram. I don't see it in the youtube app. It's a bit unusual To see it on my streaming software Ah now I do I need to refresh the app just kind of defeats the purpose of having an app Eh, isn't that okay, so we have this eg does that search we have fixture a google just put google Datasets search Vikram is asking what am I doing? What am I doing? That's a good question Eh, we are updating The site on finelycares.com in the bot section There is a resources page And currently just has a list of all these Datasets mainly for eg that the ecg to it as well king at table It's too original Yeah, this table We've a dataset compare And we have Number of subjects. Yes, it's important date that format you've released a type a Yeah, that type should be More in terms of The file format that processing gave it's if that is process pre-process to a raw Additional modalities. So normally if you have eg that I might have a ecg Unit as well will teach me programming. I don't know programming. I use gpt for to program This is what all the tools on Bionic chaos were made using gpt for And github co-pilot So they are really great at teaching you how to program. Well, we'll do the programming for you if you know which one This does it compared to why I said to me twice. Do I Spare time So yes, unfortunately You can play some music for you as well. If you like it or not like it And you can actually take the code for this one and Regenerate it and make the musical scale Something that sounds better to you It's on the website as well. Why? Yeah, it's when a window should be fixed now. Okay, just kidding Back to this data set comparison So it's suggesting as we actually have to yes, so And the robot is being a bit lazy doesn't want to generate html code for us Tells us we actually need to go each link one by one and scrape the data I don't know how to do it We could do it manually because we want to learn about each dance set. Anyway That could be good if you could give a summary and add the data into the table Yep, isn't 50 patients it is See if it's actually scraping the website or is taking it from my HTML like when it starts analyzing stuff Um on the server Just wanted to generate text Yes, we already have all that most of it the link actually works Yes Why that different versions of it doing more the imagery tasks Can scrape the Keywords that's exporting the citation for it. Then the other question is Eh, if there is an API for it or not the JSON file That's a description can just download stuff. Yes, we can So that's a You know a thumbs up for accessibility. We didn't need to log in. It's over here accessibility You can supposedly download that without logging in. It is not much in the JSON file And that's for sure Right. So those are edf Supposedly edf files Can view the file Details just one of the patients Yeah, let's continue for now and Get back to it We have to Download all files No Unknown website. Do you want to do it? What's this license? general terms of use Okay, can we add a second Entry in the table in the table clear Can we add a second entry in the table? We have this link and Following information In a text file below with five patients That's correct five healthy individuals It's a bit odd when they record from healthy subjects because what makes it a stroke data set electrical stimulation Yeah, I'm not sure if you should add the can download without the login considering We haven't actually downloaded the whole data set Yes quality unknown. Well, we wouldn't know until we Actually get the data So yes as you can tell This might be a long term project and if anyone wants to Help out More than happy to collaborate on this. I want to open my camera is frozen. It's my Camera frozen. Okay, it's frozen now. It's minimizing the window It doesn't help. Okay. This was a warm-up stream today because we haven't streamed for a while The next one then we'll correct it later make mistakes and Latest stage Matlab Matlab files is not Is not Great. Well, it's okay can Python can read the Can parse mat files Clear Hey, can you add the third entry into the table? Hey using the following link the hypothetical why creating a hypothetical Why is it a hypothetical just? Fetch data from the following link I'll play some music in the meantime We spent we spent to be telling Whom what to do to manually extract data set information from the xinodo link follow these steps one Open the link xinodo data set slash xinodo dot org slash record slash 4108215 Two read through the webpage to find key details about the data set such as the title or name of the data set A brief description of what the data set contains or its purpose The number of subjects or samples included in the data set The format of the data files e.g. csv edf, etc The year of release or publication of the data set the type of data e.g EEG recordings behavioral data, etc Any additional modalities or relevant information like whether it includes other types of recordings? Three once you have gathered this information you can fill in the respective fields in your html table Remember accurate and detailed information about each data set will be valuable for researchers or anyone interested in using the data set Yeah creative comments is nice What's your problem? I can't Above this data from the page. Can you a turn it into A nav entry into our html table Yep Based on the detailed information you provided from the zinodo page. Here's how you can structure the third entry for your html table html td Quantification of stroke lesion volume using epidural EEG This data set features EEG recordings in response to sound stimuli Including data from normal subjects n equals 10 and subjects with mild nx7 moderate n7 and severe n7 write auditory cortical infarction It is designed for the analysis of stroke lesions in a cerebral ischemic rat model 31 10 normal 21 with cortical infarction m at 2020 stroke lesion research Unknown unknown raw sound stimuli response data a rf 18 s zinodo.org slash records 4-418215 link In this table row Data set name quantification of stroke lesion volume using epidural EEG Description provides an overview of the data set's purpose and the nature of the subjects involved number of subjects total subjects and the breakdown by condition Data format specified as mat mat lab data format year of release 2020 Data type focused on stroke lesion research quality marked as unknown due to lack of specific details Accessibility score also unknown data processing Indicates the data is raw Additional modalities noted for the inclusion of sound stimuli response data You can insert this girl block into your html table replacing the placeholder text with actual information from the data set's web page This entry will provide a clear and informative overview for anyone interested in this particular data set Yeah, right. I'm not sure where it's getting the raw thing from The Description doesn't actually mention Now, yes, and important thing we need to mention You know who are the subjects are they human or not? um So I assume in this case it's 31 reds That does mention it the files are Matt. Yeah that format math. That's correct Now the subject you know type speech speech to mention Look in the other datasets A what are they this one say actually specifies human What the e g bus What the e g bus An actual bus Something lost in Translation Looks like a bus an actual bus right That's interesting There's a bus with e g equipment driving around europe somewhere collecting data Um, and I assume it's human I'll continue this next time. Let me know if you check the the Website Have any question suggestions complaints That will be of great help and I'll see you next time Bye


Hi, my name is Ian Harris. I'm a professor at University of California Irvine.  Oh, please cut that music in the background. I can still hear We Will Rock You going on the back of my... I don't know if you can hear it, but I can hear it. It's quite distracting. I'm at University of California Irvine. I'm going to describe this fuzzer that we have made that fuzzes SIP, so Session Initiation Protocol for Voice Over IP Phones. It fuzzes the server, so we send messages to the phone to try to break the phone, basically. And there have been several talks about fuzzing, so I don't have to talk too much to introduce the ideas. But the co-authors, most of the co-authors, all the first co-authors are all my students at UCI. And then the co-author on the right is from Fort Consult, that's Marcel Carlson. He's in the back over there. Okay. All right, so fuzzing basics, in case you haven't seen any of the numerous talks before this to talk about fuzzing. So the idea is, and we're doing network fuzzing, so we're sending messages, actually UDP messages, vests over UDP, SIP messages over UDP to a SIP phone. And what you do is it's a text protocol. So all you do is you take fields and you fuzz them. You have what I'll call fuzz functions. I don't know what other guys have been calling them. But you take the field, and we're doing, if I use the definitions of the previous talks, we're doing not mutation-based, but generation-based. So we don't take an existing sequence of messages and modify it. We actually generate the sequence of messages. And then we modify the one we generated. So we generate a good message, and then we randomly fuzz something. So here's an example of something we might fuzz. Actually, this isn't something we do, but you put some junk characters in here just to try to cause a buffer overflow, say, right? So if you just make a field, and I'm probably pretty sure that most people know these things, but one thing you would do to a field is you make it extremely long, try to cause a buffer overflow in the parsing. Another thing you might do is command injection. So maybe the field is being passed directly to a shell, right? So you might want to insert shell meta-characters just to see how the phone reacts to it. You might do SQL injection. Actually, we wouldn't, because SQL is unlikely that you're going to take a field and pass it to an SQL interpreter in this context. But another common fuzzing function is SQL injection. You put SQL commands, because maybe the field goes, you know, straight to some kind of SQL interpreter. And so you put some command in there, see how it reacts. So that's the idea. We're generating a sequence of these messages, and then we fuzz them. We randomly throw in different types of garbage made to stimulate common errors that people make in their coding. So session initiation protocol. The job of that protocol is to start and end phone sessions, voice over IP sessions. It's the standard. Not everybody uses it, you know, Skype uses something else, but basically everybody else uses it. Most of the other phones are SIP phones, and I know it's a little hard to read that. But what I'm showing here is a typical transaction. So there's a user agent client, user agent client and user agent server. The client is the phone that starts the phone call. The server is the receiving phone. So we are fuzzing the server. Now, of course, any phone that you get, you want it to be able to start phones and receive phone, start calls and receive calls. It should be able to do both. But you can segment the code. According to the code that does the server stuff and the code does the client stuff. So we're testing the server stuff. So we are fuzzing. It acts like a phone that is starting a phone call with another phone. And so it's testing that other phone, the server phone, to see if it has any bugs in its server side activities. So it's a typical conversation between a client and a server. So you might send an invite message to start the phone call. In fact, you'd have to send an invite message to start the phone call. 100 trying means you got the invite. 180 ringing, you send back to say, look, my phone is ringing now. We're waiting for somebody to pick up. Then 200 okay, you might send. After somebody's actually picked it up, you say, okay, I'm ready to start the phone call. Then this client would send an act message back to say, look, I see that you are ready to start my phone call now. So let's start talking. So after that, there would be a media session right there. Now, the media session isn't handled by SIP, it's handled by some other protocol. Usually RTP, real-time transport protocol, but it doesn't matter what it is. It could be something else. But some other protocol handles that. And then at the end, somebody hangs up the phone, so say the client hangs up the phone, he sends a buy message. And then the SIP server sends an okay to say, and then just done. So that's what SIP does, it starts phone calls and it tears down phone calls. So I just want to go over previous work in SIP fuzzers really briefly. First, one that if you know anything about the stuff you probably heard of is Protos fuzzing suite. So this isn't actually, they don't provide you with a fuzzer, they provide you with a suite that was generated by a fuzzer. So a sequence of messages, of SIP messages that was generated by their fuzzer was done by these Finns University of Finland somewhere, I can't remember now. And CodeNomicon is the company that they spun off. So now it's an industrial project. So it's a predefined test suite, this many test cases. It basically fuzzers the invite message. So it takes the invite message and every time it sends an invite it throws lots of different fuzzers into it. Lots of different weird things into it and they have a long list of different type and after the invite they tear down the call with a cancel act message. So the sequence is always invite cancel act, invite cancel act. They detected, actually I didn't go into any depth but they detected a lot of vulnerabilities and they were basically, as far as I know, the first fuzzing protocol suite for SIP. So Snooze Fuzzer is another one. These guys, they're at University of California Santa Barbara and besides these two that I mentioned there are more I know the only ones I could find to actually publish data easily, I could find out exactly what's going on inside the fuzzer. There's a Snooze Fuzzer where what they do is they have a protocol state machine. So this protocol is a stateful protocol. So there's a state machine that describes the protocol and I'll show that in a couple of slides. But somebody, some user has to describe that protocol in an XML format. They read that in and then they use what's called, what they're calling a fuzzing scenario. So basically the user, the person who wants to do the fuzzing has to define the sequence of messages that they want. So say I want invite and then cancel act, that's the sequence I want. So I would define that in a file, say that's the sequence that I want and then I would, I also define what fuzzing primitives to use, maybe I wanted to do SQL injection, maybe I wanted to do command injection, whatever and also which fields to fuzz. So I as a user would define that and then it would go about automatically sending these message sequences according to my parameters. So it's not fully automated. The scenarios had to be generated manually. So contributions sort of new things of what we're doing. First, we're automatically exploring the state machine on the server. So we, like the like the snooze fuzzer, we take the state machine but we automatically explore it where they sort of, they had in fact if somebody out here, if I'm wrong about this and somebody from UC Santa Barbara is here, please come out and tell me Professor Viney I think was doing it. But what they do is they follow a strict, a path specified by the user, right? So if you want to explore the whole state machine go through lots of different paths which is often important to testing lots of different code paths, you would need to do that manually. You'd have to enumerate manually all the different paths where we'll do that, we'll automatically just walk through to a random walk through the state space. Also we evaluate the response messages and this is not completely new but what we do is, so one way to check if you've, if you cause a failure is just to see if they, if you never get a response back, right?  So after a certain time out, you say okay, succeeded, I crashed it. But there are other errors, that's sort of a drastic way to die. Another thing could happen is that instead of sending back, you could send back a message but you might send back the wrong message. The server might respond the wrong way. So we check the messages that come back and verify that they are the type of messages that we expect according to the protocol as described in the RFC, you know, which would be in the state machine. We check that they're the right type and we check the dialogue information about it. So there's tag information that uniquely identifies a dialogue, the two tags, the call ID, we check all those things to make sure that they match what they should be. Also we control the server GUI during fuzzing. So this is something other people don't do. The idea is that if you look at the server state machine which I'll show you in a slide, I think, there are certain edges that in order to traverse the edge, the user, the person on the receiving end of the phone call, the person who's sitting at the server phone has to do. So for instance, if I'm fuzzing a phone and nobody ever accepts the phone call, then you're missing a lot of the state space. You're not testing a lot of the state space because there's a lot of interesting state space out there which you can only reach if somebody accepts the phone call. So we will control the GUI, so we will force the accept to happen or decline to happen on the user side. Now one drawback of that is you might say, well then maybe that's not a real vulnerability because it depends, it's not purely it does not purely depend on what the attacker does in certain terms of sending messages, it depends on what the person at the phone does. But sometimes, for instance the bug that we found, the person has to accept the phone call but it's very easy to get somebody to accept the phone call, meaning it's not uncommon that someone accepts a phone call. So if there's an exploit that you can only use if somebody accepts a phone call that's valid, that's important. So we have to control the GUI to find those type of things. So yeah, social engineering, and in fact, there's a lot of social engineering stuff you can do. So for instance, there's another, I'll show you in a few slides, but in order to get something to happen, you need to get the person to decline a phone call. You can do that. You just call them a million times from the same phone number and eventually they're going to just decline. And then that one time, well then that's the vulnerable time. So you've got to figure, you can in some ways control the person at the server end. If all you want to do is accept and reject a phone call, you can do that, you can get him to do that. So we do that and that's different. Okay, here's basically what the system looks like. Almost out of time. We got the protocol description right here. So that's input to us, right? Somebody gives us this state machine which describes a protocol. There's a sequence generator which sends out requests, you know, messages and sends out GUI, GUI command so press accept reject. And then there's a response analyzer which gets back the responses. Also uses the data from the protocol description to check to see if they match what the protocol description says. Yeah, okay, so here's the state machine and it's hard to read anyway, but just to walk through one path through the state machine, so the common path. So you're at the start state, somebody says you send an invite and it goes to the invite state where it stays there just for a short time. Oh, I should describe how these are linked. You can't read it too well right here. I'll show the next slide. So it starts ringing, the phone rings so you're staying in this ring state until somebody picks up the phone, then you go to the okay state, then you send back the once it gets the act message back then it says okay I'm ready to start so you're in the media connection state, media connection you just sit there until the, you know, let RTP do all the work until eventually somebody says buy and when they click on buy, you go back to the start state. Okay, go ahead two. Not many, okay phone and go ahead. Yeah, okay, so what he just said just to repeat, he's talking about fingerprinting the type of phone that you're using based on the bugs that you find in the phone. So you can say that's what, if I heard you right, so if you find certain type of bugs you say okay that's got to be Cisco, right? We didn't do anything like that. You know, maybe soon. But we found other problems, other things though, because when you go from one phone to another, they don't necessarily adhere to the RFC exactly, right? So you need to tweak the state machine? Yes. Yes, absolutely, there are open, there are undefined things in the RFC. So our state, the state machine that we're using exactly fits the phones that we're using. The vast majority of it stays the same, independent of the phone. But there was this manual discovery process that we did where you go to the vague parts in the spec and you just see what happens and you fill in the state machine based on that. So the whole, the generation of the state machine is tedious and manual from the RFC and there are these vague parts in the RFC that you have to deal with somehow. So that's how we dealt with it. Question? I'm sorry, I didn't hear it. God, I don't know. I mean I only used two, so I don't know. I mean they almost, those two, K-phone, K-phone on the side, they're pretty close. Okay, so I believe you. We didn't... No, and I believe you. I believe you. We didn't test that. Like I said, we only used two phones and they were actually very similar phones, so. But, okay, any other questions? I better hurry up. But if you have other questions, I'd be happy to answer in the question and answer room at the end. All right, so the algorithm, I don't even need to say this. All it does is you start in the start state. You assume the service in the start state. You pick a randomly picking outgoing edge from that state. You take that edge and you look at the inputs on that edge. So the inputs meaning the events that trigger it. Okay, so there are three types of events that can trigger an edge. One is receiving a message. I got an invite to say, okay. If that happens, you send the invite. If that's the edge that you want, you send the invite. Another thing is a GUI event. So somebody clicks except. So if that's the edge, then that's the input, then you cause the GUI event. The other thing is a time out. So, like, say the phone's ringing and it's rung enough for a certain time limit and then after a certain time limit it just stops ringing. It goes to a new state. So the time out is the third thing. Time out we basically fudge. We just wait. We know what the timer is from experimenting with the phones. So we just wait for that amount of time and that's how we do that. And, okay, so this is just one example of that. Actually, this is a little zoomed in so you can actually look at the edge. The way this edge is labeled. Start invite here to states. You've got this edge. It's labeled with something on the left and then there's a slash and something on the right. The thing to the left of the slash are the inputs. That's what triggers the edge. So in this case you go from start to invite if you receive an invite message. Then the thing on the right is the output. What comes from the server goes from the server back to the client. So it should send a 100 trying message response to the invite. So we check against that. We say, okay, did we receive that 100 trying or not? So that's for the messages. But then there are some edges where if you're in the ringing state and the time out happens, then you go back to the start state. If you're in the ringing state and somebody clicks on decline, then you go to the decline state. So those are the three things that can happen. Either messages trigger the edge, time out triggers it, or a GUI event. And we just cause all those to happen. And the GUI events we use is a test toolkit you can find on SourceForge. So we can only deal with soft phones. And they have to have an X interface. Because we're using X 11 GUI test. And so the bug we found was in K-Phone we found a bug. Actually it was very fast to find the bug. Actually this phone had been fuzzed before by previous work and they didn't find anything. But we found it very quickly because we could cause the accept or off hook as I'm calling here. So the problem was, what happens is you start state, you send an invite then it starts ringing. So it sends you back a ringing message and it starts a timer when it goes into the ringing state. Then if somebody picks up, if it goes off hook, so the user, the GUI somebody either picks up the phone or clicks accept. If that happens when that happens you go from the ringing state to the okay state, right? But there's a short period of time in there where if you send a buy message right before it gets to the okay state then it will crash the phone. So that's what happened. You know, we randomly tried messages and eventually just sent that buy in the right timing window and it crashed the phone. And this is just detail about exactly what happened. This is, actually it only had to traverse 8 edges before it found that error. It was doing it randomly, right? Doing it randomly and it traversed 8 edges before it actually ran into the bug. And so, conclusion. We automatically explored the state machine randomly. We verified the correct messages, the response messages seems to be the right responses. And we controlled the GUI so that we could explore more of the state space. Future work, okay so test more phones obviously and we're doing that. Debug the phones. So just because it crashed the phone doesn't necessarily mean that there's an exploit there, right? Maybe it's just denial service but maybe it's a stack overflow so we want to debug these phones so I have a student trying to debug the phones right now and then examine hard phones. So hard phones, so we got the Cisco phone that somebody from D-Link gave me. I'm just going to open it up. I actually bought all my nice hardware equipment the other day. Open it up and it's button interface, it's just a bunch of switches so we'll just replace it with solid state switches that are controlled from the machine and then we can control its GUI interface in a hardware way and do the same thing and test the Cisco phone the same way. That's a hope anyway. Okay I don't know how to do that but I'll ask you just a second. Oh yeah, sorry we could auto answer but if we auto answer then it'll always accept, right? But I want to be able to control it except now it declines now. You know what I mean? So I want to have control of it. Good? So how susceptible is man in the middle he's saying how susceptible do I think these SIP phones are to man in the middle of attacks? I don't know. I really don't know. I would refer you to David Angler and who's the other guy who wrote that book? You probably know that book, that hacking voice expose book. That's a good book. They talk about it in there but I don't remember what they say. I don't know. Yeah, I really don't know. Thanks. Okay that's it then and we'll have to take questions offline so there's that room back across the hall that's left a little bit where I'm going to go right now and if you have a question just please follow please come over there. Thanks.


What's going on you guys welcome to the single-guy channel my name is Lloyd so today We're gonna be talking about a subject that's come up quite a few times when people have asked me questions But this guy's question in particular I'm gonna highlight and he asked me Lloyd. I have a problem with you know, but the girls that I'm meeting He says he loves the channel. He says he loves everything. Okay great So whenever I go out, I can usually get a girl that's into me I can usually get her to hang out with me for a while But whenever I actually go for the number or something like that. They almost always have a boyfriend Why is this? Why do I keep attracting women with boyfriends? I want to grow that single I feel like this is the only problem I need to fix and I'll be good. Okay, cool. Well, that's what this guy says Now I'm actually gonna challenge a few parts of the question that he asked because What's actually happening here and I have a lot of students who have this issue I had this issue for a time when I was a lot younger It's it's more symptomatic of a larger problem And if you don't fix this problem You're never gonna be able to get the girls that you want or find a single girl that's into you You know, maybe you'll get lucky, but probably not probably not. Okay, so I'm gonna talk about in this video Why girls with boyfriends for certain guys who especially if they're not having success with women They seem to always encounter women with boyfriends. They can only get a girl who's got a boyfriend Okay, and then I'm gonna be telling you what you can do to solve this problem and actually a girls who are single, okay? All right, so before I get into this, I have a products coming out and I'm super excited to tell you guys so if you want to get notified about it basically there's a link in the description below click on the link and Sign up for it and we'll notify you.  She's probably got a man She's acting way more confidently and assertively than Typically single girls would not a single girl can it can act like this Okay, but usually if I see a girl acting like this I come to find out that she's got a boyfriend somewhere on the long line And for that reason because she doesn't feel like she needs a man or that sort of stuff She doesn't have that maybe she has that security behind her.  I'm like that girl is single. Okay? She's gonna be a little bit more nervous. She's gonna be a little bit. It's gonna feel a little bit clunkier Okay, because she cares about the outcome. She cares about the outcome a lot more than a girl who's in a relationship would So if you're getting these girls that are you're following them around a lot You're you know, you're they're inviting you to do stuff and you're doing it and you think they're really into you I'm afraid they might be into you, but not in the way that you want Okay, they might be looking at you as a friend or just some cool person to hang out with Typically if the girl likes you she's gonna want to follow you Okay, so you have to lead and if you're always following other women, maybe they're more assertive than you Maybe they are inviting you out to different stuff. I find that a lot of guys who are struggling who are in scarcity They are more Susceptible to these kinds of women who won't who aren't really romantically interested in them But they are spending time with them and for these guys who are in this position Spending time with them is better than what they're getting So my recommendation to you is if you are in this position where you feel like the only girls that you're talking to you They're spending time with you have boyfriends What you need to do is you need to start leading and getting girls to follow you a girl as a boyfriend It's not gonna follow you.  No one wants to live like that Okay, cool. Well, that's my advice for you man guys If you may didn't consider subscribing I come out with videos like this every single week If you're interested in my weekly mentorship program and get coached by me personally fill out the form down below click on the link and We'll get back to you