Now we are moving ahead for the next session, so which is Elasticsearch. We have here with Dhanamji sir, he is a python developer and design and build intelligent and slick software suit at DirectEye. He is going to speak about Elasticsearch and how he leveraged his awesomeness in the massive infrastructure. I hand over the podium to him. Hello everybody. Yeah, that's a large crowd, so anyway, thank you. So good afternoon, all of you. Today I'll be talking about Elasticsearch. So the idea of this talk is to get you acquainted with Elasticsearch, which I'm really not an expert on, but I've been using for the last one and a half years at my job at DirectEye. I'm responsible for designing and building the central operations platform. And we leverage Elasticsearch in multiple ways, right? So I know when we start introducing a new database, the first question or we kind of end up in this slug fest over what my database is better than yours or I use Mongo, I use SQL or whatever I do. Okay. Guys, can we please settle down first? Let's settle down before we, so that we can start first. Yeah, cool. So the first questions that fanboys are going to put out there is, can it do X, Y, and Z? Is it open source? Is it scalable? Is it redundant? Does it support Hadoop and analytics? It's REST? Well, I just thought I'd put that out of the way and say, yes, it does most of those things. It comes with Hadoop connectors. It's based on Lucene, which is quite brilliant in its design and the way it's actually built and powers everything from simple search applications to the blue gene that played Jeopardy, IBM's blue gene, right? So it's incredibly powerful and it comes with a large subset of functions, but they are slightly hard to use and understand. And Elasticsearch kind of builds around that. The second question you're going to ask is, is it ready for production use? Is it web scale and blah, blah, blah, right? So for a project that came around in Feb 2010 and released the 1.0 release, this as recently as Feb 2014, it is incredibly stable. It is used in production. Wikipedia's search is now powered by Elasticsearch. They moved over from solar. Before Graphsearch came to Facebook, it's also powered by Elasticsearch. GitHub's code search is powered by Elasticsearch. Add DirectEye on large bunch of teams use Elasticsearch in different ways. Come to the next point, you need to understand something very fundamental, right? When you're using a NoSQL data store, you have to use it for what it's designed to do. You can't use a cannon to kill a fly, right? So what is it good for? Obviously, since it's called Elasticsearch, well, full text queries. The second thing is geo queries, right? If you wanted to find an aggregate metric of how many people here, like Thai food or are talking about angry birds on Twitter, and box that by geo grids, that's incredibly hard to do in your usual database. So that is incredibly simple here, big data, right? Elasticsearch is not asset compliant, and it does not support transactions, right? So obviously, you cannot use it for something like a banking transaction. But then you could use it for really large amounts of logs and data structures and stuff like that. It's incredible for aggregations. It's good for scripting, percolation, and well, these three points I'll cover in more detail. Time series data is probably the most used form of Elasticsearch, where you pair it with the Elk stack and you put all your logs into Kibana and LogStash. And Elasticsearch powers that. And they have a special section on auto completions, which I will show you, right? So when you're dealing with large clusters of distributed databases, ideally, it's a pain to set up. Even setting up SQL replication is not really the easiest things to do. If you're setting up something else, you probably want to set up something like ZooKeeper to synchronize your nodes. In Elasticsearch, the four commands you see is all you need to do. You can automate it using whatever automation tool you like. And you run the fourth command multiple number of times. And they'll discover themselves in that cluster and you have a really large cluster coming up. Hit any one of those nodes on port 9200 and you're dealing with the REST API. It's completely based off REST for any language but Java. In Java, if you're using the clients, you can talk over their custom via protocol. So if you want to handle really, really large loads, I would advise you to use Java. But if you're using Python or if you're using Ruby or whatever else. In fact, even if you're using JavaScript in your browser, you can talk to your database using JSON and REST, right? So it's important to understand this mapping. Now, the way we have been brought up and the way we started programming, we came up with, we understand SQL constructs really well and we try and map those when we start developing applications for a new SQL case, right? So it has a concept of indices, which can be analogous to your databases in SQL. It has types, the types are pretty interesting. Types would be like your definition of a table and a table would look something like this part over here, right? And it would give it structure, it would give it form. You could decide in settings, you could decide how many shards, how many replicas do you want, what is the purge rate. A lot of things that you probably don't want to know. And mapping would be where you describe what your JSON is about. If this field comes, how do I map it to a certain data type? Now, it's really interesting because Elasticsearch by design is schema-less. So the first time you pump in a JSON that says like the JSON over here, if I pump this in, so you can clearly see info is already a nested document and interest is an array type over here. So the first time you pumped in a document with these fields, you would automatically initialize it to an email type, to a string type, and do an integer type. This can cause problems because at times when you input float data, the mappings are not valid any longer and you can't actually do math on them. And a lot of the use case of Elasticsearch is to do a lot of calculations, right? So a document looks like this, which can be analogous to your SQL record. And the advantages are pretty apparent, right? In SQL, you would have a record and you would have a foreign key and another table. Essentially, the developer has to flatten out his data structure, right? And that is not necessarily the best of practices. You end up with those crazy outer joins and inner joins and stuff like that, which is honestly a pain at times, right? So what is full text search? How does this database work internally? And understanding this is pretty key. If I had a bunch of documents, what I would do is, if I were to invert the search on its head, right? And if I wanted to know what documents contain football and what documents contain job, what you would do is then you would, you could iterate through every document and look for the word football or job. But clearly that won't scale, that'll be too time consuming. So what you do is take the data that you have from here, you analyze it, split it up, get each word. And now each field, right? Each field, like you remember you had name and email. Name would now become a hash map where you would say Niners is a key and the values are now the documents it belongs to. So if I search for Niners or if I search for buildings, he'll tell me it belongs to document three, right? So well, that solves the problem of a single word. Now what about a phrase? If I wanted to know about football job for instance, right? I'm looking to coach a team. How do I do that? So what I do is just make a small modification and I store the document and the location as a tuple, right? Now I know football belongs to document A at index two and document B at index three. Now I look for job with the same analyzer and I find in document A it's at index ten. So clearly it does not meet football job. But in document B it's at index three, which is an offset of one, which was exactly what your query was. So turn it on its head and for people who do you use elastic search, if you don't want to store something, right? We are very used to storing is person an ideal field, right? In SQL and we set that to a false. Don't do that, just let it not be there. If it's not there, you can use a missing filter on it. It's just easier to do a set operation than a search operation, which is a login operation, right, at best. So use that. So breaking that down for you, I think it's really important to understand what happens to your data when it goes in, is you can define analyzers, right? You can define filters and analyzers and you can say turn every instance of ampersand into and, okay? Into A and D, the characters. Or you could come up with something more complicated using Hunspel, where you would say translate the words if they are in German into English and store them that way. Now, the user doesn't need to know that. The user is gonna input a recipe in, say, German. If I've translated that to English while inputting the data, and then I look for recipes of a fish. In my query, they'll still hit the right documents, right? And you can choose how to start this data and you can have both the German and the English meanings there. You can get rid of stop words. In most cases, you don't need stop words, or these words you wanna probably throw out. And another common use case is you build a system to scrape a lot of webpages. And you wanna get rid of all the HTML tags because that's spam. So that comes built in. So you actually put these together in a stream and that is how your data goes in. Now, that seems pretty simple, right? That's doable in any other database. The interesting bit is you can apply the same filters and analyzers on a query. So even though you store data in a certain way, you can enable your query to be analyzed in a different way, which will then map to your data that you stored. So if a user typed in something in Italian and you defined a Huntsville translation over there, it would just work, right? So this gives you incredible control over what data actually hits your index. And this is something really amazing, right? So let's get to the basic operations. What is inserting? And inserting would be as simple as that. You need not even have those indices created, right? So PyCon 14 would be the index here. Attendee would be your type, and one is the ID. And I send this as a put request or a rest. And this would get stored, he'll figure out the types and it'll work, right? Now I've got an attendee in, similarly any tweets about Bangalore. You use an analyzer, figure out all hashtags, map to tags, right? That's how you input your data. If you define mappings on them earlier, the mappings will be applied. And if it finds a mismatch, it'll give an error. It'll throw back 502 or whatever, right? So it's really that simple. And, okay, show our fans how many of you have heard of Lucene? Or worked with the Lucene syntax, right? So the simplest form of search, which is why you got down to this database in the first case, is to do queries, right? So the Lucene syntax looks something like this. When I say q is my query string parameter, which is a Lucene query. And I say tags, elastic search, and user dnscorsate. So it's a way of saying, give me all documents that have elastic search and this user. And you could use and or minus, like, that's the Lucene syntax, you can look up the link. So that's the simplest form of quick querying you could do on your browser. So all this time, you need not even have any library set up. You're just interacting with it on your browser. This is ideally what a query would look like. It'll be, it could be longer. We have queries that are about 300 lines long to do certain stuff. And it gives you incredible power. So just to give you an idea, I'm not gonna, I will explain this in a different way, right? So let's, you saw a filter part here, right? So filtered query and I say range timestamp. And I just say now minus 20 minutes to now. So give me data in the last 20 minutes. And this is the first part of where we actually get to learning what filters are. Filters are a kind of way to say that I do not want to query this data. First filter the data set for me on this filter and then run a query on this filter. And the idea of filters is they are incredibly fast. They're about 10x faster than queries. And obviously they're boolean only. If you want, boolean algebra can be done really quickly, right? So they're designed in a certain way that you don't really analyze them or query them, you just use boolean logic there. And they are scoped to your query. And also it has a very cool form of caching, right? So in most cases what you would do is you'd make a query. The cache would use a TTL and it would be cleaned up. In Elasticsearch it does it differently. Any document updates that happen after you've defined a filter, right?  And each filter gets updated automatically as documents change, right? So it's all caching in memory, which makes it really, really fast. And this is just an example to give you a geo distance. If you were trying to find all people talking about PyCon in 100 kilometer radius of this latitude and longitude of where we are, you'd just run this filter. And that would automatically take tweets that have lack longs within a 100 kilometer radius. Coming to queries, queries can be quite detailed and they can be fuzzy. The default is the must. Must indicates that it must match something, right? It must match this pattern. Must not, obviously, is self-explanatory. And should means it should compulsorily match it. Must is still fuzzy. It need not really match it. So what really happens in this case is when your data comes out, it gets scored with something called TFIDF that scores your document matching a query. And they are automatically scored according to how you write your query. So if you notice I'm running a multi-field match, I'm checking if the word Elasticsearch exists in either the tag or the message. I'm running this on a stream of tweets that are coming in, right? And I can give something called a query field boost on tags. That if someone has taken the pain to particularly tag Elasticsearch in his tweet, it's probably more relevant than someone talking about Elasticsearch randomly. So my query will automatically give it a 10x boost. And it will show up on top. Because search needs to be designed in a way that the results that are most relevant should show up on the top. How many of you have ever gone to the second or third page of Google? Or anywhere beyond the fifth page? Okay, well, very few. Obviously, you had a lot of time, right? So there's something called fields, which is essentially like in SQL, you'd say select star or you'd say select A or B, your tuple tables where you can select fields. And you can do a date desk that's basically selecting a sort order. So you'll get them sorted by that, but they still have the score relevance sort remaining on them. So it's like sorting inside buckets. One more insane thing it does is highlighting. So when you do a Google search and you get a result that says this document matched and you get a bunch of highlighted text telling you what that document is about. Now try implementing that by yourself in any other search database.  And here you can actually define how many words will show up in your result. And he'll give the highlighted term of why a certain document matched you. In what way? And he'll give you an analysis of how he did it, right? So it's that. One more thing they've done really well is suggestors, right? Now, in 80 to 90% of the cases where you are provided a search box, you know what the user is looking for. You're building a shopping website or you're building food, something like Zomato, right? When you start typing in TH, he's probably looking for thai or he's looking for shoes. Now, a lot of times you don't really need to hit a query at all. You just need to autocomplete his query and infer what he's talking about. So what these guys have done is all your documents, when you put them in, you can say that this is a type of string and it suggests an autocompletion. You can define its payloads. And it'll automatically start suggesting for these inputs. If someone starts typing nim hands and he is on your page and you are has geek, right? Then you would start automatically going to PyCon India at this point. Because that is what he's looking for, it's context. And these are basically built using finite state transitions. So they're really quick and you can have fuzziness in them. So even autocorrect works on them. So you really don't need to run searches. You'll already have a cash aid result for stuff talking about PyCon. You shouldn't be running a search in that case. So this is something really, really cool and a tip out there if your Suggestor is taking more than one millisecond, then you're probably doing it wrong, right? As I said, I'm gonna not talk about using elastic search in the conventional way. I'm gonna be covering topics or lesser known topics that you would like to discover and use and what you could take away from them. So this is scripting, right? Scripting is incredibly powerful. How many of you have scripted in Redis? You can write little Lua scripts in Redis. Now the idea that you can script a database is fairly impressive. In an ideal case, what you tend to do is you look for a record. You fetch your record. You update your record and you send an update request, correct? And especially this comes matters a lot in NoSQL because you don't have control over each row. Every time you update a document in most NoSQL databases, you actually just mark the old document for deletion and create a new document at the end of the index and then you merge it. Which is fairly inefficient. But it's the way they're designed for very obvious other purposes, right? So what you could do is you could use your documents as finite state machines. And what we actually do, this is some of our production code. That when we see an action on a particular page inside our infrared on the tool that I build, it automatically refers to the document that I'm dealing with and alters its fields and alters its stats without actually doing anything else. And it guarantees atomicity for me. So even if this document is being updated at the rate of say 100 requests a second, it'll guarantee these atomic step changes through going through that process. And that becomes really important when you're trying to keep track of thousands of servers, right? And also it takes lower bandwidth, obviously. Now you're just referring to a script. I call this on action. I'm referring to a script and I'm only passing the parameters that the script should execute. The script is already sitting on my database. So I don't need to get data and send data. It's just so much less work. It's just much easier. And you can write scripts in a variety of languages. The current default is Mevel. In the next release that's coming out next month, there's going to change to something called Groovy. But you have modules that allow you to script in JavaScript, in Python, in a bunch of other languages, pure plain Java too, if you want those kind of speeds. And the interesting part about scripting is you can actually write a script that will then be used on your query. So if you wanted to write a script that says, run this particular thing, that if the document contains these two words and it's by this user, update 10 more things and get me the result and run an aggregation on it, at query time it'll happen. So you can refer to scripts in your thing. This is perhaps my favorite part, right? I ran a query for, I ran a query over a stream. I revert Twitter's host pipe into my database. There's something called a river. I can revert entire Wikipedia as it changes into my database. And the database will do the pulling for me. And what I did here is I defined something called a split 10 minutes. Every 10 minutes, I want to date histogram based on the field time at an interval of 10 minutes. Well, I name my field that way. So this will take all the documents that entered and filter them in buckets of 10 minutes, right? 10 minute buckets. The second aggregation I used on them was term tags. What that will do is for each bucket, he will build for the buckets with every tag. Say you had tags of pizza, Python, you had tags of Cuban Park or whatever else. He'll create tag buckets out of those.  You still have documents of his entire tweet. Is he happy when he's talking about pizza? Is he enjoying the talk he's attending at PyCon, right? You'd write a little script and he would give you all those aggregations in buckets or, and you could actually graph them out using something like Angular or D3.js and plot them on a timeline. There's something even more very interesting, right? If I got all these tags and I use something on a top hits aggregation and I sub aggregated that with something called a geo grid, right? He would divide the world map into geo grids and you can define everything down from 1.6 centimeters to a 5,000 kilometer by 5,000 kilometer box. And he would give you for each box, what are the trending phrases every 10 minutes and you could zoom into a box and see how, what people in a particular region are thinking when they tweet. Or this is just an example of Twitter, right? So, and all this in one single query, right?  So, this is impossible with any other, I don't know of any other database you can do this with, perhaps solar, but that's about it. That's also, it's like a competitor, right? This is something that's really, really, really unique, right? Now 90% of your use cases, what you are traditionally used to in a database is you store a lot of data and then you run queries on it. Simple enough, that is the idea we have of a database. But if truth be told, you really don't need to do that. In most cases, if I developed a mobile application that learned about me, I see I was to develop Google now and I start, I want to start suggesting places around me, food around me, movies running in the mall next to me, right? You'll have a bunch of queries that you would run against each data entry you got and now scale that up to a billion people, it won't really work. So, what Elasticsearch has come up for this is something called percolators. In percolators, you store your queries, okay? With a query ID, you store your queries. You do not store your data against a particular index and then you run the data point that you are going to enter into the database against the percolator. And all the percolators that match this document, right? Say I set a query that said match tags Elasticsearch and I started showing in a lot of tweets about, from the global host. Every message that matches this query will return something like this. It matches one query on index tweets one, which is exactly the percolator ID that I registered with this. So if I had a thousand or a million, there are people using million queries at a time and they can be used for a trigger of alarms. They did this stock price. Suppose I have a way of entering stock price into my database. If stock price belonging to category pharma went up by 10 points in the last 10 minutes, then well, send an email to me, right? Now that would be something really hard to do with the amount of data flowing in a convention stock market. Here it would be really, really simple. So this is one of our favorite new things. We use it to tag events flowing into maintenance modes and stuff like that in our current architecture. That's percolation. Obviously it's Picon.  I've been talking about a database in rest all this time. In all honesty, there have been a bunch of iterations about how I would device a client library for Python or for any language for that matter. I use a lot of Ruby these days with the same Elasticsearch Ruby clients. So the idea is Elasticsearch is evolving extremely fast. They come out with a new release every two months and the release is actually worth upgrading to. I mean, I'm actually waiting for the next release because I want some features, some kind of new aggregators that they've built. So it's extremely fast. There is no real way to map a DSL from Python into these rest JSON queries. Even for Ruby, it's really hard when Ruby is actually much simpler to frame DSLs in. So why should you use a Python library and not use, so finally they said, let's ditch this idea. Let's build a basic library that would take a hash or a, right, Python calls it a dict, right? So you would take a dict and you would enter that dict as your parameter. And your basic stuff like index could be mapped into this name tuple, you would have doc types and IDs. So the idea is you get persistent connections. Now when you're building applications that do tens of thousands of queries every year, you don't want to open a new TCP connection and send requests every time. So your libraries would have a connection pool manager. You would have load balancing between, you could fire your queries remotely into any part of the cluster using a wild string or a round robin or whatever logic you choose to use. You'll have thread safety, it'll be easy to integrate your models into Django or Flask or whatever you plan to use. Also error handling, right? If you were to do a rest request, it will give you a 502. Now a 502 is not too useful. Did my database fail? Was the query invalid? Did the chart fail? Did I run out of memory? You know, what's the deal? So these libraries have basic error handling and exception handling built in, which is super, super useful. You would just put a try except loop around it. And failed connections, right? So connections to databases do fail. And you need to automatically retry. Now what the Python library currently does, the official Elastic Search Python library does is puts all this work that you don't want to deal with. You focus on building your application using this. But you use the same hash to send your queries across the server. So that is probably the only reason you should be. And in fact, you definitely should not use rest to talk to the server and use the library if you're building a proper, respectable application. Production gotcha, right? So we've been running it in production for a while now. And some of the problems I thought I'd share with you is firstly memory. It's written in Java and it's slightly dumb in its design for memory. What it does is in most databases, if you want to load data sets into memory, it first checks if I have enough. Elastic Search does not check. It just tries to fill up memory and goes OM. Even if you have 48 GB of memory. So an ideal use case would be, if you gave your Java VM 8 GB of memory, keep 8 GB spare because he will overwrite more into memory. And it's really simple to scale it across. You can score terabytes of data, just add more nodes. It's not really hard to do that. Manual sharding, right? As I told you, it is a no-SQL database that writes and does not really update. So what we had to do in one of our use cases where we had, we were actually using these state machines to toggle stuff, was we took a shard and put it in memory. And we named the shard with the different suffix. And it automatically, you can query indices using just commas and wildcards in your query strings. So we could modify stuff in memory and ram. Well, you don't need to do segment merges. So it just works. Networks always create your applications so that they can deal with network errors. Because network errors will happen. I mean, if you design applications with the assumption that everything will work, you're probably doing it wrong. Like, you have to assume it doesn't work, right? There's one concern that hasn't been solved in Elasticsearch yet, which is a slight pain. It's called the split brain. We've thankfully not faced it today. What it does is when you have a bunch of nodes in a cluster, then it auto-elects a master node where all the collection. So Elasticsearch essentially does internal map reducers. They call it scatter and gather. So you send a query, it scatters it to all shards and replicas, and then gathers the results and compiles them back, right? So it needs some kind of coordination. I mean, it could happen on any node, but there has to be one master for at least writes. So there's a configuration option called minimum master nodes. And a safe way to get past this at least 99% of the cases is keep this value as n by two plus one. If you have n nodes in your cluster, you should have, so if you have three nodes in your cluster, at least two should be up. So that your cluster doesn't go down. In Elasticsearch, it's designed to be run on cloud environments. So if I set up 100 nodes and I had enough nodes up, if I lost 20, I probably wouldn't be bothered if I had enough replicas. And you could just put them up and down as you please according to your elastic needs. And it would rebalance itself out. So that's something really amazing. Security wise, it comes with basic HTTP auth, as Kiran was telling you about. But you usually never expose your database to the outer world, right? Easy way of getting around this is use a proxy. There's something called ES proxy, it's a Node.js application, which can plug into any authentication system. And the best part about this is you have an Elasticsearch.js library that you put in your web browser. And you're actually talking to your database through JavaScript code using jQuery, which is pretty insane. You try doing that with other databases with as much ease. And yeah, your client should be designed to handle failure. Like nodes might go down and up. It should retry to one of the node, and it's not really hard to do that. So that's about it. And finally, the Elk stack, which is probably the most popular use case of Elasticsearch today, is you just, you take something like a Kafka broker, put all your logs, use LogStash and just chuck like loads and loads and loads of logs. Like millions of billions of entries we chuck into our Elasticsearch cluster every day. And you could run analysis on them. You could find out who's calling from where, who's doing what, what server, what kind of issues, what was the CPU history, what was essentially anything. You could use it for Facebook. You could use it for tweets, as I was giving an example. And I think you should go home and check out this database.  I'm done. Thank you. Any questions? Any questions? Yeah. What is the difference between MongoDB and Elasticsearch then? It provides all the features. They have all the features like aggregation, text search, geo-special indexes. So aggregations are not really the same. In MongoDB, aggregations actually write your map and reduce functions. No, they have aggregator. OK. Percolators, do you have percolators in Mongo? Yeah. Well, it's a new technology. I liked it. Mongo wasn't really working. I mean, as I told you, you're going to start these database walls where you'll say, hey, my database is better. So the idea was to introduce you to a new one, which really works well. Yeah. We have one more question. And you won't have. Let's see in a try. OK. If you were to build a chat app that had to do a lot of search across 1,000 users, so the use case of HipChat, right? So HipChat used, if you want to run full text search on all your logs, and obviously, the thousands of millions of users. Now what you would do is put your data into MongoDB. And there's no real way to search through all these documents in a really quick time in Mongo, right? So what they did as an implementation was they put Redis in front of it, used Mongo Redis connectors, and ran Lucene on top of it. And they took some, and they reduced their footprint by order of magnitude when they shifted to Elasticsearch. So that's, yeah. Is there a standard way to back up? Yeah, that is a very good question, once again. Is there a standard way to back up Elasticsearch DB? Yes. In the 1.x release, you can snapshot, you can back up. It works really well. Does it, the backup need to run all the nodes or one of the nodes? So in, firstly, in Elasticsearch, you really never need to back up your data.  It's designed to lose a bunch of nodes, right? So in most cases, your replication acts as your backup. But if you do want to do snapshotting backups, there's a plugin called, well, there is a, I'm not sure what it's called, but we ran custom scripts because we started before 1.x. So you could just back up your entire data there using a cron job, it's that simple. And it would work. Elasticsearch and solar, right? So solar is really good, but, well, there are certain things Elasticsearch is better at. It's, especially its aggregations, percolations. Also, the redundancy and stuff is slightly better. Well, you could use either to be honest, but I just like the DSL of Elasticsearch more. You could use solar. So there's a very good document on this, on why Wikipedia moved away from solar to Elasticsearch. Maybe they'll give you a better idea. Yeah. Okay, one announcement. Please submit the feedback forms at the registration desk. Okay? Yes. So ideally, logs are, have you seen a syslog ng or log stash? Right, so by default, what you would do is you would put it in a specific format and into a JSON object and send a put request. And that should map. If you have the mappings, he'll enforce the type criteria on them, like flot or int or whatever else. If you don't have type criteria, he'll assume it's a string and he'll store them and you can still query them, but you would lose stuff, right? So if I ended up storing an ISO 8061 timestamp as a string, because I had not declared it a string, for some reason you messed up a Z, you missed your TZ data, then you probably might infer it to be a string and then you can't run time range aggregations on it. So it'll be a good idea to set your schemas. I mean, even though it's not necessary. Go to, but what O is to use? What distribution do you use? Fedora, just add the repository, yum install Elasticsearch, hit your port 9200, you'll get started. It's really that simple. Do we have the support to configure the indexing relevance? Sorry? Let's say in a document, I want to index the words in the first para with higher relevance and index the words in the second para of the lower relevance so that I can define the order of the results being returned. So what you would ideally do is that would take place during an analysis phase where you would split up the paragraph and you would index para one as para one. And in your mapping, what you would say is copy para one to total corpus, right? So you'll have a copy in para one, you'll have a copy in corpus. So you can actually return either one of them as you please. And the thing to remember about Elasticsearch, mappings is they are dynamic. They're not dynamic, sorry. Once they are set, you cannot change them. You have to re-index your entire database. You can add mappings, but you cannot remove or change them. So using this, I can change the relevance of each word. Yes, you can. So one more thing that Elasticsearch does is smaller words get a higher score because smaller words are generally more important than larger words. I don't know, but... No TFIDF comes into the picture here. No TFIDF comes into the picture here. Of course, you can tweak the scoring algorithm. It's based in, you have five or six things. You have biation filters, you have bloom filters, you have TFIDF, you can select that. You can totally select your scoring logic. You can write a custom scoring logic. So that, we've done that in one place. We've written our custom scoring logic, sure. So the solution is both based on top of what you need. Yeah. Can we expect the same kind of throughput or is there any difference in terms of the technique? So I'll be honest with you, I've not really used Solr, right? But from what Benchmarks suggest, it's pretty much similar, but... It is done by Alucine at the end of the day, but those kind of things like filters, percolators, the level of analyzers, the number of tool sets you get for it, Marvel, they're monitoring dashboard called Marvel. The plugins are available for it. I think Elasticsearch wins in those fields. I was thinking in terms of data ingestion. It should be very similar. To be honest, I'm not really used Solr, so I think you should probably Google it. There's just some more different parsers like Edismax and Dismax. Sorry? Elasticsearch supports query parsers. Query? Parsers, like Solr does. Solr has a query syntax. Yeah, yeah, yeah. So it uses a query parser for... Yes, yes, yes. It uses various kind of parsers. Like Edismax and Dismax. Oh, yes, Dismax and Dismax, yeah. They're all basically mapping to your Lucene parameters, right? So, yeah, those are supported. I mean, I couldn't go into those details. It's really large. But you have a lot of configuration possibilities. And you have access to raw Java data types and all that. So... Sir, can you give me an example? For example, in Flipkart, if I'm searching for a headphone, if I've gone through laptop to headphones as a category, I want laptop headphones to be ranked higher. In a way, all 3.5 mm jacks can be listed. Yeah, so I don't think that will be handled by a database because there's no real way for your database to know what the user was, right? Your architecture would internally then do a query on your database, yeah. I can prioritize the tags. Yes, yes, you can give it a boost. You can give a query boost. You can give a term a query boost. You can do that. Yeah. Thank you. Thank you. Thank you, Jananjit, for the wonderful talk on Elastic. Now, we'll have a 15-meters break and get an assemble back here for a...


At the end of the call, she says, Andy, what time can you make it in? I say, Vanessa, hold up, I saw it online for $2,999, what's your best price? Go. Explain how this works. I'm calling her on a 2018 Ford F-150, I'm calling her. And she's like telling me all the information, what's your name? Vanessa. Vanessa, okay, I'm calling Vanessa, I'm calling the 2018 F-150, she's like, oh my God, Andy, this is the nicest truck in the country. She's giving me the spill, baby, the sizzle and the steak, I can hear it through the phone. The price is $2,999. At the end of the call, she says, Andy, what time can you make it in? I say, Vanessa, hold up, I saw it online for $2,999, what's your best price? Go. Hey guys, what's going on? It's Andy. A lot of you leave comments telling me that you need help. Do me a favor and take the best way to get a hold of me. Give me a text message right now,<PHONE_NUMBER>,<PHONE_NUMBER>, I'll help you with whatever you need.  Let's get back to the video. Let's see, Andy. Our survey is what is called an electric based pricing. Research showed that 90% of our customers don't want to have an overpriced, but they want the best price of credit. Vanessa. What's your best set here? Okay. You got haters? Yeah. You ever had somebody not believe in you? Yeah. Well, this is the chance for you to prove your haters wrong that what you want, you can go get it. Do you hear me? Let's go. Okay. Now listen to me. You've got to own your mindset, which I've been talking about for a long time. Right?  You see all these people? They're on your team and they're here to support you. So they want you to do good. Okay? So I just want to tell you, there's a lot of people that don't want you to do good and they can do good. Did you notice? Okay. What you say, how you say it, and do you believe in what you're saying? I don't want you to speak with your mouth. I want you to speak with your heart. Can you do that? Yeah. All right. Vanessa. I saw this online for $2,999. What is your best price? Sell these models. What's your best price? All right, Andy. See, our store does what it's called, the market-based pricing. Research shows that 90% of our customers don't want to have a little price, but they want the best price up front. This is why we use a very strict but accurate tools to find vehicles just like ours on the marketplace right now. And then those tools price out our vehicles to ensure that we're always priced below market. According to the stats on this vehicle right now, we're 85% to market, meaning we're 15% below market value. Not only are we great at marketing, but we're also good at marketing. We're also good at marketing. We're also good at marketing. We're also good at marketing. We're also good at marketing. We're also good at marketing. We're also good at marketing. We're good at marketing as well. We're good at marketing and we're good at marketing. We're good at marketing. We're good at marketing as well. We're good at marketing. We're not good at marketing because of our price. But that's our goal right now to market to market, meaning we're 15% below market value. Not only are we great at pricing, but we're also great in all high critical areas that are important to you and your family, like price, payment and trading. By the way, are you going to be trading in the vehicle today? Did you see the shift? She got there, she was back there, she's like, I want to see her in your mic. Okay, it's time to get out, guys. If you're going to play, play with everything you have. Most people can't get together. Can they? She got her together quick, didn't she? She's dangerous. Now watch. Do you think she came up with that on the spot or do you think she studied that? Okay. Who taught you that? Okay. If I taught it to her the first time you heard it, it was like, am I right? And then how did you learn it? Speaking of that, how did you learn it? I just studied. Every day I either wrote it down or I studied and I actually studied with my kids. My four-year-old even knows how to say it, you know. He tells me, he's like, Mom, what's your best surprise? He don't say prize, but he says best surprise. And we say it and we just, and my daughter, she's 12. She does the same exact thing. So I just continue to keep going. Hey guys, I just want to tell you the true one percenters you made it to the end of the video. Do me a favor, share it with the friend that wants to go to another level. Make sure you like the video, comment below so I know who you are. Set your notifications and then subscribe to the channel. We got daily self-training videos dropping. I'll see you soon.


Hello, my name is Dave Robbins, and I'm giving a talk called getting naughty on the can bus with the car hacking village badge So welcome to Las Vegas The home of the $30 cocktail these days kind of strange the badge here. I'll talk more about what that is, but this is the badge and Go to the next slide. What is the what are we going to talk about? I'm just gonna briefly talk about me, and I'm going to talk about the company that I run on the CEO and I'm gonna talk a bit a little bit about the badge So just to give you the basics. So if you don't really want a lot of detail, you can just skip out after that We're gonna talk about the pipe eco, which is the the CPU or the board computer board. That's used in the badge We're gonna talk briefly about the can bus Which is the technology used for networking and many industrial? Machines including vehicles we're gonna talk about How we could generate and receive can data and then we're gonna talk about a couple methods that we can do Some of them that you really wasn't easy to do before this this new part I'll show you the Software that comes with the badge. There's actually a serial console, and then I'll talk about Other software for the badge, and then hopefully some conclusions are related to that I've been doing this kind of stuff for over 30 years It's kind of my passion computers electronics. I originally was a hardware person But there's just so much software with microcontrollers. You had to learn that and I did every kind of software Starting out with assembly code back in the day You know C code Python I did GPU programming, which is really interesting I did FPGA programming embedded software when it's programming Etc. So and we do all that at our work I started in traffic control systems In my college dorm room and in Arbor, Michigan in 1994 and we've been just supporting the vehicle electronics Industry in the Detroit area and then eventually globally And then I am a dog person, but I'm too busy to have a dog. So I also do a part-time dog watcher So now I'll talk about intrapid control systems We're a company. We were founded almost 30 years ago We provide test tools for automotive world around the can hacking a lot related to can but we do way more than can Automotive ethernet is getting bigger than can now for future vehicles Intrapid is a global company around 200 people offices in Asia three in China Japan Korea India, and then Europe. We have Germany and in college school in the UK We are a partner for the OEM So a lot of the equipment we developed is basically on their specification and we're kind of like farther Our new products are stuff that will appear in at in production like three to five years out So some of our new products like ethernet and things like that In car hacking village, you know 2026 That you know that technology will be working on that I'm also looking to combine with the packet hack a million village because it's really a lot of the same technology We've been working on cyber security tools for a long time foundational OEMs wanted to be able to benchmark other companies, so we developed a lot of tools for benchmarking Including you know, just figuring out how cars work So that we've been doing that for a long time and that happens to have a use case and cyber security Just figuring out how a system works and its flaws The aftermarket is a similar type of industry where people want to install a new radio into Take out the factory radio in the can bus is involved So they have to figure out how to make all that work You might have heard some of our main products a vehicle spy. It's common In the industry value can is our low-cost can can FD and then Neobuy interfaces which we continue to evolve. So the latest one the Neobuy fire three has 16 can FD channels and in two gigabyte ethernets. So Interpret is way more than can bus. We do every protocol in the vehicle Can bus is the most fun because it's it's it's you know, it's not too hard to learn and you can get a lot done a lot of hacking hacking done with it Okay, I'm gonna give a demonstration of the badge right now So I'm gonna switch to the full screen okay, so what we have here is a Most interesting part here is this is the pie pico right here and this is the CPU and The USB interface for the badge the pie pico has GPIO around it and we're using some of the GPIO And we actually connect it to a can physical layer here this can't physical layer. It's from microchip It's capable can FD as well. Although we're not doing any can FD with the badge that's basically something that Based on the talk you could do can FD based on What I talked about there's two buttons And then there's two switches one of the switch turns on and off the power for the the the battery The battery is just used to kind of flash an LED while it's not plugged in the USC But most of the badge features only run on USB So so basically we have some just regular LEDs that are just powered off USB and then these are actually controlled by the CPU One of the switches doesn't do anything The there's been there's two buttons and there's you can hear that there's a buzzer on the back The badge itself has a soap screen that kind of explains a lot And then there's a QR code that links to our website with has more information Just a heads up some of the soap screen has a rata in terms of what pins connected to what? So for example the buzzer it says GP 19. It's actually GP 20. So if you do Sit down and write some software for it Python or C++ or basic or whatever your poison is go ahead and do that What what it does with it when you just plug it in is you can push the button and it'll start countdown And then what that's going to do is it's going to launch a kill on the can bus So it's actually going to generate can messages out of this connector Which you would wire in which I'll show a wiring setup. I got later So just to give you an example, there's there's if you hold one button It does one kill another button does another kill if you hold both it does a Third kill and I'll explain what those kills are later, but we'll just go ahead and try one Now you can see this LED illuminate. This is the actual can bus activity. So we're sending out Can messages right now and the idea is that we're either doing a Don't have a denial of service or sending incorrect frames And there's actually three different pills, which I'll show you later, but this is Never gonna let you talk.  So use that your own risk or run a car I don't even know but the first time I plugged into like a On an old Ford not with this badge, but I just had the can bus Speed rom and the whole car just shut down. I had to just disconnect the battery and reset the whole car So it's so you know your mileage will vary I'm gonna interested to see what people in the community plug this thing into So that's the basics of the badge. I'll demonstrate the USB software a lot later So this right here are some screenshots of the USB console. So if you plug this in To Linux or Mac or whatever and you open up a serial console the TTY s s Whatever whatever USB the serial port it is This will show up and allow you to set up configure the badge the kills with the buttons are predefined kills And then the mess the menus allow you to configure out exactly what you want to do first. Let's talk about the pie pico What is the raspberry pie pico? traditionally raspberry pi Foundation has been building single-bit board computers that are based on Linux and then the raspberry Pi pico is the first single-board computer that is just a microcontroller. So the microcontroller runs almost no last or little or no I'll ask It's lower cost lower power. It's a lot different than the raspberry pi now. You might hear something called RP 2040 To clear the confusion is the raspberry pi pico is the name for the single board computer The microprocessor or the microcontroller on the board is the RP 2040 Now with the raspberry pi they didn't sell the processor independently It's just you buy the board and then you can't buy the chip and make your own board But that's different with the raspberry pi pico Now the pico board costs between four and six dollars and they just released a vert the six dollar version less I think last month and it includes Wi-Fi so you can do Wi-Fi type Projects with this that connect to an access point or act like an access point. So it's pretty cool For six dollars how much you get? The board itself is programmed and it's support supported by raspberry pi and C and C++ and Python So there's a I believe a micro Python. There's yeah micro Python port for raspberry pi or pico Which you can write You know quickly write strips The board it has the raspberry pi community behind it So I think that's what's most interesting to me is that getting support and finding people other people that did things The initial release was last year and I think maybe it didn't get as much attention as it should have Because it didn't have wireless. So now that it has wireless will be really interesting to see how this board does take off The tools are all free. So like if you've been in the embedded systems area for a long time It's tools are a pain to get or very expensive. I mean at work. We've paid You know $10,000 for a C compiler because you know for the power PC because that's that was all that was available at the time But these tools are all free GCC Visual studio code and then even a hardware debugger You can use a pie pico to beat debug another pie pico So you can use the $4 pie pico to debug the other one, which you got to have a debugger if you're doing any serious work but the most Interesting feature of this board and the only reason I learned about it is because it's available Availability in any type of microprocessor Chips right now. It's so important and he right now at least right now You can you can get these and thousands of quantities For projects so like at the for the badge we we bought reels of these Months and months ago, so we made sure that we had all the parts we needed the the the badge hosts the RP 1210 microcontroller which is When it first came out, I looked at it and I just dismissed it as another me too product But since it was hard for us to source while last year all we did was redesign our project products at work for Availability of components, which was not not the most interesting work, but we learned a lot so this year I took a look at what was available and this part is super available and It's 70 cents if you buy the full reel and you can buy it directly from Raspberry Pi so I took a look at it and I was Pretty impressed with what it what it had first of all Cortex M zero who cares but there's a new trend to have like microcontrollers that have multi cores And it opens up some interesting applications including the one that I'm talking about today But the most I think breakthrough thing that this part has is it has eight bit bang Processors called a PIO and I feel like there's so many problems in my career And about it that are solved with this. It's it's just a total breakthrough And it's a game changer especially on such a part that's so low cost and low power which before you really only option was FPGA which is a very complicated very complicated tools Very hungry power hungry very expensive. So it's kind of it's a game changer for the industry Another thing is it's very fast 125 megahertz and that's a clock rate of eight nanoseconds So you can do a lot and you know things like bit bang Canbus at 500 kilohertz is our top is well, I'm not gonna give it away, but it could be possible There are videos online where people overclocked it to over 300 megahertz. So feel free to Check those out and then use it, you know use that your own Right, you know use that your own Experience so if you if you overclock it and you test it and everything it works, that's great so if you want a bit bang things that You know 300 megahertz would be like three three or four nanoseconds or you know under four nanoseconds Go ahead and do that It has USB which is really useful for boot loading and Connecting to a PC. It's 12 megabit which is Disappointing because the thing is is that the PIO engine can move so much data. It just doesn't have anywhere to go I can't go up to a PC But it's a 70 cent part. So that's that it's really great at PWM channels Using the PIO and the PWM you can generate a PWM in every pin of this device so like things like Robotics applications where you're controlling servos or you know You have a bunch of LEDs that you want to dim and you know full-color LEDs It's really awesome and it's got like the standard digital peripherals and then the pin mapping is real flexible So it's really useful when you're designing boards It does not have onboard flash So it actually has to load code from a serial flash at Udo Which is not so great if you're used to microcontroller and there's no code protection So like if you design something with this, it would be very easy to copy and Voila This part is on the duck on 30 badge. So this part right here is RP 2040. So I'm not really sure why they chose it. I'm really interested to learn But I would guess having it available is probably maybe a good reason And I'm really excited to learn more about this badge this week And of course on the back of the badge is the Raspberry Pi Pico and it's just soldered down and You can we actually place this with our machines or have some team machine So it came on a reel. It's placed right on the board And our guys did a really good job where the pins if you look at the pin hole pins They are totally open so you can put a pin header through the defcon badge and you can use the rest of the IO So think outside the badge use it, you know, write some code use the other pins to do stuff And have fun. All right, so now I'm going to talk a little bit about can bus It's very popular and it's kind of a huge focus of the car hacking village the can bus also Used in many other purpose so like a boat if you have a boat there's something called Nima I think 2000 and That uses the can bus. So we were talking about like all the different ways you We can make different badges like we'd have boat kill all these on airplanes You have airplane kill which no one like that No one like that name because we don't you know, we don't want to get picked up by the NSA at defcon But anyway can buses use everywhere and all these attacks would work on that What is the basic when was can bus created who cares? I mean really who cares It's been around forever. It's gonna be around Forever more to so I don't listen to this when you're on your deathbed You're gonna look over at at that machine keeping you alive, and it's probably running can bus so And it's it's just a great inexpensive cheap protocol and it's it's very robust and of course when people say what does that mean? It's like real you can really configure it all around that still probably will work And which is a terrible way to design a system ideally you design the system correctly And then whatever happens in the field you know it still works But I've seen systems over the years where they just take that robustness and use it as a feature Why did they made the can bus who cares who you know go find go find another video online to learn about why they created it But we're gonna talk about the what and how and this is just a basic Can network where we have two wires can high and can low I Like to think that we invented the colors for can high and can low, but I just remembered You know at the beginning we always said yellow is like the sun and green is like the grass, but I don't it's been so long I don't remember but each end of the network is terminated so Can needs a termination because it only drives the can bus In one direction, and then it relies on the network capacitance to pull it back down So without that you your your bits will be all messed up And we actually have a feature in our product where we generate a pulse on can and based on the The fall time of the transceiver back to assess if you can automatically determine if the network is terminated What does the bus actually look like? So if you connected an oscilloscope to the bus there are two signals can high and can low And can high whenever there's a there's a dominant bit it goes high and can low goes low So it's a differential bus So the receiver actually measures the difference between the two lines and the difference is really important because in a vehicle The ground is used to carry current for all the different parts of the chassis So what happens is whenever currents are flowing through the ground? They'll be an IR drop which will change the voltage at the ECU and if it goes, you know If you're using the the ground as a reference You can have some real interesting problems like when only certain motors go off you lose communication So having a differential network is a huge benefit Now how do we get to this protocol on the badge? Sorry the the the thigh on the badge this little part That cost, you know between 30 cents and in the last year people are selling them for $8 But that part for microchip it generated it takes in a digital logic signal and it generates the actual physical layer So with the pipe you go will be generating this digital signal and the thigh will generate that What's important when you configure the network now? When you first look at can and they show you this this diagram It's just like you immediately have like you need like aspirin or you know, ibuprofen immediately just gives you a headache And like who cares about time and time and whatever The main important thing about this is the bit time and that's the length of the actual bit and Then the sample point those are the two things that care So if you got those correct The can bust the work now the sample point is when the digital logic actually looks at the bit to determine if it's One or zero, okay? so What happens is you could have it wrong, but then uncertain that works if you wire it a certain way You won't you'll get errors and there's actually attack by by Candice labs that you can do with the badge called a Janice attack which messes with this Which I'll have more information about that later So what do we want to do with the badge? We want to generate can boss frames So now I'm going to talk about how we're gonna do that What's really awesome, and I probably didn't realize this at the start when I looked at this part and having two cpu cores and embedded system is is really useful because Normally bit banging is just not possible because you have a you have a core and it's like it's got to handle usb you know or it has to You know process some data or something like that and you can kind of do some stuff with interrupts but it's it's a it's never really deterministic and You know you can stop everything in bit bang and that does work We've did some stuff with that, but if you do if you have to bit bang for too long It kind of you know things stop working. So for example like USB on this on this badge It's interrupt driven. So if you don't service it It will actually disconnect from USB. So it's kind of annoying when you're using a debugger But having a separate cpu core allows you to Do these bit bit banging and still have your full application running So you'll notice when I sent the cam messages with the badge It sent the cam messages and the LEDs flash and everything else still works. It's pretty cool Now you can have a second core, but like how does it use how do they work together and? With the RP 2040 they did a really good job of allowing it to operate independently if you use it correctly so there's Essential for bit banging is you want to be able to be deterministic that means when you write this software it's exactly going to have that that many cycles and that time and everything and They did so so the way you do it on this part is they have a feature called single cycle IO Which I'll explain There's something called a bus matrix With a SRAM and then they have five O's between the cores so they can communicate So I'll talk a little bit about that and all this all these diagrams are from the RP 2040 data sheet so you can Learn more. It's it's really fun. Now. There's other CPUs that have dual cores out there So so nxp has them microchip has them I would say I can't tell you if they're done dirt terministically Although microchip does have a DS pic core which I used to program and I love but it's only 16 bit They have a dual core now that runs at 10 nanoseconds Here we go single cycle IO so normally with an ARM processor What happens is they they have load and store instructions meaning? That's the only way memory is changed. It's through these instructions Which require access to the bus and they all they're always multi-cycle so you? To do a load it will have to go on the bus And then and then the IO itself is memory map so it will be a specific address They have to go to the change Now with a cortex M0 plus They have this feature called single cycle IO so The logic in the core actually looks at what address that you're running to and if it happens to be a specific address It will it can change that logic with one cycle so the load and store instructions for those address Take part eight nanoseconds So you can actually there's a there's a register that will allow you to toggle a GP IO and You could just have inline code that will that you could generate Clock pulse every eight nanoseconds so you can you can bit-paying Things you know up to 65 62 megahertz, you know 62 and a half or half the clock rate It's a really cool feature for a bit-paying now Do you really need this to do can because like if you look at the eight nanoseconds compared to like a standard can bit time at 500 kilobits, which is Two microseconds, that's you know, that's you need single cycle out for that I don't think so, but like if you want to if you want to do a faster protocol like maybe can D Then you want to have is you know, you want to be able to save every instruction Now this is the other part so the CPU just exists in the part and it needs access to RAM and it needs access to IO so the single cycle IO takes care of Of the IO now the other part is you need RAM and you need RAM for your stack you need RAM for your code because this is this is a processor that You know, it's there's no flash or anything and So when you go to access the RAM If you have to share the RAM with another core, that's a problem because Defining on priorities who gets it in cycle So the trick is is they created this thing called the bus matrix Which is actually kind of hard to see right here, but it's basically a logic a logic device that There's there's certain masters which in this case is there's three There's the the core one or core zero, which is the first core. It gets kind of confusing and then there's second core core one And then DMA and they all access they go into this little box called the matrix and then there's all these Things that hang off of it and the beauty of the matrix is as long as you're not accessing the same Preferral at the same time or the or this I don't know what the best just call it prefer all that there will be no weights for that that Resource so for example if core Core zero is talking to this s-ram and then core one is talking to another s-ram There will that there won't be any blocking So that allows the core to run Allows the core to run without any blocking so it gives you a deterministic clock rate for the core Now it's important to know if you use this part that depending on what address you asked us these s-rams They're striped so if like you might take one address, but the next address will be on this s-ram It's actually a cool thing because it allows if you're running code out of the s-ram from both parts They'll they'll they'll they'll not always block each other You know, so that's interesting But like if you're going for determinism, you can't use that striped feature So in the code that I wrote uses one of the one of the 4k s-ram so all the code and the stack I'll go in there Okay, so this is basically summary of what I just said The you use this independent s-ram We're calling it they call it scratch x actually in the code and then the second core another thing I want to mention that I might have skipped is On this first part here There's actually a 5-0 and this allows you to communicate between the cores and that does that's also deterministic so Core 1 there's this 5-0 that goes in one direction and then core 0 has the opposite So core 0 can write to the 5-0 and not block and then core 1 can read that 5-0 You can check it if something's there and use it or not and that's how we actually trigger the can Transmission so I'm going to stop and I'm going to do a code demo of of the CPU fit bang. So This is this is going to be visual studio itself So I'm back and I'm going to show the code for generating the can message This is visual studio 22 and I have a plug-in called visual gdb which has some tools for It has some tools for Using visual studio 20 22. Actually all the previous versions with for embedded process. It's pretty good I'm having some issues with the debugger between the two parts, but let's go ahead and look at this So this this this particular file is called can message builder and This is The But I'll show you the first step. So basically what comes happens is there's an init function here And it looks at what the type of kill it is.  So we're going to look at that So there's this object called ob can which is the actual it takes the can message and creates ones and zeros based on the information so we're loading in the can ID The different pieces of the extended frame the data and then there's causes function called canned bits So what it does is it goes through here, and it checks the bits and then adds them to an array and I tried to do can fd, but it's not supported But if you it just goes through the different fields of the can protocol and generates the bits and if you go Look at this function It's kind of interesting it actually does the CRC and the bit stuffing of can which I'm not going to explain in This video, but that's a key part of key parts of the can protocol so that generates the actual ones and zeros and then We come back to where we started. There's this build can message and What we have here is there's this class called arm So normally you would use an assembler, which is a separate source file but I've written a C++ class that dynamically generates the Cortex M zero instructions now There's only like 30 instructions in the Cortex M zero Which is why I thought it wouldn't be hard to just dynamically generate messages and it wasn't too bad The the instructions have derivatives, so I only really implemented the ones I'm using So it probably grow over time. What's really cool about the Cortex M zero is the Instructions are binary compatible with all the Cortex M. So M stands for microcontrollers. So Some of the more popular ones are Cortex M three or four That have like more advanced instructions and floating point and M seven as well. They're out there So this code technically will work on those processors. Of course the delays are different Because they're designed differently. So some of them have like dedicated RAM or they have a different number of stages and pipelines So things take longer. Anyway What we're doing the first thing we're doing is we're initializing the program at the address of that particular S-RAM Now the key part about a Cortex M Part is the addresses that you call have to have a one in them otherwise they you get a hard fault because one is The the the first bit says it's a thumb or arm instruction and the Cortex M only supports Thumb so so that's a weird thing. I had to learn lots of lots of crashing and the hard to debug One of the problems I have with this with my setup right now This is the second core is really difficult to debug So there's the tools don't that I have and I looked for I haven't seen any really great Solution to debug both cores, which is kind of a bummer So anyway, we're basically generating arm assembly here move immediate logical shift rush here and Then there's literals This is kind of interesting Arm is a 16-bit instruction set so it can only load small Constants so the way they do it is a actually just in the code itself They'll have the 32 bit constant and then they'll load that from memory So the sampler actually already does that but like we do it dynamically So back going through the Code here. Here's where we actually generate the code for generating can So we have the can we have the bit buffer, which is just ones and zeros at this point And we either generate a set instruction or a clear instruction So what with the clear instruction? It's just storing data to the memory location where it controls the Output so there's a different address for clear versus set And then there's a it does a branch in link, which is basically calling a sub routine To a delay so based on your bit time the delay Will be different and I'll show you basically that and at the end here. We actually build a delay function And depending on what you're doing you're generating you're they're gonna you're gonna build a delay for your bit period And then you'll build a delay for consecutive message and that's called the interframe period. So we'll go to this so here it You pass in the number of the delay in nanoseconds, and then it generates a Cycle accurate delay on with an assembler Now, how did I get this perfect? I use it in a oscilloscope. I verified it So step one I calculated step two. It didn't make sense Step three you go back and kind of figure out what wrong and eventually that the instructions or the the knowledge match the actual the code so What's cool is the difference between cortex M zero plus and the other court all the other cortex I'm it has two two-stage pipeline So that means that the branches take two cycles Reverses three on all the other cortex M. So if you want to port this code, that would be definitely a consideration but the end result is The main core software calls a function called Sorry, that's the wrong source code Start and it will call this thing called send message and All this does is if the FIFO is ready It pushes a value into the FIFO and then the second core This is process query one it just Basically runs the program and sends the message And it doesn't show where it's checking the FIFO. So I actually have to go to a different source file For that. So here's the actual source for the being is checking the FIFO And then if it does it calls that function that will trigger the the send So this is the actual initialization for the code And This is where you start the core. So the first core starts the second core and that generates can messages okay, so now You might not understand on that code but the main purpose of the presentation is to time kind of give you clues and Give you the idea in your head that it's definitely possible to bit bang digital protocols that In the past, I think we're too fast to do that so for example a can frame is Typically the 500 kilobits that's the bit time is is a 200 or 2,000 nanoseconds, so with the eight nanosecond clock rate you have 250 instructions between each bit that the CPU can use to To deal with can't so it just opens up some some really cool opportunities For can have you now, of course now we want to receive can bust frame. So how could we do that and This gave me a chance to talk about the bit bang processor So the bit bang processor is is a revolutionary and it's a huge part huge It's a huge piece of this part, so it's such an important part It's got its own section and the data sheet it's got it takes up a huge part of the die area but it's really just super capable and What it is is there's actually two of them and The two pio modules each have four bit bang processors And the four bit bang processors share some resources and then have their own independent resources as well And it's what's kind of cool that you can just ways that the bit bang processors can kind of start at the same time Since they're all cycle accurate you can do a lot of really cool coordination between two But it's very complicated. It's not complicated. It's actually very simple But it's how to use it for a critical application. You just have to sit down and look at the instructions to see how it works But I did this just a very simple crappy can receive so What does it have it has eight different instructions doesn't sound like a lot But low and lower instructions are good because it maybe gives you what allows you to do an understanding of it It's got five foes To communicate with the CPU So basically the CPU will throw data in the five foe and and read data out So for example for if you're doing can transmission you would put those those bits from can into that five foe And then the bit bang processor would read them out send them From this example with the receive I Sample the bus and then I slide the bits into the five foe for the CPU to read read read There's there's 32 instructions only and they're shared within the PIO by all the PIOs So the can receive or I'm gonna show For the code it's it's I don't it's it's a Takes like say 10 instructions or something All those state machines could run Simultaneously with those instructions So if you wanted to capture, you know eight can channels at the same time you can do that because it's just it all works in parallel Pretty pretty amazing What are the instructions jump? The jump it's a conditional jump or non conditional jump and it can jump anywhere in those 32 Instructions I can jump on I open it can jump on there's some scratch registers or accumulators Stuff like that Wait wait for weight is waiting for usually a pin to change state So for can we're waiting for that start of frame. So as soon as that goes low that signals the start of a new can frame In and in and out are just like USB where they're the direction to the host So in means it's going to the CPU out means it's coming from the CPU. So in is is basically shifting data in Into a shift register And then out is out From pins into a shift register and then out is out from a shift rather two pins So if you're doing can transmission use out can reception be used in push push moves data from one of the registers to the the shift register the or the scratch register into the FIFO and then pull pulls a data value out Move move just moves data between the different parts Different registers in the part an IRQ it generates interrupt for the processor, which we actually used for the can reception And then set will actually set I opens or set values of the scratch registers Okay, so now we're going to do a demonstration of The receive code for can bus Okay, so we're back to the visual studio And this is in a class called can RX So just like there was a class that was created for generating Cortex M zero instructions There's also an assembler that you can write assembly code for the PIO But I wrote a class that just generates the PIO. It's very simple And I want to dynamically change them in code and honestly the sampler it just is It's easier to use code the only downside is that You have to have this code in your project So the some the sampler that I wrote is extra code in your project where the assembler if you just use theirs It just it just works just like a C compiler, which is it's actually really cool How they did that so when you learn more about it, but we're just going to go through this code The can receive in the badge is kind of not a very good one It didn't have a lot of time to work on it, but I just want to demonstrate the PIO engine So it only works at one specific Bit rate, which is 500 kilobits So the first thing you're do you're gonna do is set up the clock period of the PIO Now it can run at eight nanoseconds What there's advantage is if you can use a small closer clock rate, which I'll explain So this actually sets it up. So you have 32 Different clocks per bit. Okay So the first thing we do we initialize the PIO engine And then we're going to set up the PIFOS the PIFOS are configurable There's a lot of cool things you can do with them But two of the features that I use in this is there's a four word. So 32 bits for 32 bit Word PIFO I'm in and out, but you can actually join them so they can be all in or all out So I actually joined those 32 bit PIFOS, which means there's 200 fit or Yeah, 32 times 8 bits and that's actually smaller than a can frame any can frame. So you can actually Use the PIFO to capture an entire can frame, which is pretty cool Without any CPU intervention. So you set the pins. We're only using one pin. This is the RX pin off the can transceiver Then we start the the assembler. We start encoding I'm not gonna talk about wrap target But we add a label. So for jumps you need labels So this is just like in the assembler. The first thing we're gonna do is we're going to set up Interframe timeout. So this is how we detect when Tucked when the can message is done is there is a certain number of bits in between the messages Then we're gonna go ahead and sit ahead and wait for can to go low. So this false means low So this is the weight instruction Now each instruction can have its own delay. So if I look in there each instruction have its own delay So that means after the instruction executes, it will actually wait this amount of time to go to the next instruction So that's where the time determinism of all of the state machine is extremely useful So what we're doing here is we're just waiting for the sample point to be 75% And then what that is is since I set up 32 clocks It's 24 clocks to the sample point So here we were just gonna fall into this and then we do a jump and we do a bit a jump based on the can Rx to determine if it's high or low now What's weird about the jump instruction? It only will jump on a high So you have to design your code if you wanted you wanted to do something when it's low Then you got to just jump somewhere else, but it totally works. It just uses up another instruction. That's all So if it doesn't jump can is low Well can is low we refresh the timeout And we're storing this into this Y scratch register And then we we call the in instruction to sample the canline and put it that puts it into the shift register And then we just jump always back to checking the canline now How why am I checking the canline immediately? I'm actually not because in this in instruction. We're waiting 32 Clock so 30 clocks is one whole bit and I subtract away four because there's actually each instruction takes one clock So basically I took away the the clocks of the instruction and then resampled a bit if can is high Which is idle bus we sampled a bit and then we jump but in this jump we actually decorate the Y scratch register So what this does is it it will if if Y is zero it will it will not jump If not it jumps and then decrimates why so it's kind of a post decrement if Y is zero at this point and we fall through the loop to this and then we we flush the FIFO so The FIFO will automatically flush when it fills 32 bits But since the can is not gonna end on a 32 bit by me We have to flush that data in and then we we actually generate an interrupt to the processor. Okay, and Then this just goes back to The weight which is waiting for a starter frame So this thing is just gonna run and then anytime it sees the starter frame It's gonna sample the can bus till it's idle and then it will put that data in the FIFO Now I call this a crappy can decoder because it's missing a key feature of can and it's called resynchronization so whenever there's a Dominant edge whenever the can is driven by the transceiver You're supposed to check that the time Resynchronization so like if you if you know a lot about you are you you are time the receiver Since it's an asynchronous political it has to derive its own the sample clock from the bits So can what it does is it anytime? There's a high to low transition. It does what's called a resynchronization Where where it compares to where it thought the edge was and then adjust its clock now I didn't do any of that is it possible to do absolutely It's possible to do it's just more work and I didn't have time to do that in this this this Example and and and honestly with the PIO received it can totally work. The only problem with receive is acknowledgement so like The the PIO I don't think can keep track of which bit is which and decode But maybe maybe a handshake with the CB you can do that But like knowing where to set the acknowledge bit is is definitely going to be the biggest challenge now normally If you're just connecting to existing bus, you don't care about the acknowledgement or you're just spying So definitely PIO for receive totally could work And transmit really is very simple you generate the bits and then you just compare That's the sample point is that in the bit that I sent said Is it the same if it's the same then you're trans your transmitter if you're not then you just stop transmitting? So that I think that would be very easy to do PIO So let's look at the other Functions in this so how does the CPU actually get access to that data? so Every so often this the CPU In its main loop will call process and what it does is it looks at the message data count And then it just it just parses them and it changes the bits that de-stuff some some can bit stuffing and then it builds a can message Now where does this thing get filled up? There's actually an interrupt that's called so we have an interrupt callback here and Then that is called here and we actually read the read the 5-0 into this temporary date data section Which is basically then eventually processed by the main core, which is this code So just this is just an example of it can be done And we'll go back to the ending of ending this presentation the demo We have our pico scope up here and I'm gonna see some quick settings so First of all channel a I have a 10x probe. I want the range to be Plus or minus 5 so we can see the voltage and I have the probe here and I'm Going to touch the Touch the I open of the can transceiver. All right, so it looks like we got a good Way for him there. So I'm gonna do is I'm gonna go run one of the kills I Don't have another camera on the badge. It's because I ran out of USB ports So we're gonna do is we're gonna zoom in here and this is just generating error frames I'm gonna set up the trigger and this little diamond is the trigger point. You can move it wherever you want. So we're just Put it right there We're basically just change train the horizontal and I'll go ahead and stop that and we have our curse curses here now This is the logic waveform. So outside of the transceiver and we're gonna go ahead and look at measure this and You can see we're at Almost exactly 16 microseconds. So a two microsecond bit time is is That's it's gonna be eight pull eight pulses.  Let's go ahead and do that You can do it from the USB prompt or it's just more fun to do it with the badge You know, all right, so go ahead and run that and I'll turn the trigger back on and Yes So it can message so you can see it's generating So this is the idle bus and then this is basically can ID zero The Pico scope has some can decoding features on it. That's really cool But I don't have time to show that right now and then finally the random kill Which is not that is much This is exciting Hold both buttons for the random kill Now this one generates random delays between the messages as well Just random size message random can ID. It will actually generate errors once in a while as well Okay, so that's the the predefined kills And let's go ahead and I'll talk about the software. First of all, there's a version number I think we're releasing five or six Six has the power management features to run off the battery. So that might be the one you get It shows how many message bits you're generating and it tells you the time so this You know standard can messages like 250 microseconds And that's this particular message right here There's 16 stuff bits. So it's kind of a waste and can has to generate these stuff bits So depending on the data in the message, it's either going to be shorter or longer The bit period. So this is the actual baud rate.  That's this one down here And then you can actually generate an interframe period. That's the nanosecond accurate as well So if you wanted to do like 10 milliseconds between the message Response that lets you see can messages Can ramp the cam data. So that will change the data and the can message Or interframe then ramp the interframe period That's the the width between the message you can ramp. You can actually ramp the bit period So if you wanted to send, you know, 500 kilobits And then mess with the bit timing to see how any c responds you can do that And then ramp can ID will change the can ID. So These are the different settings. So let's go ahead and change the can message So this is the can message configuration. You can change the can ID The length of the frame, the bytes You can add invalid CRC invalid r0, which is a Common issue that with a lot of can controllers. It wasn't defined what to do about that You can change the interframe separation time and the end of frame separation time So if you want to violate that that rule where you send a message earlier, you can do that Generate an error frame. This just generates that error frame that you saw in the and then in jeopardy kill And also you can have an extended cam frame was 29 bit IDs And that's what's used in the trucking and the industrial applications a lot boats Remote frame. This is a frame that no one really uses But you know, why not see what happens So I'm going to go ahead and and then generate a different can frame.   There's there's your message. Now. What's cool about the 555 its Messages is that it's shorter. I don't know if you know it Because it has a lot less stuff bits. You don't need stuff bits if every other bit is different Uh, so, uh, let's do something. I'll let's let's I'll Send a kill And we'll we'll ramp the the can data So I'll select Six and then there's different patterns you can generate Walking ones is pretty cool And we're just Shifts a one through the can data every frame Did you catch that this is where the vehicle spy tool will come in handy to see actually what it's doing Now just to show some of the other features. I'm not going to demo The uh The can response And what this does is that you can set up a can id and that will trigger the arbitrary kill So you can transmit arbitrary kill. So when it gets 342 it will trigger the kill There is a bug with this version where it actually kills the badge. So it's kind of like Suicide feature I guess So the kill kills itself But you can you can just power cycle the badge and it will it will do it And then there's a can monitor can monitor is actually using the pio code. I just showed and it's uh, It only really decodes the can id at this point and there's some issues with the data But the data is correct. It's just the the decoding Transiting the bitstream into the can messages. There's a bug in it anyway, uh That's pretty much the badge and if you have any questions I'm going to be at the def or a defcon village on friday all day Uh, I might be there some saturday. I'm not sure depending on what else is going on. We're just wrapping up here uh There's a company called canis labs That had Came up with the idea of using the raspberry pi pico with can and they made a couple of products one Is just the raspberry pi pico connected to a transceiver That's called can hack and then another one. It actually connects to a micro trip can controller so it can do Can boss and they have an api but they support python very well The the cto of canis labs ken dr. Ken tindall He Has ported their software and he's he's did a number of talks about cyber security online that are pretty interesting in a lot of attacks But he designed those attacks into his pi pico code What's awesome is that the his can hack board is basically identical It's the pi pico connected to a can pin So he ported his software to the badge and it's available for download So you can generate different attacks from a can or a python repl So I encourage you to learn more about That port and as well as the different attacks that badge can do So he's got some research that he's done with it So so please check that out. It's really really cool and a real great bonus for for the badge And finally conclusions The rp20 40 Take a look at it. It's super interesting the dual core cpu and the bitbang engine At high clock rate, which you can probably overclock you probably do a lot of Destruction to digital protocols. Everything's really a digital protocol Now so like wireless all it is is digital protocols, you know, and they're and they're on a carrier and What's what's interesting about engineering and where cyber security happens is People are testing things But they don't test it to the point Where they need to it's it might be just impossible It's just hard enough to get things working these days So you shift the product and then maybe 10 years go by and then you come around with something like this And then you you interact with the protocol in a way No one ever thought and you can discover some interesting, you know, zero days and things like that So I think this is a real interesting tool for any protocol. This is just an example on the can bus So you can use that cpu cycle accurate eight nanoseconds, you know, or or or less And you don't have it's not like cumbersome like an FPGA So you can do a lot of cool stuff and can you do can bus with rp2040? Absolutely To 500 kilobyte can bus you have 250 instructions between bits Now you would have to do it the exactly opposite way as I did it So what I did I used the cpu to do transmission because that the arbitrary transmission was the main feature of the badge And then I use the pio to do reception Actually, the opposite will work where you use cpu to do reception because then you can do things like decode the can bus data length Generate the Generate the The acknowledgement in the proper location things like that, which you really need you really need to keep track of numbers and do some math But the pio I absolutely can do transmission And they're they're they're totally independent processes. So you can transmit with the The pio while you're receiving with the cpu and you can do arbitration by just Checking the bits if you put on the bus not very stopped. So totally possible I don't I don't know about knfd. I I uh, there's some bit time changes and stuff like that I I would have to say, you know, I only spent like a few days of this and there's a lot of creative ways to use the cpu and to use pio So I think you know someone with a lot of time and talent probably could do a lot with this Anyway, and the whole clear the conclusion is the chv badge that intrepid has designed It's pretty awesome And maybe I would like to think it was one of the best badges at least in terms of functionally what it can do And it's a great value. So hope you get pick one up. We made a ton of them in our factory in Troy, so please Take a look and I would be great to meet you in person and answer any of your questions in the village Uh, uh, uh or contact me on twitter or linkedin and um, you're super talented, you know, intrepid's always hiring So that's that I put the plug in But enjoy doffcon. Enjoy vegas and have a good trip home


Brilliant's Audio presents the unabridged recording of Glory by Vladimir Nabokov, performed by Luke Daniels, Tuvera, Forward. The present work closes the series of definitive English versions, in which my entire set of nine Russian novels, written in Western Europe between 1925 and 1937 and published by Emigre Houses between 1926 and 1952, is available to American and British readers. He who cares to scan the list given below should mark the dramatic gap between 1938 and 1959. Mashenka, 1926. Mary, 1970. Carol Dama Valet, 1928. King Queen Nave, 1968. Zashita Luzhina, 1930. The Defense, 1964. Saglia Datai, 1930. The Eye, 1965. Podvig, 1932. Glory, 1971. Kamara Obscura, 1933. Laughter in the Dark, 1938. Acheanya, 1936. Despair, 1966. Srilashinya, Nakazin, 1938. Invitation to Obahedding, 1959. Dahl, 1952. The Gift, 1963. The present translation is meticulously true to the text. My son took three years on and off to make a first draft, after which I spent three months preparing a fair copy. The very Russian preoccupations with physical movement and gesture, walking and sitting, smiling and glancing from under the brows, seems especially strong in Podvig. And this made our task still tougher. Podvig was begun in May 1930, immediately after my writing, Saglia Datai, and completed by the end of that year. My wife and I, who were then still childless, rented a parlor and bedroom on Lutpolstrasse, Berlin West, in the vast and gloomy apartment of the one-legged General von Barteleben. The old gentleman solely occupied in working out his family tree. His large brow had a somewhat Nabokovian cast, and indeed he was related to the well-known chess player Barteleben, whose manner of death resembled that of Milutian. One day in early summer, Ilya Fondiminsky, chief editor of the Sovrimenia Zapinsky, arrived there from Paris to buy the book Nakarnu. In the rooted state, set of grain fields before they are harvested. He was a social revolutionist, a Jew, a fervent...