So, hello everyone. My name is Ben and today my talk is about TSDB Compaction. So, a quick introduction about myself. I'm an SRE at Baddance, working on Kubernetes and observability. And I'm a Thanos maintainer and also a Prometheus contributor. So, first of all, I want to give big thanks to Ganesh for his great TSDB box series. So, if you are interested in TSDB and want to learn more about Compaction, please definitely check out his blog post. So, in his blog post, he defines the Compaction as a process of creating one new block from one or more source blocks. And this is actually a high level overview of TSDB Compaction. And in this talk, let's dive deep into it. So, because the definition mentions a TSDB block, I will first introduce what is a TSDB block. So, a TSDB block is created from an in-memory head block to disk every two hours. And this way, there's no time overlapping by default. So, different blocks will cover different time ranges. So, you can think of this as a way to organize your time series data partition by time. So, for example, when there's a query comes in, it will only touch the blocks within the required time range. So, if you query the most recent data, it will only hit the most recent block. So, each TSDB block is actually a minute database itself. It contains four parts. A JSON file to store some metadata. An index file for helping you search series and a trunks directory to store your data samples. And finally, a tombstone file. So, let's start with the metadata file first. So, it includes some block level metadata like the time range and some TSDB stats like number of samples, series and trunks. And some additional information about Compaction. And in other systems like tunnels, we extend the metadata file to include some information like the external labels or down-sampling resolution of that block. So, next, let's look at what is index and trunks. So, we all know that in TSDB, the smallest unit is actually a sample which just a pair of timestamp and value. So, let's imagine if we map all these data samples to a two-dimensional graph. For example, the x-axis is time and y-axis is series. Then the graph would look like this. And you can find that each row just represents a series. And a series includes two parts. On the right-hand side, we have some label sets, which are just some combination of label pairs used to uniquely identify one series. And for simplicity, you can think all these series labels on the right-hand side can be stored on disk in some special format. And this is called the index file, and it's used for looking up your series faster during the query time. And on the left-hand side, there are some data samples. And each row of samples actually belongs to each series. And to store these samples on disk, they are encoded in some special format to save disk space. And this is called trunks. So, when we run a query, it will first try to match the series from the index file. And for each series, it will try to fetch the corresponding trunks file on the right-hand side and finally return the results to the user. So, yeah, we covered the index and trunks. And I want to skip the tombstone file for now, and let's talk about compaction first. So, in this diagram, we prepare two blocks to compact together. Block A and B are created from the head block, so they don't have any overlapping data by default. And as you can see, they have some common series, like job equals HTTP server. And also in the meantime, they will have some different series, like this series from Kubernetes environment. Because we all know that Kubernetes environment is very dynamic, so the parts might be created and deleted at different times. So, it's very possible that some series will exist in block A, but not in block B. So, after this setup, let's start the compaction. So, I think the first thing in the compaction phase is to merge the several index files together into a larger index file. So, you can see we have two index files from A and B, and we merged them into a larger one. So, during this process, the common series labels, such as the labels from HTTP server and node exporters, they are merged and deduplicated, because we don't want to keep two copies of them. And for the different series, like the Kubernetes metrics, they are just sorted and combined together. So, after the index merging is done, next we want to merge the trunks together. So, let's switch to the two-dimensional graph again. So, what we want to do is to first iterate over all the series from the newly created index file on the right-hand side. And for each series on the right-hand side, we will try to fetch the trunks from different blocks and merge the trunks together. So, this process would look like this. So, because the two blocks don't have any overlapping data, we just need to combine these trunks together horizontally. And, yeah, you can see for these different series that only exist in block A, because B don't have this kind of data, so we don't need to merge anything. And, yeah, finally the results would look like this. So, we got a larger newly compacted block, and the index and trunks are merged together. So, this is how the compaction works. But this is the most common case, because there's no data overlapping. And another case is called vertical compaction. And, yeah, you can see block A and block B have some overlapping data. And this case happens a lot when Prometheus supports backfilling. However, the compaction phase is actually still the same. What we need to do is still merging the index file together, and then we're going over all these series on the right-hand side and try to merge the trunks together. So, the only thing different is that we might get some overlapping in this case, right? And, finally, we will get a newly compacted block like this. And the only overlap trunks is outlined here. And for this overlap data, we need to rebuild the overlap trunks and maybe do some deduplication. But this process is actually very naïve in Prometheus right now, because the deduplication is only one-on-one, which means only exactly the same trunks can be deduplicated. So, yeah, I think that covers all the process of compaction. And next, let's take a look at why we do need it. So, yeah, let's think about the query scenario. So, before compaction, if we do a long-term query, it will try to match all these small blocks and hit them so that in this way, we need to go through all these small index files and try to merge the results together back to the user. But if we can do some kind of compaction, then, finally, we will only have one larger block in this case. So, the same query will only hit one larger block and we don't have to do anything related to, like, merging results or some online deduplication in this case. So, query performance will be significantly improved. And additionally, because each index files from the small blocks might contain some common series, as I mentioned earlier. So, when we do the compaction and the merging process, we can save some disk space by deduplicating the index data. Okay, so, yeah, finally, let's take a look at the TomStone file. So, why do we need it? So, let's imagine the case if we want to delete some serious data from a block, then we can send a deletion request and specify, like, the series matches and the data time range we want to delete. And for performance reasons, this data to delete are temporarily saved in a file called TomStone. And this data are deleted during the compaction time. So, while we iterate over all the series, and we find that if the series and trunks actually match the data from the TomStone, and if it matches, then we drop this and so that the newly compacted block will not have this data. So, and additionally, if the TomStone file contains enough amount of serious data, the compaction itself will be triggered to clean up the data in order to save more disk space. So, now I think you have a high-level idea about what is a TomStone file, but in the perspective of compaction, it's still just some kind of modification to your index and trunks data. So, yeah, to me, I think the essential part of compaction is merging the index and trunks. So, yeah, I think we cover the compaction, but actually before the compaction phase, there's another phase called planning, and it's used to choose which blocks we want to compact together. So, in this diagram, there are five blocks in the TSDB directory now, and the planner chooses to compact the three blocks together into a new block F. So, to choose these source blocks, it follows some planning algorithm, and right now the planning algorithm in Prometheus is a little bit simple, and it only checks the time range and number of TomStones of each block, and this information can be easily found from the metadata file. So, I will not dive deep into the planning algorithm here, but ideally we can extend it to be more intelligent, and in Tunnels, we have supported one planner called index-size-limit planner to limit the maximum index size of the compacted block. So, yeah, I think that's all for the compaction of Prometheus TSDB itself. Next, let's take a look at some, like, more large-scale scenarios in some system like Tunnels or Quotex, and actually I will not cover Quotex compactor here because it's built on top of the Tunnels compactor with some, like, additional features. So, a common, like, Tunnels deployment looks like this. So, we might have two clusters, and each cluster has one Prometheus with one Tunnels sidecar, and in Prometheus, we will configure the cluster name as the external labels, and the sidecar will try to inject the external labels to the TSDB blocks and upload them to the object storage every two hours, and besides, we also have one central Tunnels compactor running against the object storage, and the compactor works in four steps. And first, because compaction needs to do some planning first, and planning step only requires the block metadata. So, the compactor will have a separate job to fetch the block metadata from the object storage and download them to a local disk. And after this is done, then it can start planning using this local metadata. And the third step, if there's an available compaction plan, it will try to download all these required blocks and compact them locally. And finally, it will try to upload the new block to the bucket and delete all these source blocks. So, as you can see, you can find that step two and step three is exactly the same as the Prometheus compactor, and the Tunnels compactor just adds step one and four to get it working with the object storage environment. And actually, there's a little bit of difference in the planning phase. So, planning is actually done separately in different groups, and each group is identified by the external labels added to that block. So, you can see here, we have blocks from cluster Europe and US. So, these blocks should be planned and compact together because we don't want to have super-larger blocks for maybe all the cluster go away. And, yeah, so finally, we got two blocks for Europe and US. And yeah, that's how Tunnels compactor works basically. And next, let's see some extensions we made to the Tunnels compactor to deal with our large-scale data use cases. And, yeah, the first scenario is actually super common where we deploy two Prometheus instances at each cluster for high availability purposes. And in this case, a new external label called replica is added to identify the source Prometheus of these data blocks. And then the problem of this setup is actually the data duplication problem because of the grouping mechanism. And the external labels for different replicas are actually different. So, we will have two groups for two replicas, but actually they are still in the same cluster, right? So, finally, we will have two compacted blocks like in this cluster US with totally overlapped time range for the two blocks. And obviously, you can see this is not good because firstly, we double the space usage by having two blocks with almost the same data. And also during the query time, we have to do some online duplication because also the query touches the two blocks. The query performance is not as good as with only one block. So, to solve this, actually, as I mentioned, we can use the built-in vertical compaction mechanism to merge the overlapped blocks, right? And the tunnels provide a flag called the duplication replica label. And if we specify it, the replica external label can be ignored during the grouping and planning phase. And all these six blocks will be planned and the compacted together. So, yes, this is good because we utilize the default vertical compaction mechanism and use it to improve the space usage and the query performance. But the question is, is it really good enough? And what about the trunks data? So, actually, the trunks data is still problematic because the trunks data are from high availability, pre-missius pairs, and actually they cannot be perfectly deduplicated by the default vertical compaction mechanism because we have two pre-missius instances and they cannot actually perfectly collect samples at exactly the same time. So, the timestamp of these samples from two pre-missius cannot be the same. However, the default vertical compaction only works for one-on-one deduplication, but in the case of different timestamps, it doesn't work. So, finally, we still have double space usage for these replica data. And it would be better to have some smarter vertical compaction algorithm to do this deduplication for this use case. And actually what we want is to have something like this. We want to maybe only keep one sample from only one replica, and that's already good enough to us. So, this can be done by extending the deduplication algorithm used in the vertical compaction, and in tunnels we supported the same penalty deduplication algorithms used by the tunnels query and we applied for deduplication at the compactor side. So, this saves the space usage a lot. And another challenge we made in tunnels is about how to deal with some dedu-manipulation requests from the object storage. For example, we want to delete some high-cardinality series or we want to relabel these series or rename these metrics. And the problem is that these blocks are from object storage and they are immutable by default. So, can we modify the existing index in the trunks easily? So, yeah, because today's topic is about compaction and actually tunnels uses this compaction mechanism to achieve this request. So, let me use relabeling for an example. So, we have this kind of relabeling configuration and what it does is that the first relabel request just renames these CPU metrics to another name and another relabel request just drops this label with the name code. So, in this case, we are still trying to do a compaction for this block we want to modify. And while we iterate over all these series on the right-hand side, we just do this relabeling to each series label. And if this series label changes, then we can do some actions accordingly. For example, in the case of label replace, it's super easy because we just need to rename the labels on the right-hand side. So, only index parts need to be changed. We don't need to modify anything related to trunks. But the label drop case is a little bit trickier because the code label, once the code label is deleted, then these three series will be merged into one series, right? So, in this case, we need to also merge the corresponding trunks files and combine them and do some deduplication in this case. But anyway, finally, we relabel this PCB block and we got a new block after this compaction. So, this is cool. So, why not we have this support in premises as well? So, recently, I opened a PR to support this kind of use case in the PROM tool. And actually, the required change is super easy. So, first, I added one interface called modifier, and it has only one modified method. And this is just used to apply some additional modification logic to the index files and to the trunks files during the compaction phase. And for the Prometheus built-in write method, it's extended with a list of modifiers to apply this logic. And to view it more easily. So, this is the graph again with the modifier. So, you can see the modifier just works as a middle layer in the compaction process. And during the compaction phase, we still need to merge index files and trunks together, but we apply the modification to them before writing a new block. So, what can we achieve using this modifier? So, let's do some brainstorming about down-sampling. So, you can simply think about down-sampling as a process of increasing the script interval in premises. And in the diagram, you can see we down-sample data from 15 seconds to one minute resolution. And after down-sampling, we have fewer samples in this case. And you can imagine, in the case of down-sampling, we just need to change the trunks file and we don't need to modify anything related to the index. So actually, we can implement down-sampling modifier and with the configuration on the left-hand side when we do the compaction, we find all the matches series with this matches label and just rebuild these trunks with this configured resolution. And the last brainstorming case is about dynamic retention. We can also implement a retention modifier and during the compaction time, we can check whether these series matches are required matches and for these matches series, we move them to a new block and keep them for a longer time. So, yeah, I think that's all for today about my talk. So, yeah, thank you for everyone for listening and I'm ready for the Q&A part. Thank you. We're going to have a slightly shorter Q&A for this session to catch up a little bit on time because we've run a little bit over. Does anybody have a question they'd like me to repeat for our virtual audience? All right, I have one then. You mentioned the two dimensions of vertical and horizontal that you compact by. I think I'm correct in saying horizontal is your time there and vertical is overlapping data, that sort of thing. Yeah, but actually, I think vertical and horizontal is the same thing. But in the graph, you can see when we try to merge the trunks together, it's just still like horizontal emerging, right? But if there's no time overlapping, we don't need to do anything. We just change these trunks together. But if there's any overlapping, we need to rebuild the trunk. So, yeah. I was going to ask, do you see any other dimensions that besides those two that you would like to... Yeah, I don't think so. I think only two dimensions. Well, thank you very much. Okay, thank you.


Finally down to Brainerd, they're also at home hosting Fergus Falls at start at 123. It's Brainerd's Chad Orzburn. He'll pin an order to the map for the easy win. That would be the theme all night. Next, Kyle Ostenbacher. He also gets the pin here over Fergus Falls. It's Caden Olson. This one may be the fastest of the match. Warriors dominate, start to finish. They cruise to the win 71 to 6. If you enjoyed this segment of Lakeland News, please consider making a tax-deductible contribution to Lakeland PBS.


From New York, it's theCUBE. Covering Big Data New York City 2016. Brought to you by headline sponsors, Cisco, IBM, NVIDIA, and our ecosystem sponsors. Now, here are your hosts, Dave Vellante and Peter Burris. Welcome back to the Big Apple, everybody. This is theCUBE, the worldwide leader in live tech coverage. Peter Burris and I are thrilled to be here in New York City. John Furrier courses at splunk.conf, knocking the town there. Bernie Schieffer is here, he's an IBM fellow. Bernie, great to see you. Nice to be here again. Were you on the rooftop last night? I sure was. Beautiful view. It was an excellent announcement by you guys, a fantastic event. IBM obviously knows how to throw a party, but there was a lot of meat behind the bone as well. I had a lot of people ask me, is this really an IBM event? So I thought that was a couple of it. All right. Well, it's a little funky down here, right? There's not like big hotels and so, great job. So anyway, congratulations on that. But we're here. And a number of extremely happy developers. Yeah, yeah, right. IBM gave away some great prizes to you guys. $100,000. Yeah. The prize was $50,000 and two guys split it and the second prize was $25,000, a single individual won that. So you had three people walked away with $25,000 and then some other nice prizes as well. It's going to be one of the most talked about things on the show floor today. Yes, so. And Rob Thomas said he was going to do it again. So that makes it. Yeah, it was good to see Rob up there and outchecks. We like this. Good role for him. And sorry, we're here to talk about a little bit about ODPI, a big sequel. So what's your role at IBM? And let's get into it. So I'm a longtime IBMmer. Have been working in the data field for my entire career over 31 years. But for the last four or five years, I've moved strongly in the world of big data. The IBM distribution of Hadoop called Big Insights. And a couple of years ago, we created a new offering in Big Insights called Big Sequel, the most powerful sequel on Hadoop Engine that we have. Okay, and you're building that into a new Hadoop distro with Hortonworks, is that right? And you're using ODPI as the framework? Talk a little bit more about that. So we've had Big Sequel for a number of years and it was distributed via our own IBM distribution of Hadoop, which continues. But the market in the distribution space is fragmented. Clients have chosen typically one or two distributions as their primary distributions. And unfortunately, not everyone has chosen Big Insights. So we could, just like in the days of relational databases where you support different operating systems, HPUX, Solaris, AIX, Linux, Windows, and so on, you can make software portable to different platforms. And to me, Hadoop is very much like a platform, a big data operating system, if you can call it that. And so with work, you can make anything run just about anywhere. It is after all just software. But Linux standardization helped make it easier to port and run on different levels of Linux. So ODPI is trying to do the same thing for the Hadoop ecosystem. So what is Hadoop really? The classic definition of Hadoop, which people still call up, is it's HDFS, the persistent file system, plus the map-reduced paradigm for processing data. But to me, today, it's much more than that. Why? Because people have replaced components of that. They don't always use HDFS, they use GPFS, or there's other open source file systems, Eluxio, and so on coming along. And really, map-reduced in the classic sense is really maybe more of a legacy system today. Spark is the hot topic. I mean, you heard that last night at the Data First event. So, but does that mean Hadoop is dead? Absolutely not. I mean, it couldn't be more vibrant. Look at this conference. I mean, it's exploding. So what is today? To me, I think Hadoop should be redefined as an ecosystem of collaborative tools. Very much like Unix became an ecosystem of tools. And so we need to be able to interoperate software that sits on top of that platform. So the more similar different distributions are, the easier it is for people to build middleware or applications on top of Hadoop that can run on a choice of Hadoop distributions. Okay. And more rapidly, you can create experience, understanding, and knowledge so we can start focusing on use cases and the domain expertise associated with use cases and the real insights that these tools are supposed to do. Which is the real value everyone wants. The rest is sort of plumbing, right? And you want all the tools to run on a variety of distributions. Well, I got a lot of questions. Actually, we want to come back to IBM's distribution, Hadoop distribution.  I mean, it was Fujitsu and Wendisco and, you know, Silicon Angle, right? What's going on? I didn't realize. No, we used to joke about it because everybody was announcing them. And so, you know, IBM maintained its Hadoop distro.  But it's not make or break for IBM. And if you're not the worldwide leader in Hadoop distro, it doesn't really make a bit of a difference to your business, right? You're doing it because you... Well, maybe a bit of difference, but... Maybe a tiny bit, right? But I mean, it's got Watson and analytics and cognitive and AI and machine learning and all this new stuff. And so, you maintain that because you've got a customer base, right? And there are clients who really value the, you know, superb level of support that IBM, you know, offers for all of its products around the world, 24 by seven. So, and there's a comfort zone saying, when you buy a Hadoop distribution from IBM, it's going to be around and it's going to be supported and it's going to be supported well. Right, and you can make a case that this is more integrated and we're going to support it and blah, blah, blah. Okay, great. But it's not like the be all end all of Hadoop distros. Okay, so we just wanted to sort of establish that.  ODPI is, you guys are involved, obviously, Hortonworks and others. Talk a little bit more about... Many others. Talk about ODPI, why did ODPI come about? There were many naysayers when it first came about. You know, particularly the two other major, you know, distro vendors. What is ODPI all about? What has it done for businesses? So, I view it as having two levels. It's brought groups together that at some level compete in the marketplace. Say Hortonworks and IBM Big Insights. And at the same level, at the same time, we also have the desire to grow the market, to enable clients to be successful. And as you said, to generate those insights, the holy grail of all this big data activity is not the data. That's just sitting there doing nothing. It's to get insight from that data. So, again, an analogy I'll draw is that, you know, in the database space, IBM has competes and has competed and continues to compete vigorously with the oracles in the Microsoft SQL servers of the world. And yet, at the level of standardizing SQL, we all work together. So, you know, there's no disconnect between competing at the level of your product offerings and collaborating at standardizing SQL for the benefit of clients and the industry as a whole. And so, ODPI, to me, is an exact analogy of that, which you want to bring different companies together that compete at one level, but they collaborate to make sure that the plumbing is sufficiently common and standard. So that the layer of middleware and applications on top have an easier time, can get more traction, get more clients, and deliver more value. And so, there was a whole bunch of companies, some of which are, I would draw them into tiers. There's the foundational ones. There's four or five that produce Hadoop distributions. The biggest, the most famous of those are Hortonworks and IBM. Then there are people who sit on top and there's even consulting companies as well. So, different layers that use Hadoop from different layers. And customers. And customers as well. So, the four core components were HDFS, MapReduce, Ambari, and Yarn. And then Spark comes into play. Obviously, IBM's putting huge emphasis on Spark. You mentioned, you know, that MapReduce, in many respects, is legacy. So, how does ODPI evolve to include new innovations? Well, you have to start somewhere and you have to start with what a group of people think they can standardize. Some things that are, you know, imagine if you tried to standardize machine learning. Well, that'd be tricky, right? Because, you know, is it Spark? Is it TensorFlow? I mean, there's this profusion of techniques and companies. So, that means that that part of the ecosystem is not ready to be standardized. So, they had to pick the things that were most ready to be standardized, that were already quite similar. You know, that maybe they were like 98% the same. So, that last 2% is a solvable problem. And that's what they did. So, HDFS is relatively mature. You know, HDFS is going not evolving so quickly. And so, it was easier to converge. Same with, you know, MapReduce and also Yarn. Ambari is, you know, the next wave. So, that's kind of the one that's actively being worked on right now. And Ambari has, you know, installation configuration and monitoring. And the first piece of Ambari that they're working on is actually to, you know, make the deployment the same. In the same way that, think back to Windows. You know, there was a, Windows installation used to be a gazillion different methods, right? And then there was sort of a standard Windows installer and most companies converged on that. And that made for a similar, you know, install experience, then similar entries in the registry and so on. After a couple of lawsuits. And that's one of the things that you're trying to get collaboration, cooperation early on. So that you reduce the uncertainty about how some of these things are going to work. And collaboration where you're not really differentiating.  I mean, that's not really where you're creating value.  Okay, and then I'm trying to, I want to unpack the UNIX analogy versus the SQL analogy. And I'm thinking the SQL analogy maybe is better, but I want to explore that a little bit. So, you know, the UNIX analogy, you know, it ended up, UNIX ended up not working, really, as well as we had hoped. You know, the vision of having sort of binary compatibility. Well, you remember and DF. Yeah, well, exactly. And DF didn't quite play out. But we're talking about, we're talking about a level up above. Well, but so that's what I'm trying to understand is was Linux that level above that? And the SQL maybe the better analogy, which you used as well for what's going on with... So no analogy is perfect. You know, I think if Linus hadn't come along and invented Linux, who knows what would have happened in the world of UNIX? You know, there were... Sun might still be here. Possibly. Right, so I think SQL is a good analogy because of the active participation of all the major SQL vendors, even the ones that are, you know, definitely competing head on, you know, IBM and Oracle being a, you know, a good example. All right, so big SQL and SQL generally in big data, right? It's like the killer app for Hadoop. Ironically, right? Because, you know, when I was first introduced to Hadoop, it was kind of like the no SQL space. It kind of turned into the new SQL space. So I... That's the way of putting it. Oh, it really is. And I think it's not that SQL is perfect, but, you know, the thing that it is powerful. There are abundant skills and there's lots of tools. And those are things that really help people get going. And I think it's that second point that's so dominant that there was an enormous amount of experience around Unix that wanted to focus more time and attention on delivering value well above it. And so when you came up with a common way of looking at Unix that worked very well, obviously Linux, Linux, then people gravitated towards it. And it was the experience that made that so successful. And the same thing in SQL. There's an enormous amount of experience around SQL that's in manifests and tools, because the tools are pedagogical in that regard. And so at the end of the day, it's the community, the customer base that's driving this new notion of the commons around skills and requiring this kind of new approach to focusing on the underlying tool set so that that skill can be applied to bigger basis problems. Is that accurate? I would agree with that. You know, it raises the interface level to one that is known, proven to work. Familiar, it creates a new commons. Right. Okay, so what else can you tell us about what's going on at ODPI? Like we were talking about Spark a little bit. Does Spark fit in here? I mean, IBM's obviously making a huge investment in Spark. Are you pushing for Spark? Spark is my equal partner in ODPI. So, you know, Spark is a patchy project. So, I mean, a first-class citizen I meant to say. Right, so it's already being distributed by most, if not all, of the Hadoop distributions. I know MapR has it, Cloudera has it, Hortonworks has it, IOP and Big Insights has it. So, you know. That little Databricks company, right? Yeah, yeah, yeah. Well, they don't have their own distribution. They work on Spark and Spark fits into the Hadoop ecosystem, but of course, Spark also runs outside of a Hadoop ecosystem, which makes it interesting. So, but to come to your question of Spark for ODPI, you know, I'm gazing into my crystal ball. I'm going to say I would love to see Spark be a next project for ODPI to tackle. I think it makes a lot of sense, especially since we already talked about it. MapReduce is kind of this legacy variant of the Hadoop ecosystem. You know, kind of like Mahoot is a little bit legacy as well. So, it has a natural successor, more memory centric, you know, compute intensive, you know, has a lot of backing from many people, very active Apache project, seems like a really good candidate as a next wave. Now, you know, I'm not that deep into the org, so I don't know exactly where it is on their roadmap. I think it would make a lot of sense. All right, good. All right, Bernie, we'll give you the last word on Big Data Week, Big Data NYC, Hadoop, Strata Hadoop. Well, great to be here, great to see the energy. I'm, you know, I'm pleased to be able to present tomorrow about the availability of Big SQL on Hortonworks, something that many clients have asked. And, you know, I've had other clients say, and, you know, when our cloud era and map are going to join ODPI, so we can have Big SQL on cloud era and MapR as well. Ha ha ha, when ODPI gives up on Embaree, that's when... Ha ha ha, Bernie, thanks very much for coming on theCUBE. Really appreciate it. Thanks for the invitation, great to meet you. All right, keep right there, buddy. We'll be right back with our next guest right after this short break.


Just four months after the devastating October 2017 wildfires, more than one million tons of fire-related debris has now been removed in Lake Mendocino, Napa and Sonoma counties. Debris removal is now completed in Coffee Park, one of the hardest-hit neighborhoods in the town of Santa Rosa, and the debris removal process is also 100 percent complete in Lake County. The fact that these major milestones have been reached in such a short span of time is the result of the consolidated effort between local, state and federal officials working together to meet these deadlines. We wanted to aggressively move on getting the debris cleaned up. We knew that winter was coming. We have not seen something this large and this complex from the debris standpoint since the 1906 earthquake. And so, you know, we wanted to get all hands on deck moving very, very aggressively. It's a monumental task to do it in the time period that we're able to do it in over winter and heavy rains, difficult times, with over, some days with over 2,000 trucks moving in a day through the holidays, just a monumental task. As we look past and previous disasters or fire disasters, it's taken us years to get to this point and to be able to get to this point where we're actually starting to be building within months after a fire of this magnitude is a enormous task. The magnitude is truly significant. When you look at over the past three months where we've been and we're at now, two days ago we went over 3,000 parcels completed of debris removal operations to highlight 2,300 here in Sonoma County alone. That puts us at roughly 65 percent complete. While the neighborhoods like Coffee Park and counties like Lake have been completed, there is still much work to be done, but with each completed lot, crews are able to move on to other areas and continue the progress. The good news is now that Coffee Park is complete, those crews that have been working here are moving on forward to other focus areas and that allows us to expedite our schedule and complete as soon as possible. I can tell you that we are collectively moving, we are setting a new bar on the speed at which we have responded to these things. There is no other state in the country that states that have been hit by major hurricanes in Texas and in Puerto Rico and in Florida are not even close to where we're at today and our events happened after theirs. For more information on the debris removal process from the October wildfires, you can visit wildfirerecovery.org or sign up for the latest news at oesnews.com. For all of us at Cal OES, I'm Brian May.


Two members of our crew working in Akiva. What they're doing now is isolating and shaping a stratigraphic column prior to removal. The next step after shaping it will be to wrap it in plaster of Paris so we can take it out in one bulk just like the one you see on the bench there. Originally we had a a complete column but because of weight factors we cut it in half. The purpose here is to take the column's intact back to the laboratory. Separate them. Subject them to flotation and pollen analysis so we can get an idea of plant succession on the site in the nature of botanical remains after the abandonment of Akiva. Yeah Try to get all the air I was asking Bill if we could possibly get our prehistoric as well as historic environment out of this column try to recreate the past and the present in environmental manner. Bill was explaining to me that possibly in certain cases we can. Well like most of the dirt in the column is post-occupation stuff that's come in since abandonment. And it's going to contain a lot of information about what's happened at this locality in the last seven or eight hundred years is you know Bill came in here all that time you know accumulated that's kind of what we're getting now. Charcoal from a brood roof and right under this we know that there's a layer or it wasn't the primary floor and then this a fill here from down here at the bottom of whether we have a definite floor up to here is all filled during the occupation and that's where we'll get a lot of information about the environment at the time of the occupation. We're from this layer this part in here. This surface is a little obscure. The largest most obvious structure on this site is a pit house which we believe to be now to Pueblo 1. This is the floor. We cover about a third of the house. The floor itself is only one of the important things that we've noted in this site in this wall here. We've been able to pick up the collapse of the roof, an area of a period of abandonment indicated by thinly bedded water wash sediments. Then a second short occupation with a deposition of trash indicating the Pueblo 1 period again. Another minor period of abandonment again the sands and then a second occupation again Pueblo 1 period of abandonment. This time an occupation during Pueblo 2 times. Final abandonment with a very minor use indicated by a fire heart here. The upper fill here is probably modern with disturbance by amateur excavations. From the floor of the pit house we find above it the roof fall. The area was then abandoned as indicated by these thinly bedded sterile water wash sands. There was a short reoccupation indicated by a dark fan here. Another period of abandonment. Then an occupation where there was a buildup of cultural fill from the Pueblo 1 period. We had another period of abandonment again the sterile sands. Another trash layer. This time Pueblo 2. The cave was again abandoned, occupied for a very short time, a small fire heart present in the wall here. Then final abandonment by the Anastasi and the modern deposits among the material disturbed by the previous excavators of the site. We know that the people who lived in this cave were farmers from the corn, beans, bits of squash that we found in the deposits here. They probably farmed the deep alluvial soils of the river bottom depending on the river to fertilize their crops and to provide the necessary moisture. But these people also depended upon the wild crops about them. We know that here they hunted deer, elk, large game were a fairly large part of their diet. These hunters would also provide the necessary resources, sending people far and wide to see how the crops were kind of the wild grass seeds. In addition to farming, we know these people were hunters, deer and elk, formed to make a part of their diet. And these hunters, as well as hunting, bringing meat into the camp were the contacts that watched the plants grow, watched the wild seeds, the pinion, wild grass, juniper, serviceberry, cactus. All the things that these people were able to utilize throughout the different parts of the year as supplements to their agricultural food resources. The unique feature of this site that shows us just how important the uplands were are these steps which lead up and provide access to the shrubbery, the vegetational community of the slope up above. We know from dried materials found within the shelter that the anasazi that occupied the cave utilized the cactus among other plants. They utilized the pads, burning the spines off, roasting them. As well the fruit of the cactus was used. The fruit could be eaten as it was picked, of course after removal of the spines, or they could be taken back to the cave and dried acting as a self-contained package for the many cactus seed within the fruit itself which themselves could be ground into a flower or a gruel. In addition to food, the Dolores River and the slopes of its canyons provided most of the other raw materials utilized by the anasazi. For instance, information in the cave indicates that yucca was a primary source of fiber from which were made sandals, mattings of all sorts, cordage, snares, ropes. The leaves of this plant could be pounded producing the long fibers which could then easily be woven to provide the necessary materials. Another possible use of the yucca to the anasazi suggested by the early Hopi practices is that the root be extracted. And a soap made, tuberous root could be dug out of the ground, pounded between two rocks and the fibrous parts of the tissue releases a natural detergent. As well as the plants, we know that the anasazi depended on these slopes to provide many other materials. Sandstone slabs such as these were used for building stone drawn from the more thinly bedded portions of the formation. Chirps eroded from up above, washed down in the slope were used to make tools, axes, knives, points for arrows, the tools for farming and hunting. The anasazi took a lot from this environment. They utilized it, were changed by it. And as we begin our research here, trying to understand how the interaction progressed, how the anasazi met change with the changing climate, then perhaps we'll start to understand a little bit about why this area was abandoned by them ultimately. And in that gain a few insights into how we ourselves should utilize this land.


It's a pleasure to introduce Chris. Don't know for a long time. Done lots of things together. Enjoyably fun, enjoyably chaotic. But needs absolutely no introduction. So dragon, Chris Roberts, sir. Thank you, sir. Appreciate it. Thank you. So I was over in, I just got back from Skytalks. And so how many, there's at least one, two, or three from Skytalks. So I'll apologize. You'll see a bunch of these lights that are going to be similar because I didn't know how many people. Anyway, so how's it's keeping? Let's see if I want to see if I can tether myself a little bit further, but I can yell out and feed back. All right, so the stuff that I'm doing here is research that is literally being done in the basement. This is hacking in the basement. This is really fundamentally where we've all started off from. And I'm hacking the brain in the basement, which kind of makes like a Frankenstein hacker. Yeah, it's going to be fun. It has been then in close proximity to a lot of alcohol. So we have done a little bit of drunken brain hacking as well on this whole thing. And yes, I've also tried posting data back through the brain, which has got mixed results. My daughter came down the other day. She's actually over in the SE. It feels really, feels like there's feedback or something, but that's just me. Is that just me? All right. Yeah, she came down the other day and I had like the 14 unit on. And she's like, what are you doing? And I'm like, I'm trying to send data into the brain. She's like, Papa, are you trying to fry your brain? I don't think so. All right. For those of you who don't know me, there we go. Thank you, you rock. That's way better. For those of you who don't know me, I've been messing around in this industry for more years and I get to think of this is, I think, the 19th year I've done this lunatic asylum known as Defconn.  Yeah, exactly. Him too. We've broken everything from trains, ISS, Mars Rover, all sorts of other things. Obviously, you're not hanging out at L.A.R.I.S. doing a bunch of stuff where I was just trying to change the industry a lot. Currently, doing a bunch of work with humans, a lot of work with humans. If you look at the security industry as a whole, we keep trying to apply technology as the solution. We keep saying, oh, buying another pair of blinky light buy something else and we keep forgetting that there's a human element to this that we need to deal with. And I might have a slightly obsessive whiskey collection. All right. Simple question. We have basically spent the last 12,000 years or so upgrading ourselves and our surroundings. We have spent a lot of that time trying to improve ourselves, our surroundings, our lives, and everything else. But if you start looking at the last 10, 15, 20 years worth of change, especially in this realm, so everything from biotechnology, nanotechnology, consciousness, and everything else, there's an argument to say that the tech itself is outpacing us. So the question is, what the hell do we do about it? And then if we have to evolve, how do we evolve about it? And we start taking a look at where we were. For those of you that follow me on Twitter or LinkedIn, I might have posted this one and I couldn't help it. We start taking a look at what Mother Nature's done and how that damn it, how Mother Nature's evolution takes a little bit of time. And then we start taking a look at the human software development life cycle and that's really been the last 12,000 years, give or take a bit. So this is a slide I like. I've been using it on a couple of presentations. This was us. This is us. And that is going to be us from a technology standpoint. It's getting kind of interesting as to where we're going. Part of the logic is obviously we're putting more and more technology into our hands. You go back 15, 20, 25 years or something like that, we had the portable device that we lugged around the size of a suitcase that weighed 50 pounds if we were lucky. Nowadays, that same technology, obviously we're happily sitting here with phones, with devices, with embedded tech. There are what, 50, 60 plus badges being given out at DEFCON this year. Every single one of them is a hell of a lot more powerful and move down stuff that we had a while ago. The problem is we're handing it to a population that doesn't necessarily fully understand or even necessarily want to understand what the hell to do with it. So there's also that argument to say, hack, we'll just let technology deal with security. That can get a little dystopian. So what the hell do we do about it? So, simple proof. I've been going to these for 20 years.  We have blamed everybody. We started off by blaming users, then we blamed developers and we blamed the CFO, then we blamed the network team, then we went back to blaming developers and we blamed grandma, we blamed kids.  This is setting the scene. This is the why do we care and why do we need to care? So we keep losing lots and lots of data. We keep losing almost as much data as we keep producing. So at this point, we got a couple of options. Do we let the system evolve? Do we keep looking at how the system's evolving? Do we keep looking at what technology is doing? Do we keep going to a stage where the actual technology does become sentinel and all those other wonderful things? If we do, then there's some interesting decisions that we all might or might not have. Or do we see if we can evolve as effectively as those systems are doing, which is where we obviously come to nano and bio and all the other stuff. And yes, there is option number three, which is eventually we find a power cord and kick it out. Personally, I'd rather do some of this. I'd rather look at the evolutionary side of it. So again, for those of you that were over at Sky Talks, you saw this one. This is some stuff I'd been working on a couple of years ago and this really sets the scene for what we were talking about. How many of you are familiar with the nano tech and the bio nano tech side of the world? Anybody familiar with it? A couple of folks kind of familiar with it. A couple of folks heard the talk earlier on today and familiar with it. All right, these are amazing little critters. I would highly recommend you're in the biohacking. This is biohacking combined with our world hacking. Nano-carbon architectures, basically a carbon-based molecular at a molecule level, individual molecules, you can feed them into tubes, you can make them flat, you can do all sorts of interesting stuff with them. Depending upon what you do with them, their properties change. Conductive, reactive, reacting to elements. You can do all sorts of interesting stuff with them. And as individuals, and if you're getting into the biohacking side of the world, you can do some fairly cheap electronics. You don't need an entire hospital full of stuff to mess around with. The stuff we've been messing around with and a lot of the research has been done with about a couple of hundred dollars worth of either Arduino's or we cheat a little bit and use the hack and the blade RS. We have to learn from Mother Nature. We all know what vaccines are, whether we agree with them or disagree with them, we know what they are, we know how they work. Basically take a host, try to tame it and then put what we want over the top of it. In the molecular side of the world, we did the same thing. In the molecular research we did a couple of years ago and this is, I've had this stuff out for a couple of years and every now and again I come back to it and I update it and we do a few more things. Some of the recent stuff we've done is now combining this with the consciousness stuff. But the basics come down. We have to get the carbon nanotubes into the body. We have to understand them. So basically we infect something. In this case, we infected bird flu. So we looked at ribonuclear acids. What we ended up doing is a little bit of fun. We turned them into our uses. In our digital world, we know how to build viruses, we know how to build Trojans, polymorphs and everything else. Take a leaf out of Mother Nature, we can do exactly the same thing. We can do it two ways. Primarily, nuclear acid robots, the biotech and the bionanotech and no others using the body's own chemicals, using its own agents as a defense mechanism, as a building mechanism, the bacteria-based, though they're more complicated or the nanoscopic machines, the assembler machines. This is becoming a lot more prevalent now and the costs are coming down drastically. What used to cost like several thousand dollars per microgram, you're now looking at two or three dollars per microgram or for carbon nanotubes that we can now start messing around with. So, like anything, you have to teach it and train it. In this case, this was back in 2016, EPFL's lab. Kind of like kids, you got to teach them and train them basic language, basic understandings. In the case of what was going on with the carbon nanotubes here, it was like, okay, eventually these are gonna go into the body, how do we manage them? How do we control them? How do we have communications with them? This is a lot of the work they did to teach them basics. A lot of the communications here were done basically using RF frequency waves or using MRI scans or using lasers. In other words, you're inputting the control with an external interface because you can't exactly jack in the USP and go, hey, give it a second, I'm just gonna recode it. Little too small for that. 2017, they told him how to swim by basically applying a nanotube and then a tail to it. I'll make sure these slides are available if that makes it a little easier. I'll just hit me up afterwards, I'll make sure the slides are available. For those of you that know me or recognize, I obviously have a fairly decent LinkedIn profile, I actually have a draw box, I will put these slides up on draw box, just hit me up for the link somehow or other, I'll make sure you get it. So this is 2016. Attach a tube to it, basically hit a laser height or electromagnetic pulse light to it. That actually causes the tail to move, that friction of the move moves it through a liquid, in which case you can now move it through the bloodstream. What you can also do as well is you now teach a basic chemical. Can it recognize certain chemicals inside the body? Answer, yes. Why do we know this? Because we now fast forward to 2018. Bottom right-hand side, turbo-charged sperm. Yay. Let's just make it easier. Let's just improve mother nature or not, as the case may be. Taking a flat carbon, basically instead of taking a carbon nanotube, you take a flat-pane architecture, you apply a cylindrical shape to it, and then you apply treatment to it, and then you just heat it with a pulse, which point it just turbo-charges the sperm. It's quite fun to watch. Or not, depends on how you look at it. This is the one I like. I came out of the military, I got shot up a few times, and when we went back in, we had basically the chop guys would basically sell us back up to make sure that we could get in the back without getting yelled at too badly. Nowadays, what they're doing is they're embedding carbon architectures into some of the stitching system. That way, the carbon nanotubes and the actual receptors and the antennas are able to detect chemical imbalances, chemical changes, and more critically, heat change in a wound. Now you can monitor for septicemia. Way easier, way simpler. And you can see all the other stuff that's getting used on the body as well. Oxygenation, hydration, all this other stuff.  So what do we do?  As I said, these slides will be available. The two things you wanna pull away from this very complicated slide Verilog is actually the circuit function language. We're supposed to use cello to design the sequencing. So again, this isn't a matter of just basically jack in and have at it. What we actually do, taking your leaf out of XKCD, we basically do the code hacking. Those of you that are used to coding, we take something very similar to a combination of assembler and basic. We then turn it into almost an... I just can't stand that, can I? It's gonna get yelled at. I'm good, I'll stand here. If you go that way, you're gonna get a full range of those cello samples. I don't wanna go on the slides. Why? Can I have a laser pointer? I could be like proper, can I? Who's civilized? I say it. All right, anyway. In the middle is basically taking that code architecture and turn it into a gated sequence. In other words, electromagnetic. For that, how many of you are on the engineering side in this world? So you should understand some of that stuff. It should make sense. Now, what you do to get it in from that area into the body, you've gotta turn it into a wave sequence because it's gotta be understood. So that's the far right-hand side. So again, we have our keys. We take something, we have hexadecimal six-station ring. This has been done in the lab. We take the transport, SRNA, in this case it was bird flu. We have our bypass tools. We have our basic reporting tools, our payloads and our decoys. Does this sound like a hack that we would use on computers to anybody? Henry, we've messed around polymorphs, viruses, Trojans and other sorts of interesting things. Yeah, exactly, we build them. This is kind of how we build the damn things. We do it on the body. That's what we ended up doing. We took bird flu, SID, NRA. We applied nanotubes, so multi-walled nanotubes. So we have the ability to do different things depending upon what inferences we're doing. Fill the body into things as good. It's a vaccine. Propulsion system, tracking system, decoys delivers. This is fascinating stuff for the medical field itself. You start taking a targeted drug delivery. You start taking a look at what they're doing with cancer research, take a look at what they're doing with Alzheimer's, stone breakdown, the whole lot. The challenges at the moment on a lot of this stuff is you're still dealing with single machines. What they're now working on is making those machines more complicated. In other words, the machine has the ability to either combine an inference frequency in which case it can break down a bladder stone, but it still needs to communicate that which still means you need an antenna-based system, a propulsion system, and everything else. Think of this as an early version of a car. It had wheels and an engine, and that was kind of about it.  These are getting more complicated and more complex as we go on.  We don't need an entire medical facility to do this. We actually cheat it. We end up using, not just the arduinos, we end up using the hack RS and the blade RS for these things. This is where it's going. No, command and control. Purely command and gear. You've got to find a host or a system. We are producing them. I wouldn't even go down there.  It's way easier, way simpler. And if you want to do anything on the body inference, you start looking at the ones that are carbon-based using gold as the base underlying molecular architecture because gold is useful inside the body, whereas silver and some of the other ones can't be absorbed just as easily. Yeah, there's some really interesting nuances, should we say, when we're looking at it. So anyway, yay, it's coming to us. Yeah, GMO on steroids. It's not going to be pretty, which is kind of why we have this one. All right, so that's some background. That was all the fun stuff I have been doing. This is the stuff I'm currently working on. That's actually not too dissimilar from how I look in the basement sometimes. Yes, we're going to hack the brain. Again, for those of you who've seen the slides, I apologize. This is not how we're going to hack the brain. You can put a USB drive in, but it really doesn't go much further than that. That's a one-time use hack on the brain. Yay, but not good. This is Elon Musk's, or anybody else's. You like a complete twonk, but it is. This is a better version. So this isn't something that you can just do in the lab. You can go out and buy these systems. The one that's on the bottom left is one system that I have. It's a 14 point system. You can pull out from all sorts of signals on the brain. The one on the top right-hand side there is actually just a two port system. The one I use, and I'm using at the moment, is another pair of glasses. My old set of glasses is very similar to this, that I've taken some of the EEG monitors out of a couple of these. I've gone to the companies, and I've gone, hey, what is the best quality monitor you have?   They're anywhere from like a hundred bucks, or if you actually buy the actual dry electrodes and solder them up yourselves. If you don't have a solo, go to the freaking hardware hacking village.  Actually, what I want to see next year is somebody come with a death combat with an EEG meter on it. Do it. All right. So you can get it with two EEG monitors just behind the ears. It's effective, it'll pull signals out, but it's not effective enough to do some of the stuff we'll be talking about. Currently, what I'm using is a set of four. So I have two sitting on here on either side on this part, and two sitting in the back here. There is a bunch of resources being put out at the moment that actually will do an inference behind and in front, and also combining with the jaw. I've looked at it, I'm kind of liking it. It still makes you look like a bit of a wombo, but it's getting better. This is the end goal. This is mapping. That's actually my brain got a year or two ago now. Looking at a couple of sequences. I haven't put updated ones on there for a bunch of reasons, one of them being because I'm actually trying to protect some of this stuff. And the right-hand side, those are three different sets of graphs. Obviously, we have to train, we have to baseline with scientists at the end of the day. We have to get an understanding. Each one of those different colors on there represents me walking up to three different objects. One of them is my computer, one of them is my phone, one of them is a module for the car. What I'm doing is I'm training the system. I'm helping it to understand who I am, what I am, and why I'm actually focusing on something. In this case with the computer, what I'm trying to do is walk up and recognize the computer, have my brain recognize it. In turn, I also have the computer reading the signals that are coming from me and it has a baseline understanding of who I am. In other words, we basically and simply become our identity. We don't have to worry about passwords anymore. For 20 odd years, we've been bitching and screaming about them.  Why can't our very simple existence become the identity that we need? For logic here is simple, and it's two-fold. One, the object I'm looking at has to, we have to recognize it. It has to recognize us. Secondly, it has to have an underlying understanding as to who or what we are and it's the right individual. The way I have it at the moment is this. I've tested it in the lab. I've tested in the basement basically at a house, which is really my lab.  It's recognizing that the object I can discern is the computer, that the computer recognizes that it's me thinking about the computer because it has a correlating set of software on it as well. At which point in time, I don't need to worry about studying passwords. I have the same thing running on an Android phone. My normal work phone is an actual iPhone, but the phone I do all my testing on because obviously the architecture is the hell of a lot more open is the Android system. And the car module works about 60% of the time and I've managed to start the car in the garage a few times and almost put it in fricking gear. Not a good thing to do when the garage door's closed. So the logic here is simple. Let's get a little bit more detail on this one. Our thoughts, our general concepts need to be, we're at the stage, it's well documented that the brain can be used to identify thoughts, feelings, ideas. We can pull message blocks out of it. We can pull concepts out of it. We can pull objects out of it. But what we're now looking at doing is defining that a lot further. Take us as the authentication model, take us as basically our existence and everything else. This is where we are the last six months, which is why I haven't put this up and it's nowhere else. So obviously situational awareness becomes very paramount.  I came out of SkyTalks, there'd be F-bombs all over the place, I'm probably more civilized here.  There has to be a level of situational awareness, hence the drunk programming as well. There has to be an awareness of stress levels and all these other things. This is where there's a very fine balance between how many signals you pull out of the brain and how much you apply any filtering to them, any noise reduction to them, any analysis to them, any mean standard deviation to them, all of these other things at the moment. I'm running at about a 90 to 95 percentile. So the computer has to recognize me in a very, very narrow band, not just that I'm thinking about it. In other words, what I don't wanna do is walk down the stairs and every piece of equipment goes, hi Chris, how's it going? That's not gonna work, that's not gonna be pretty. What I have to do is walk down the stairs and go either I'm thinking of my laptop, I don't have to think about the word, I just have to visualize the laptop. That visualization is pulled in and because I'm pulling from four of the main cortexes and I'm amplifying and looking at the specific signals, what I'm trying to do is not just pulling the fact I'm thinking of my laptop, but I'm pulling historic data, I'm pulling situational data and awareness of that laptop out. So I'm basically putting myself in a position that it's mine. Now can somebody else walk down and do that? Yes, but there isn't the underlying signal that it's me doing it. And that's where those two pieces have to come together, which is where the computer looks at what it's getting in response from my signal from the EEG and then basically from the Bluetooth or RFID, whichever one I'm using and goes, oh, it is him, it's the six foot three hairy thing coming down once his bloody computer opened. It's kind of like Jarvis with a Scottish attitude basically.  Trying to work on the humans as one time use pads and obviously capturing what makes us us. What is this? Is it the consciousness thing? Is it a humanizing thing? Is it signal detection? In other words, on this one, it's removing the squirrel element because some of us in this industry, let's face it, can be a little squirrely. So here's what I'm kind of messing around with at the moment, better metrics. So trying to get that balance between how many data points I have to pull out from versus how many I need to get a decent signal. What I'm relying on at the moment is two things. One, the paramedic signals that are coming out. How closely can I get and pull those signals out? How much noise is around them and how much can I actually eliminate that noise? The more the computer understands it's me, the more it realizes as I walk down, I look at the window. As I walk down, I trip over my feet. I'm thinking about the whiskey lab. I'm thinking about other things. All of those elements can be pulled out as noise. So then it's focusing just on it, the computer. It basically becomes a selfish signal. The other thing I'm trying to figure out is how much other work I can do on this. In other words, basically anything else, come up to the ATM, go up to your car, any of these other things.  If you take the concept here and you start taking a look at nanotech and bionanotech, specifically the antennas that are being built, the biggest challenge we have is pulling signals out from the brain in a way that they are much more readable and usable. At the moment, I've got the glasses that make me look a little kind of dorky with the other ones that I have, but not too much. There's a couple of bloody wires running down the same with this little signal holding off the back of this one, but that's about it.  So that's something that we're working on. All right, obviously security. Well, it's me.  I'm trying to unhack it. I'm trying to reverse engineer it. At some point in time, what I do want to try to figure out is how breakable are we as humans? In other words, can it be replicated? Can it be replayed? Everything I've tried so far, the answer is no. As a human, every single time we walk up to something, it is a unique set of situations. We're basically walking one-time use paths no matter what we're doing. So how much of that can we use versus how much of that do we have to ignore? There's all sorts of interesting stuff on that one. And as it says, I got to try and patent this or do something with the damn thing at some point. So we're pulling stuff out of the brain. The next obvious question, can we put something back in? That's what I'm working on. What I have at the moment on the computer, stand back here. I can't do it, can I? Damn it, stand here. What I have at the moment, one of my other systems is reading what I'm doing. In other words, as I'm sitting at the computer, as I'm looking at the computer, as I'm walking towards the computer, as I'm basically interacting with it, with the headset on or with the glasses on. Typically when I'm doing this, I've got a commercially available 14 unit on that I'm actually messing around with. What I'm actually doing is taking a look at machine language and the predictive side. In other words, how can the signals that are being pulled out, the situational awareness of the system, knowing it, measured with the machine language and go, I know what he's gonna do next. I know what key's getting depressed. I know what system's getting depressed.  That's what I'm working on at the moment. At the moment, I'm getting about a 75% hit rate on effectiveness. So I'm training the computer to basically think as me. We test some really fun consequences and some really scary ones.  This is like, hey, go play with this too. Go research it, do analysis on it, go pick all the paperwork's up from the various different universities, from DARPA and everybody else that's working on this stuff and see what else is going on. The biggest challenge on any of this is, again, what we can pull out from usable signals. So again, if you combine what's going on with consciousness and with what we're doing with this and you combine this with what's coming out of the healthcare field with the carbon architectures and signal analysis, it gets really interesting. Now for those of you that are old farts here and were here at DEFCON in the early days, I had to put this one up. How many of you remember the matrix? Yeah, we were creeping around DEFCON 10 and DEFCON 11 over at Alexis Place and we went through the matrix time when everybody turned up with a pair of freaking neoglosses and the damn overalls on. I kid you not, how many of you were here? I know you were. Not many, can't you guys see young buggers? All right, yeah. All right, don't do it, it was bad. But it brings into that question of what can we write back in? Can we write any kind of situational stuff back in? At which point can we get to a point where the computer system itself is affecting our actions, our reactions and our perceptions of reality? Answers are, I don't know, but I think it's gonna be a yes. So at which point in time are we at tinfoil hat time? I can currently pull signals from about an inch or two away from the scalp with a system that I have. So I don't actually have to have something about an inch or two from the scalp, I can pull signals out of. We can detect signals six to eight inches from the scalp. Doesn't mean we can do much with them but we can detect them. But again, if you merge this in the architecture, I'm probably not gonna be too far off of a tinfoil hat, especially if we start letting the government play with them, which is why we have this picture, which one I get to drink. All right, the future, this is what I'm working on. I think one of my taglines on LinkedIn or something, on Twitter I think it is, is at some point in time in the next 10 to 15 years I would love to be able to retire to a nice AS 400 in New Zealand. That's what I'm working on.  Can we actually take much more advantage of what is nothing more than a bunch of basically electromagnetic signals going on? And if we can take advantage, what are we doing? Oh, by the way, I keep pointing up here, for those of you that know the brain, it sure as hell isn't just the brain, all of this other stuff that's in the rest of the body has a huge influence on what goes on, not just up here, but even some of the stuff that doesn't bother, these things have completely independent thought almost, let's be perfectly honest. So that's part of it. You start taking a look at where we can take this for the future, if I can actually get to the stage where I can digitize what I am doing with consciousness, guess what, it's ones and zeros. From a transmission standpoint, that makes life real easy to be able to move consciousness around. Societal changes, we start taking a look at who's going digital, who doesn't, why is the wherefores all this other stuff, so then the question becomes one, how far should we take this? Simple answer, we're already on the journey. The train has left the station. The question is, what are you gonna do? Are you going to get involved? Are you going to pick up, do some research? This is death con, we kind of buck the trend. Our job is to be the voice of reason a lot of times.  That's our job, do it, be those voices. So, wrap up, I like this quote, I've got it in most of my slides now. The measure of man is not where he stands in moments of comfort, but where he stands at times of challenge and controversy. We are there, we are there because we have this technology tsunami that's hitting us. We are there because we're basically boggling into this whole idea of consciousness and nanotech without considering societal or security changes. We're doing a lot of it without involving us, therefore get involved. Biohacking Village, basically start building some of this stuff, because quite honestly, this should be our future. If we do it right, there is so much good that can come out of this, but like anything in tech, we fail and we screw it up. This is how and what we are facing, it's going to be a little bit messy. Simple, this is my last slide almost, and I like this one, we need to work together. There's 28, 29,000 of us here at Vegas this year. That's a lot of voices, collaborate, work with everybody, communicate, peers, friends, families, the whole lot. This isn't just spreading the word, this is just how can I get involved? Sound good? Okay, final slide. As always, tribute to Douglas Adams. So long and thanks for all the fish. All right, questions. Everybody sat through this, it was fairly dry when you go for it. With the motion, so if you're mad at your computer that day because it's not working, is that safe for each question? All right, so the question is, does it filter out emotions, yes? So in other words, have I got the filtering in for that? I actually do have some of that, yeah. It's actually really interesting. Again, the cool thing about doing this, I would, if you're interested in any of this, literally go spend your money, go borrow somebody else's money. I'd say go grab a credit card on the internet, but we're being recorded. So, this isn't Sky Talks. Don't steal anybody else's credit card. But seriously, go out and do some research on this one. It's actually fascinating. Yes, as you walk out of the computer, you're happy, angry, sad. All those emotions, peak and spike, all the different stuff.  There's stuff in there with me with a couple of drinks inside me. Oh yeah, there's some fun stuff. So yes, it does change it. Because it's pulling in other signals. So even though I got the full running, what it's doing is it's also pulling in my history with the computer. So if I'm thinking about my history of the computer, and I'm not just thinking about walking up to the computer, one of the key indicators that I had was actually, I started to replay the memory of when I actually walked into the damn Apple store to buy it. And I captured that signal architecture.  And the timing's up. The problem is you're recording yourself while you're recording yourself, while it's, it's kind of fun. But yes, you can actually signal that out and go, that is my reaction to actually the acquisition of that computer, and I can normalize that. But you can also pull in other things as well. There's a bunch of other ways of normalizing stuff. Yeah, it's kind of fun. Questions? What do we got? Like one more? I got one more question. Yes. Different, yeah, oh no, it's an entirely different set of, yes, an entirely different set of patterns that's getting pulled in. Yeah, I've had her come down with the headset on and hers are completely different from mine. So it's really nice to look at the different humans. I've had a couple of other friends come over, different walks of life, different, all sorts of other things.   It's actually fascinating. So, but there's also some underlying stuff that you can normalize. So some of the thoughts on the computer are almost normalized, but the problem is the relationships of them with the computer and everything around them is some completely different. It's kind of fun.  Thank you very, very much, everybody. Much appreciated.


Welcome to another edition of JPL in 30, the highlights show for the Jamaica Premier League with just a handful of games to go before the playoffs, five teams solidify their positions in that playoff race with two of those teams, Portmore United and Cavalier, featuring the first of our double-header highlights. Let's check out how both renal contenders fared. Portmore in their full blow with Red Trim doing some defending, Cavalier shows a better energy in the first moments. Kyle Ming with the strike was going well wide and probably would have picked out a teammate at the back post. And then this moment for Martin Davis hesitated with the shot, pulled it back, and then this decision was the wrong one when Alex Marshall was free on the penalty spot. What Alex Marshall didn't know is that he would have another chance from the penalty spot. A minute later, but then Shanil Thomas cut on to his left-hand side. He was leading the attack for the Cavalier team. And then this moment, Steven Barnett bundled over into the box, and Alexis Perri points into the spot. Rightful penalty for Portmore, and Alex Marshall made no mistake, sent JD right the wrong way. And his left foot made no mistake. Third of the season, one nil Portmore after 36 minutes. This was the second half, the first minute of it, and Kadeel Irving, the captain of Cavalier, hitting the crossbar. They would find the frame three or four occasions. This moment inside the area, Davis just couldn't organize himself. His footwork wasn't the best, Davis, and his head couldn't find the right direction. Then this on the edge of the area, Howard letting fly just wide as Portmore continued to attack. Then Cavalier finding this opportunity, McCleary with the strike initially. The captain Irving driving into there, McCleary again, and then Calvin finding the frame of the goal for the second time, Cavalier. Calvin, their leading goalscorer, had the beating of Williams, but not the goal. Post. And it remained one nil. Then Portmore driving into the area, Alex Marshall looking for a second of the game couldn't find the target over the top for the Portmore captain. A last minute opportunity for McCleary, heading it over Williams, but again for the third time, the crossbar. Then letting fly Tyrone Williams probably was missing the target and Williams was making sure a clean sheet for him, and a big three points for Portmore United. One nil over Cavalier. Portmore United, one nil winners. Let's talk to a man of the match, the captain, the goal scorer, Alex Marshall. Yeah, today's man of the match, Alex Marshall, captain for today as well. You got the pivotal goal, but you did a lot of work off the ball also. How do you describe your performance? You know, it wasn't the best performance for myself, but you know, I helped my team defensive, so I'm grateful for that. And we got the three points, you know. Congrats to my teammates as well and the coaching staff. You know, you've been drawing a lot of games lately, but I'm sure this win has given you a lot of confidence beating a big Cavalier team, especially going into the playoffs. Yeah, you know, it's always good to be Cavalier, you know. Tough opponent, you know. They got a few chances, which could have, the game could have went either way, but you know, thanks to our goalkeepers and our defender, you know, we kept a clean sheet. And you're seen as a talisman for this team rightfully, so a player of your quality. What, how far do you think this Portmore United team can go? Are you eyeing the title? Yeah, we're eyeing, you know, the title, but you know, we want to get back into the CFU and that's the aim right now, but you know, we're just taking it one game at a time and hopefully being the top two are, you know, a chance for the playoffs. Excellent performance today. Great win also. Best of luck for this season. Yes, thank you. The win for Portmore United see them jump in third place in the standings, whilst condemning Cavalier to their second straight loss as they sit in fifth place. We take our first break here on JVL in 30. Stay with us for more right after the break. In 30, the second match of Monday's double header saw home side Arnett Cardi hunting back-to-back wins. The strike hit in the upright. Jaheem Thomas, he had an excellent probably just tipped at the end of it for the man of the match award. A fever chambers after nine minutes, put Humberline ahead. Against the runner play, you'd have to say acrobatic finish and Giovanni picking up the assist. And what a big lead it was for Humberline at that point. Fabian Reed had his chances. Looking to flick it over, couldn't do it.  Rashiki Kelsen, pulling that one all towards the back post and disappointed that his teammate couldn't find the finish. Then this, putting it through the salad of the Humberline defense line, and Rick Gordon, the recipient. But his delivery to Fabian Reed found the arm of the Arnett captain and the goal was disallowed. This one certainly counted. Shavonny Willis rising high to not down at Joel Jones' cross. And Arnett finally found their equalizer. This was in a 39th minute good finish by Shavonny Willis. He had an outstanding game, especially in that first half. We fizzed some really brilliant 30-yard crossfield passes. Jaheem Thomas with another potent strike, which was well saved by Barnes. Twice he hit the woodwork. Jaheem Thomas. And Arnett, they just kept coming. Here's another hit of the woodwork. This time it was from Shea Smith, the outside of it. And he missed his opportunity. He would later be substituted. Arnett would find their second. And this time no woodwork from Jaheem Thomas. Nice pass from Kelsen, who deserved the assist. And Thomas is pulling that beyond goalkeeper Barnes and into the back of the net. His fifth of the season. And Arnett goes with the 2-1 lead at the start of the second half. Humberlain had a free kick. Vanzi, comfortably over the bar. But Arnett, they were just too good. Kimani Arbonne came on as a substitute and produced this for Fabian Reed. Still has the pace in the legs, Fabian Reed.  And look at that for a finish with the left foot. It looks easy, but it certainly wasn't. Clinical finish by the seasoned striker. And they had the choreography and the dance moves as well to celebrate. Arnett Gans, but they weren't finished. They were hungry for more goals and a rise in the table. Fabian Reed freezing the defense. And oh, wow. Look at that finish. Just opened up his body so well. Look at this here. Just a staggered movement to freeze the defender. Pushed it through his legs. And Barnes had no chance. He was so exposed. What a finish. Second of the game for him. And Arnett Gans were three. Four one up at that point. The best was saved for last. Shavone Willis from the dead ball. Some 35 yards. Oh, look at that for technique. Barnes had full stretch and couldn't get close. Willis with a brace. His first two goals of the season. And Arnett with a 5-1 performance. Lige Williams is with our man of the match. It just had to be that man from the 35-yard mark. Shavone Willis at the mic. Yeah, I'm here today with Shavone Willis. Today's man of the match. He scored two goals today, but there's no doubt which one was your favorite. It has to be the free kick. It has to be the free kick. I've been practicing these free kicks every day after practice. So one day, one day, one more score up in tonight. And you practice free kicks. You practice those passes across the field as well? No, that's a natural thing. I think God bless me with that. So it's become easy for me to do it. A high five performance by playoff-bound side Arnett Gardens who picked up your 13th win of the season. While Humbleland continued to find the going tough, following your 13th loss. Much more action still to come as we go to another break. Stay with us. Tivoli Gardens kicks off our mid-week kick maligns United. Chris Taylor takes us through the highlights. Tivoli Gardens driving forward early into the area. Into Dunn and he just couldn't miss from there. That was his first goal. Bound well by Anthony Nelson who showed good strength to get past Barrett and then Samwell's went in wildly with the challenge. Just in done, he kept his composure. And that was number 12 on the season for him. His first of the afternoon and Tivoli Gardens with the lead after just eight minutes. Or should I say after 16 minutes? Two minutes later. Just in done from the penalty spot. And this was after maligns had the bitter of the play in the first 15 minutes with a couple opportunities. And Tivoli in the space of two minutes had a two-goal advantage. Just in done clinical from the penalty spot. Tivoli would well maligns would work hard to get back into it. Tivoli with lackluster defending. Both teams showing real signs of poor defending in that first half. Tivoli guilty on that occasion. Jason Wright slotting it away. It was Daniel Hardy's effort here to just nudge that away from Shaquille Jones. Slot that pass. Slot that pass. Richard Brown did Jason Wright. It was two-one at that point. And then it would be two-all at the start of the second half. Jason Wright putting that beyond Nicholas Clark who went the wrong way. That coming after 54 minutes and maligns were all square at two apiece. It wouldn't stay that way for long. Steve Clark off the bench. Look at this. A bit fortuitous that he got it back. But very composed with the finish just to dink it over.  Didn't make his body larger at any point. And Clark said thank you very much. A 68th minute finish. Clark's third of the season. And Tivoli Gardens had regained the advantage. They would hold it for just four minutes. Look at how Jason Wright allowed that to come across his body. The initial strike saved by Clark from Wright's left footed effort. But immediately following up off the head to nudge it into the far corner. Got a bit of the upright as well. And helping it in. A hat trick for Jason Wright at that point. That was after 72 minutes. And my lines had drawn level at three all. Tivoli had all the ideas. John with the strike off the chest of Harrison then forced to pass Howard Morris off the crossbar and the Dean Penny Cook in the right place at the right time. Brave finishing. Four on the season for the Tivoli captain and center back. Very close to his hand. Tyrone Robinson said no the head was enough. Tivoli kept on coming forward. He said four wasn't enough so they made it five. Rodico Wellington off the bench as a substitute. Combining with the other substitute, Horatio Morgan. Just the second on the season for Rodico Wellington. The 2015 is Premier League career. Tivoli is number ten making it five-three and they still weren't finished. Penny Cook lobbing that into the area and Horatio Morgan was brought down. Enrique Godn just moments before that should have probably have conceded a penalty again and it gave Justin Dunn the opportunity to complete his hat trick. His second hat trick of the season. His fourteenth goal overall and a big three points for Tivoli guns. They run away with this one by six goals to three over my lines. Justin Dunn, the leading goal scorer is today's man of the match. I'm here with Justin Dunn. What was the hat trick for you this season? More goals. What do you make of your performance first of all? I think it was a well played performance by me. As you see we start the game very early. I got two goals and then a lot more lines to get back in the game. We went two-two and from there with a rugged contest but we came out and get the victory. And I'm sure you're pretty proud of your team as well. It wasn't only you scoring a lot of your teammates. The substitutes also came in and did really well. It's a really well-rounded team at Tivoli. From week in training we were focusing on finishing and we came to the match and executed very well. As you see we get six goals. It's not easy to get six goals in this league against anybody really. Does that show that Tivoli is the best attacking team in the league? Yes, in training we build momentum and each game we take it step by step and the result will come. Well, excellent performance today. Another hard trick. Golden boot leader and best of luck for the rest of the season. Thank you. Your coach was a game of nine goals. I know no coach would be happy with that. What do you make of your team today? I mean we played in patches. We went down. We came back. We fought again. They went ahead. We came back again. But we were largely disciplined today. We made some horrible decisions in the back of the pitch. We weren't as tight in the middle of the park. In the end we gave up a lot of free passes. Three true passes and that hurt us really. Defensively I'm actually saying and commenting a lot there was a lot of individual errors that led to the goals. Is that something that you feel that you can get the better of in the coming weeks? Definitely. It's coming to the end. We would have basically secured our spot for next year. We would have to go back and look on the motivational levels. Check the team that is playing. See those who are not. And then see if we can now get some guys out there who will be willing and hungry and play to the end. Yeah but coach you know one positive of this game is the fact that you're still getting goals.  Is that something that acts before the game? Is that something that you think that you're really keying in on? I mean that's definitely a positive. We have been creating loads of opportunities. We created before Tivoli scored. We created about probably two or three clear opportunities. We never took them and they scored against us. But guess what? Those are positive. Some football games you win some you lose. Some you'll draw. This is one of them where I thought we never played for 90 minutes and it really affected us today. Well thank you very much coach. Best of luck for this season. Yeah man thanks much. How do you feel about the performance? Well first and foremost congratulations to the Tivoli gardens unit. Well tried my lines. Today if you realize it was really an open game based on the footing condition because the wet surface resulting a lot of errors resulting in those goals. But in the end we came out victors. Hang on you've now scored 45 goals in the league campaign at least six more than any other team and you spoke about your top before the game and you must be extremely pleased with how your attack is firing. Well pleased in the sense yes but when you analyze the game how many chances we miss and those are clear cut chances you know they were created not only in the 18-yard box but also the 6-yard box but it's a work in progress. And you know going into the playoffs now you I'm sure you're putting a lot of fear into teams especially teams probably don't want to play going into the playoffs and in the playoffs as well. What do you feel about that? No you are you thinking about the playoffs I still have three more games so it's not to jump the gun before time you never take it in stride. I know you're into the top two now I don't want to go too far to the playoffs but now you're in the top two not only will they get to buy in the playoffs but also qualify for the the qualification for that. Is that a big deal for you? The country of Caribbean Cup is the top three that qualify not the top four. So you would be in the semifinal after Premier League and is you know how to take it from there to ensure that you're in the top three? Well thank you very much coach and best of luck for the season as always. All right. Yeah well it's the first time in over a decade that Tivoli guns have managed to score six goals and what a performance. This is how match week 23 has looked Portmore United with a 1-0 win Arnett with a 5-1 win a big win for Mobile United 2-0 over Dumbledore Holden they go into seventh spot and Dumbledore Holden dropped to eighth in the table from sixth Harbour view with a six-star performance big moments for them against Lyme Hall and of course we spoke about the Tivoli 6-3 win over Malines Waterhouse currently lead Treasure Beach by four goals to one a hat trick for Andre Fletcher so far in that one and Gibison getting the other four goals so Waterhouse currently sit in sixth position Mount Pleasant well yeah they were held goalies against Vera United that was played at the Wembley center of excellence Mount Pleasant as you can see here will still lead the table and now by five points the extended lead at the top did have an opportunity to go seventh but now just a five-point lead Tivoli Gardens coming to that second automatic spot 25 their goal difference and 45 points four teams tied on 45 points from second to fifth Cavalier dropped to fifth now Portmore and Arnett third and fourth respectively as you can see now Waterhouse in that sixth position and look at their goal difference compared to Mobile and Dumbledore Holden so three teams tied on 34 but the goal difference a big deal for Waterhouse and Marcel Gale we've seen three hat tricks this match week Andre Fletcher just in done and Jason Wright because goal scoring for quite a few of the teams Arnett Garnes as well with one of their best displays attacking wise this season Treasure Beach and Lyme Hall on the losing end of things still in the relegation and at this point they will be heading back to tier two twelve and seven points respectively Harbourview with a big 16 win day did their goal difference some justice carrying it now to just minus two but 22 points no chance of really making the top six that's how we put a wrap on JPL in Turkey on your home of champion sportsman see you next week for another exciting edition