I'm going to be talking about the intersection of sort of my two main affiliations now, which are the Open Connectome Project and the Institute for Data Intensive Engineering and Science. And that is data intensive computing, which is the subfield of high performance computing that's focused on problems that are IO bound, that are limited by your ability to examine and process data. So let's start with what the Open Connectome Project is. So it spans data collectors, data scientists, machine learners, computational vision, and statistical data processing. The goal is to create a computational forum for data analysis of massive data sets. For us, this mostly means 3D electron microscopy mouse brain image data to date, although we're not at all married to that data. We have mouse monkey about to have some human brain data. We have clarity data. We have array tomography as well. But what we want to do is take these massive data sets, so right now we're at 100 terabytes of data and link them with high performance computing in a way that you can do computational vision at scale to extract brain structure. We also want to provide the ability to examine this brain structure through spatial queries that ask for things like clustering, distances, volumes of structure. And the ultimate goal is to create a fully annotated graph that you can do statistical analysis on. Now so my research agenda, I thought it was my research agenda. It's not. Barack Obama made it clear that I'm doing his research agenda in the last two years. And this has happened in March of a year ago. He made an announcement about big data and the way that the Office of Science and Technology Policy works in the US is kind of interesting. So they're actually in office with no authority. They just give a mission and say a number and then the world scrambles to make it happen. And so the president said, do you big data? I said, ooh, I've been doing big data for 10 years, that's great. And then the president this year said, do brain. And this is either a large initiative or a small initiative, depending on which side of the pond you're from. But it is the US initiative. And so I'm being driven or pulled asunder, not by my own will, but by the president. And so brain data is big data. I think we know this, some numbers that really reinforced it. We can talk about the sort of fundamental size of the brain, 100 trillion synapses, 10 to the 15th neurons, sorry, other way around. But I also like to characterize what the fundamental data size of the task of imaging the human brain at, let's say, four nanometers by four nanometers, which is a resolution where you can see all the fine scale structure arguably that you need to. And that's about an exabyte of data. And so that's a lot of data to manage. And that's the size of the data we want to process. Right now we have several projects that are imaging data at about a terabyte a day. And this data is really rich and high-dimensional. It has, it's 3D, but we also add time series and multiple channels. And there's many imaging modalities and it's multi-scale. So this is just a daunting data problem. This is a movie that I think gives a sense of what this scale looks like.  Now we store about 11 zoom resolutions for this data, but this is all about 10 or 12 orders of magnitude. And so it gives a sense of really just how finely detailed it is and how much data we're going to need to accumulate to get from this level of detail to hopefully not Bobby's brain. Bobby is a postdoc in Jeff Lickman's lab at Harvard and is one of our key data providers. Okay, now I'm going to change gears and I want to talk about big data and data intensive computing. I mentioned that data intensive computing is this data analytics high performance computing field. I equate that with big data. And that is not necessarily a reasonable equation, that there are many other things that are big data, machine learning, scalable statistics, sensor fusion, semantic integration, and I need to be cautious not to use big data to mean just the problem I care about, which is processing vast amount of information. We have a history of big data at Johns Hopkins. I very fortunately walked into one of the great data science projects that was going on, which was started at Microsoft Research and Collaboration with Johns Hopkins by Jim Gray, who was a former intern award winner. And we built this machine in 2006 called Gray Wolf, which was really the model for a data intensive computer. It stored a petabyte of data at the time and it could look at 80 gigabytes of data a second, which was a really staggering rate in 2006 and still is. And this won the Storage Challenge Architectural Award at Supercomputing. And every great computer needs a great application. And so this computer was co-developed with the Sloan Digital Sky Survey, which is a catalog of the output of the Sloan telescope. It's a map of the sky in the visual frequency. This was done by Alex Soleil and this has become the most important telescope in the world in the sense that you can always get time on it. There are never clouds. It's never day. We have 6,000, sorry, we have 15,000 registered users, which is notable because there are only 6,000 professional astronomers in the world. And so this Sloan Digital Sky Survey is really a archetype of what a good data intensive application is. It's taking a massive science data set, imaging data. It's processing it to enrich it, pulling out things like clusters and frequency filters and doing image processing on it and then providing it to the public through a web service where they can do deep analysis and discover lots of new things. It's been the basis for projects such as Galaxy Zoo that's created 20,000 Galaxy classifications. It's a crowdsourcing project. So this is where we started with data intensive computing and at the Institute for Data Intensive Engineering and Science we've branched out and the model is to take a scientific domain and the capability to do deep data processing, combine them in a way that meaningfully transforms that domain. And we've done this to, this chart is just a sort of family tree of the open connectome project. We've worked on oncology, half of these databases are related to astrophysics, environmental science and the father of or the parent of the open connectome project is the magnetohydrodynamic database, which wouldn't necessarily seem to have a lot in common. This is a database that captures a high resolution output of a magnetohydrodynamic simulation. It has much in common in that the data is both high dimensional, spatial, high resolution and needs to be studied in depth. And I think it's a good example of what the capabilities you'd like to provide for connectomics and I point out that studying data after it's been generated, whether it's been imaged or simulated has great value and we found this in magnetohydrodynamics where we were able to go back and look in a dataset that was collected for a completely different reason and identify and verify stochastic turbulence in the data that just was there. It was an 80 year old hypothesis, it just got resolved, we published it recently and I think that all I did for this result I can barely understand the paper. All I did for this result was build a data system that allowed people to study the data. And now we have a next generation machine and this is the machine that hosts our connectomics data, this is the data scope. So it is a 10 petabyte machine, it does 500 gigabytes a second of IO and this is for a machine that only has 90 compute nodes. So it really is a data intensive machine by that definition. So for comparison, Titan, which was the world's number one supercomputer until really recently, does about 720 gigabytes a second of IO and it's about 3,000 times the compute power of this machine. So the point of this machine is to be able to access data and bring it to processors where you can run streaming algorithms on it. And we have a 40 gigabit per second upload, so this is the workhorse for all of our datasets. We collect datasets from people that are willing to contribute them to Open Connectome Project. We ingest them and make them available and you can run analyses on this machine. All right, so now back into the Open Connectome world. I wrote the following language recently, which essentially is my goal about the Open Connectome Project, which is I'd like this compute infrastructure to be a place where I can put my statisticians and my experimental biologists together. And so that's my goal, which is I want to write software that people use to characterize brain structure. And when people use this software and do interesting things, I've succeeded. Now that is my goal. That is not necessarily my collaborators' goals. As I mentioned, this goes from people that do electron microscopy to people that are just statisticians. I do want to point out a couple of my key collaborators. One is Jacob Vogelstein to maintain this very US-centric theme that I'm presenting. His goal in life is to mitigate neuroscience surprise. So this is to create biologically inspired computing architectures that keeps the US defense infrastructure ahead of the world when it comes to tasks like computer vision, image processing, anomaly detection. And then my other key founder in this is Josh Vogelstein, who's a statistician who refused to take this question seriously and wouldn't give me a statement of purpose. So this eye chart is the founding figure. And I include it to confuse, which I think it will accomplish, but also to give a sense of the depth of the pipeline that we, what we put the data through at Open Connectome. And so this chart is organized into sort of three layers. One is workflow at the top, and the second is sort of our data products. And then the bottom is how users, public users, interact with it. And so we'll actually stream data from the internet into our computer where we'll do alignment and color correction. And then having done that task, we'll both pile the data for a sort of Google map style interface, as well as build volume databases, which are our key data products. There are these databases that, rather than representing the data as image slices, represent the data as a bunch of small compact cuboids. And it's effective for reconstructing sub volumes, sub dimensional regions, extracting regions of interest. And that's actually really the goal. Once we've done this volume indexing, we can run vision algorithms against it that will do things like detect synapses, do automatic segmentation. And then the products of that vision will be co-registered with the original data. And then you can do statistical analysis and hopefully do graph extraction. And we have a complete pipeline working. The graphs it extracts, you don't love yet, but what they are, is they are big graphs that were extracted from large data sets. So we provide a handful of services. These are focused just on the data services. And to sort of understand the Open Connectome data infrastructure, I like to break it into sort of three tasks. One is to cut out data. This is to subset a region of image data or annotation data, which are labels on images. And to look at a specific range of data or a specific subset of the dimensions. To annotate data, after you have cut out data, you might want to apply labels to that data that describe what it is. So we'll look at all the synapses in an image and then label those regions as synapses. And that is labels in both to a spatial database as well as creates objects that represent the metadata in an object database. And then you can query that object database, which is to ask questions about the objects themselves. And that's where you can do things like come up with spatial queries and look at volumes and distributions. Okay, so with these services, we can then build data products. And sort of the, I'm gonna present a couple data products. The first one being a Google Map style interface. So this actual cat made is the viewer we're using here was developed by Alberto Cardona and his team at Janelia Farm. We use cat made as a front end for our database so we can dynamically load data from any of our database into cat made. And this displays some of Jeff Lickman's EM data overlaid with annotations. Those annotations are themselves a database and those annotations are linked to objects. And so this slide shows what object representations provide. Once you've represented all your data as objects, you can then query the objects individually. In this case, I'm showing you a full field and also a query, this is a sample restful query that you don't need to be able to read. It's too small for you to read, which is probably intentional. And that just extracts the dendrites. And so you can do a task like this, which is you can look at the data in a field. And then you can say, I just wanna look at the subset of the data in the field that are dendrites. And then you can say, how long are they? How big are they? What's their volume? How far away are these dendrites from the synapses that connect to them? With the same registered data and cutouts, we can plug that into 3D visualization frameworks. These are just kind of fun to look at. This is one of the ways that we found that's useful to visualize the data is to take a 2D image plane and then essentially pass that over and build up the annotations and raycast them as we do it. And then this other video shows that these annotations are actually objects and that you can manipulate them as objects. That you can click them and exchange your metadata. This screen will flash out. And then we can look at the objects themselves and you can highlight individual objects. Now these are interfaces to the data products. But the data products really drive the capability. This is our MATLAB API. And I bring this up because we have a companion poster that will be presented tomorrow that talks about how you can use MATLAB to interface with all these services and extract data from the Open Connectome project into MATLAB and essentially write your own scripts. And the goal here is we provide both Python and MATLAB bindings to all of our data products so that you can program in the environment that you'd like and use massive data and distributed data sort of seamlessly. So there's a whole poster about this, I'll not dive into that. Okay, so those are our services and some of the data products. With that I'll give a couple examples about how you might use these data products. So this movie shows a subset of 20, actually 19.5 million Synapse Detections that we took from a four-ter voxel or that's a four-terapixel volume. This was done on data collected by Dobby Bach in Clay Reed's lab. The way that this was done is by taking this entire image region, breaking it up into cubes, and then processing those cubes in parallel. So we ran this on 192 cores, it took slightly less than three days. This was an exercise in scalability. So architecturally what this looks like is that we have this data cluster that's running on data scope on our large machine at Johns Hopkins. And then we connected that up to a small compute cluster at the Johns Hopkins University Applied Physics Laboratory. Which despite having the same name as a prefix, Johns Hopkins University, and only being 30 miles apart is an internet distance apart. Which means that it has internet latency and internet bandwidth. And we're able to demonstrate that you can do deep data processing by doing cutouts across the internet, doing processing locally on your HPC cluster. Finding those synapses, uploading them to a database, creating a new derived data product, and marking those as annotations. And so this is sort of a visualization of the way that that pipeline runs. It shows the data overlaid with a whole bunch of synapse detections, and those synapse detections are the small colored dots. Another example of what we can do with the Open Connection Project is scalable image processing. We can take an entire image stack and we can run algorithms that process the entire stem image stack and do so globally. This example is going to be doing color correction, which is a approximate iterative global Poisson solver that was implemented by one of my colleagues. And the goal of this color correction, the left hand panel is the image as collected in the Z-stack. And you can see that there are anomalies in resolution and not resolution in exposure from slice to slice. And then the right picture shows that these should be cleaned up at the end of solving this Poisson process. What we do is we smooth all the low frequencies, but that produced a relatively low quality image. We then take the high frequency data and add it back in to both correct for the color and preserve edges for edge detection and computer vision algorithm. And this ran against four terabytes of data, takes about six hours to complete. So this video shows the outcome of the color correction process, which is the right hand panel is the original data, which is flashing. We've corrected that color in the middle panel, but that also mutes a lot of the high frequency, which is bad for algorithms that want to do computer vision. And then we add back in the high frequencies in the right panel to show the same process at a little bit higher resolution. Shows both how it washes out that the color correction process washes out the high frequencies, and that we do reintroduce them. And so this is one algorithm for the task. The reason it's notable is not necessarily that it's the perfect to write algorithm, but that it is a large data processing task that we can conduct on the server. So a couple of comments now on the design of the Open Connectome Project. The Open Connectome Project data architecture is what I would call a scale out no SQL architecture. So if you're into cloud, data intensive, data processing, those buzzwords all make sense. If you're not, no problem, scale out means that we're taking lots of nodes, putting data on lots of nodes. And then doing queries across lots of the nodes. So we can incrementally add storage. Right now, we're at seven data nodes. We're bringing another four online. Scale out often refers to hundreds of nodes in the cloud literature. I find that it's more effective for us to build high-capacity storage nodes and have tens of them. Our typical node is about 20 terabytes. And then node SQL refers to that it is a relational-like data store. We use the relational properties of data storage to get this cutout performance where we can both do dimensional subsets and spatial subsets of the data. We also take our data and we split it across two types of nodes that we have what I'll call standard storage nodes that host image data. And those are implemented with this storage for capacity reasons. We also have SSD nodes that store annotations. And the workload is very different. If we think of the Synapse finding example, when you do the cutouts, you're extracting terabytes of data. The data is spatially co-located. And so you want good sequential performance on high throughput and massive storage for writing all the synapses back. We need to do 20 million small random writes to 20 million objects. And so we do that by building a special class of node that supports really high throughput rates. This particular node, we got a million IOPS. That is one million IOPS per second out of commodity hardware. We're going to report on that in supercomputing this year. And so the data itself, most of the data we've seen has been 3D EM data and annotations. I also want to note that we do support time series data. And this particular picture is about flexing our multi-channel muscles. This is some rate tomography data from Steven Smith. And it shows that each different color is a different channel of the same data set. So we typically have six dimensional data where we have time series channels, projects, which is you may have multiple annotations of the same data set, as well as the imaging dimensions. We take that data, we get it as a whole bunch of images. Some of our biggest images are 200,000 by 200,000 pixels per slice. That's not a very useful image representation for us. We reorganize it into cuboids, which are just small, compact chunks of data in three, four, or five dimensions. We typically image x, y, z and time, but not channel or project for a variety of reasons. And then we build a resolution hierarchy of those cuboids. And so we have all this data in a compact, dimensional representation at multiple resolutions. We then take those cuboids and we order them according to a space filling curve. And this just says that when we lay the data out on all these disk drives, we lay it out in such a fashion that data are clustered by their spatial location according to a space filling curve. This has the property that it minimizes the number of IO requests. As an example, this image, which our service can generate on the fly in a couple hundred milliseconds, in the original image stack would take one pixel from one scan line from each image. And so the image that we can process, produce in 100 milliseconds, extracting it from the raw data would involve scanning over the entire image stack and take hours. And that's really the value proposition of dimensional imaging. So this shows the actual curve we use, which is called the Z curve, flattened into two dimensions. And the point of this is to show that the curve, the colors correspond to the location on disk. And so that this curve localizes compact regions of data onto small regions of disk. And this has a property of speeding up access to sub volumes by a factor of two at minimum to thousands, which was the example I just gave, where you're extracting a single pixel in every scan line. The next NoSQL principle is sharding, achieving scale out. So we now have a big data set. We want to get this data set onto multiple nodes. We need a principle for doing this. And the principle is sharding. This is also a term from the cloud community. And sharding typically takes an index and spreads the index across multiple nodes, and it's self-organizing. So that if you add more nodes, you can further refine the index. We do sharding right now manually in the application because we have the Z curve. That Z curve is the index we need to distribute. And so this figure demonstrates how we'll take a Z curve and divide a Z curve by ranges and place it on four nodes. So this process has been effective for us at both scaling storage and scaling throughput. We've gone from having data sets spread across one node to data sets spread across eight nodes. But it's sort of the most primitive example of the scale out storage principle, which is to be able to add nodes and not have to change your software or your configuration. And in the value proposition of scalable storage that is incremental and dynamic, and you can deploy it on the cloud. And that's actually one of our next tasks is to create a cloud deployable open connectome project so that we can host this data on the cloud or that you can essentially build your own open connectome data storage stack without using our hardware if you want. In adopting scale storage will also benefit us and that we'll be able to push some function into the storage system such as indexing in these cutouts. Right now we're evaluating two different technologies. One is Apache HBase. The other one is SideDB. These are both really great products that I think will be transformative to data management for connectomics and spatial data period. The state of the world right now is that neither one of them is complete for our feature set. And so I put them up there as ones to watch that we'd like to adopt for our design. Right now in Apache HBase and other NoSQL stores like that, such as Cassandra, you have the ability to do spatial locality and implement order partition so we can do the type of Z-curve indexing we want. But it doesn't have any computational capabilities. Whereas SideDB, which is the database that's being co-developed with the LSST telescope, has some really nice array storage and slicing and assembly capabilities where it can do this dimensional slicing and data assembly dynamically, but doesn't support ordered partitions. So I think both of these technologies are really quite promising for adoption. One of the hard tasks in dealing with connectomics data is extracting objects. So after we've done a vision, run a vision algorithm, and identified a bunch of objects, we need to then go find them. So this is essentially you want to query the spatial extent of a structure, which you need to do volume queries, you need to do distance queries. And it turns out that most traditional spatial indexing techniques, which come from geographical information systems and like applications, perform pretty poorly. And our trees is the variance or the classic method. And they have the property that regardless of which archery variant you want to use, when the bounding boxes of your objects intersect, the archery performance will degrade. So this is really a problem for neural structures because you'll have really long, sparse, skinny data. So for example, the largest dendrite in the data set that we store for Bobby Kasuri and Jeff Likman has about, see if I'll get this right, about 10 million pixels in a 100 billion pixel region. So that's less than 1% selectivity. So we need an indexing technique that can dive into this data, find these objects that have large spatial extent efficiently, and our trees don't work. So we're exploring some sparse indexing techniques that are also built on this Z-curve technology in which we look at enumerations of the Z-curve and represent that as an index. It says this object exists only in these cuboids. And then we can just go read that subset of cuboids.  I don't love these. They're not optimized. But what they do demonstrate is that on a single node, we can do 150 megabytes a second of data for cutouts through memory and over 100 megabytes a second when we have to go to disk. So this means that when we can pose these nodes, we can right now reach a gigabyte a data a second, which is a it's not optimized for the hardware. It doesn't compare well with this 500 gigabytes a second that the machine can do, but it is a high throughput data processing task. Now, the scalability is performing as a function of what you're doing. If we're getting the data from disk drives, it's IO limited, and that's going to scale with the number of disk drives we have. And so that scales on a per node basis. If the data is in memory and you're cutting out data that you've already looked at before and it's in cache, then the overhead is the reassembly of the data from its cuboid representation into its reorganization based on your query request. And that's memory bound and scales with the number of cores we have. All right. So I've been at this for about two years in the Open Connectome project. I'll start at the bottom. I love this project. It's great. This is the best scientific data I've seen. It's my fourth discipline. I'm done with astrophysics. I'm still a little bit into computational fluid dynamics. But what makes Connectomex data so wonderful is it's got really deep and rich pipelines. We're talking from instruments to visions to analysis. And that just the data processing task requires that the data systems connect all these different communities in a meaningful way. And that makes it a really cool infrastructure to work on. And the data themselves are very complex and not complex in one way. They're big, they're diverse, they're sometimes sparse, there's matrices, there's graphs. And I'm just talking about one type of data, not talking about multimodal data and anthologies and atlases. And so there's a whole level of complexity that we're not even touching on in the Open Connectome project. And the neuroscience community gets big data. That one of the things that came out of the brain initiative in the US is that the money and the interest is not in doing data analysis right now, but in developing the tools to do data analysis. And that's actually the challenge is we need informatics tools before we solve the problems. We're not ready to just go solve the problems. And so a really wonderful topic area for me as a big data scientist. And I'll finish with my team. Most of this talk has just represented people or tasks that are at this data infrastructure and data management. But again, it does span data collection. We work with the Deseruf Lab at Stanford and with Stephen Smith and with Jeff Lickman and with Janelia Farm. We owe a lot to the people that develop software tools, such as Stefan Solstheild and Alberto Codona, and to people that do statistical analysis, such as Carrie Prieb and Joshua Bugglestein, and many others. So it's a really wonderful community. Thank you. Thank you. Thank you for providing Formula One car for neuroscientists. OK. Now it's time for Christian and comments. Any Christian or comments? Thank you. So spectacular and amazing and fantastic talk I've blown away. So one of the things about neuroscience and especially about neural circuitry and neuroanatomy is that the features that a lot of people talk about at a high level are ensembles of all of the stuff that you're putting together at the low level. And so listening to you present your talk, I'm struck by how informative your work could be to us and to everybody else in the community who hasn't kind of dived down into the depths of the individual synapses yet in terms of if you had access to all of the data down to the four nanometer scale level and you could represent everything else, could you help us develop and think about the kinds of representations, for example, of neural circuitry based upon how these things combine?  Your system and the data that you have within your system kind of provides a really straightforward vehicle to kind of start thinking about how one would build quantitative functions to do that. So could you comment? Could you offer some insight? What do you think? So that's a wicked question. So I'm going to answer it a little bit backwards, which is about what we build and deploy and how we go about it. I would say that our development strategy is immersive in that we work very closely with people that collect and analyze data and that the world is better, in my opinion, when those people are different people and we're emerging or we're converging to that point. I will build a function for anyone that will use it and that actually is a pretty extreme gateway that having data is not interesting for entry into Open Connectome. If you have data and you want to analyze it in a meaningful way and use scalable services in a way that you can't analyze it without a data-intensive computer, then I care. And so a lot of the tough issues, representation, metadata, searchability, querying, I feel like my project is very primitive at when I compare myself to what I've seen in the poster session and what I know of my collaborators here. But there's a lot of people that are working on multimodal data and representations that are thinking about it at a higher level and doing a better job. The focus for us has really been in supporting vision because in this particular world that is electron microscopy, connectomics, we have a throughput problem in terms of representing segmentations and detecting objects. And the algorithms that are for it are not very good or accurate. So I'm asking questions that are how can I build the best system to support computer vision? Now that is a very narrow view of the world and my view is getting bigger as I come to understand that multimodal and multi-scale and physiology all need to be part of this data environment. But I'm gonna have to work in a bigger data community than I am right now to solve those problems. Thank you. Hi, of course I would never question your sincerity in your project. But as I see all your petabytes of data and your storage facilities, a kind of question, what is the ownership of this? Okay, so I appreciate that you've never questioned me and as does my mom. I'm a man of marginal integrity. Okay, so the machine itself has been sponsored by the Sloan Foundation and the National Science Foundation. It was built as a data scope and so this is an analogy for a microscope. It is a machine that is to do deep analysis of massive data sets. And so connectomics is one of the named applications, one of six named applications for this machine. So right now I'm in this luxurious place where if you have data in a problem, I can send you a whole bunch of disks or we can stream it over the internet and I can take the data. That's not a sustainable model. The question of how do you sustainably fund large scale data management is I think one of the thorniest questions in big data science. I participated a lot in looking at data sustainability proposals or data sustainability as part of proposals and centers and the answers are never satisfying. The only thing we have going for us in data management when it comes to sustainability is that the data that we're processing today is going to occupy just a small corner of the next machine we built. So let's take the Sloan Digital Sky Survey as an example. This project is now almost 20 years old. We just got a sustainability grant to keep it alive. So that's first of all not the greatest thing in the world. You have to get a sustainability grant to not drop the data on the floor. But the first data release of the Sloan Digital Sky Survey was 600 megabytes. The second data release was 1.3 terabytes. And as long as we're living in this exponential world, we can sort of skunk works our old data sets into the corner of our next machine. Don't love that answer either, but it's an answer. And the NSA are not tracking our every move into your database. Into my database? I have no idea. Have you ever considered using crowdsourcing or was there a need for that to do the imaging segmentation? So crowdsourcing is I think incredibly important to addressing this segmentation issue. And this is something that has been expressed as a thesis in iWire, Sebastian Sung's website is all, is about that. And I think that people that do automated segmentation algorithms and use them right now are in a place where they feel like crowdsourcing is the way to get this data that is high enough quality to be used. Our project is currently doing a crowdsourcing synapse classification task with I think the number is 400 high school students. One of the big questions about crowdsourcing is how do you capture the hearts and minds of people to log on to your site and do the task? By routing it to high school students, we have a captive audience. They require a little bit higher touch. If we take astrophysics as an example, Galaxy Zoo was a crowdsourcing platform built on top of the Sloan Digital Sky Survey that was incredibly successful. But astrophysicists are some pretty dedicated, strange people that don't value their time. So I can't speak to neuroscientists, but they may be pretty good at organizing their time. I certainly hope not. Okay, you got the rest? Just out of curiosity, I mean, you've been probably looking at a couple of data sets over and over and over again. Is there any point where you're saying, look, reanalyzing this chunk of data doesn't make any more sense. We need fresh data to get something new out, which in a way is there, in your mind, a point where it's like, look, put that data to rest. We need new stuff and let's not reanalyze because we're not getting anything new out of that. So no, there certainly is a need for new data. And the notion that you can make a single canonical data set that will answer all the science questions is nonsense. I am amazed at how quickly data becomes irrelevant in this field, just in the connectomics world where you have preparations and stains and polymerization techniques and some mesoscale, light-based techniques are gonna compete. But so all of that is true and there may be a case for discarding data. I feel like for the open connectome project, the criterion we need to justify its existence is that individual data sets that capture interesting features are too large to analyze on one or multi-core machines in a workstation environment or a lab. Okay, thank you, Rector. Thank you.


Welcome to All Hands Update, I'm Petty Officer Brandi Wills. These are your headlines from around the fleet. The Navy released a new app to help sailors and civilians understand why energy conservation is critical to completing its global mission. Energy Warrior highlights Sailor's efforts in taking innovative steps to conserve energy. The app also provides facts about worldwide energy use, U.S. oil production, and the Navy's ongoing energy projects. Check the Energy Warrior website for more details and to download the free app. The Navy's semi-annual Navy Family Accountability and Assessment System, or ENFAS Verification, is underway. All Navy personnel are required to update and verify their contact information and family member information in ENFAS by September 30th. Log on to the ENFAS website today and update your information.


Drone footage showed wide-scale destruction across Al-Zahara city on October 19, as Israeli airstrikes continued to target Gaza point for residential buildings were destroyed in the city, located just south of Gaza city, forcing hundreds of families into displacement. According to Hamas media, Israel began its most intensive ever bombardment of the Gaza Strip, a 45 km long enclave home to 2.3 million people, after the militant Palestinian group Hamas rampaged through Israeli towns, killing 1,300. Until on October 7.The Israeli military said last week all civilians should leave the northern half of the enclave, including the main Gaza city, as it prepares a ground assault to wipe out Hamas. Israeli bombing has killed more than 3,000 Palestinians in 12 days.


righty! I have a construction update for you guys today, so this was filmed Monday. As you can see, Backlot Cafe is under a pretty serious renovation, so for those of you that have been keeping up with my videos, it is speculated here at Amusement Insiders that they're going to set that restaurant up, similar to Lazy Bear Lodge in terms of the styles sliding down, picking your main and then your sides and all that. Secondly, the drill bit that was doing the reinforcement of the first tunnel is now off-property, so they are done reinforcing that tunnel. It'll be interesting to see the next steps, but from today's construction update, it looked like they were removing the support structure from the inside. Nothing happening over at Extreme Skyflyer's Potteryland, nothing notable happening at Front Gate, but this giant excavator is now parked down by the tunnel, so it'll be interesting to see if they're going to go out and rip the old tunnel apart now that they have the new reinforced tunnel, or if they're just going to build the reinforced tunnel around it. Now, I'm expecting that they're going to go in and remove this tunnel, but that could very well end up being wrong, but as you can see here, the new steel piles are showing on both sides, so yeah, really curious about what's going to happen. As you can see, there's a guy in there removing. Here's Backlot Cafe, again, a very large dumpster and a very large team in there working really hard on that restaurant. A lot of money going into Canada's Wonderland this off-season. Again, we had a lot of money last season, but this off-season, it's even more, so it's pretty crazy that Wonderland is getting so much money thrown at it in the last two years, and it's only going to continue, guys, so I'm very curious to see what this park's going to look like in the next five to ten years, because if it keeps going, it's going to be a whole new park eventually. Over here at the Funnel Cake Building and Jester's Courtyard, some interesting work, very little, but the main fryer for the Funnel Cake Place was removed, so now that's making me question if that might be the new VIP area. It's a little small, so I'm not quite sold on that, but can't take that off our minds as I do think, slash no, it's going to be an existing building with a name change for the VIP area, so we'll have to see which building that ends up being. Over at the tunnel, they are getting ready to pour the side walls. On top of that, there is evidence that there is going to be a garage door system installed here, so this is definitely looking like a maintenance tunnel or a launch tunnel, but more so definitely to get supplies in, because if they're installing track in that tunnel anytime soon, you have to picture a track piece going in sideways into the tunnel, so it has to account for a lot of space. That first tunnel simply cannot house a track piece going in sideways, so again this tunnel looks like it'll be that, and they'll probably use it for shipping and receiving to get the construction materials inside the tunnel. With that being said, they are really digging deep on that old Thunder Run station, removing that foundation, so no, there will be no building replacement coming in time for the 2024 season unless some sort of miracle happens and they're able to build that extremely quickly, but I just don't see it. There's PDI Presley, the people who removed, not extreme sky rider, Thunder Run's exit is still demolished with no signs of rebuilding yet, and yeah, I'm trying to think if there was anything else. Definitely looks like the drink stands coming along nicely and the Tiny Tom's replacement, unless Tiny Tom's is the one going over to the funnel cake building in Med Fair, so we'll have to see. I do know that Grab and Go will not cease to exist, or at least what they sell at Grab and Go, so we can confirm that as well, and yeah, so I'm really curious to see what this is going to look like after they pour the concrete, because that'll be a telltale sign. Over in Kidsville, this is where a lot of work has started to take place, so this is where it's really interesting, so they are looking like they're going to start to repaint the helicopter ride, as well as brickwork around the little kids show building, and refurbishment in the store, or the craft building for Winterfest, so definitely a lot of work going on in Kidsville that's really interesting, not quite sure what it is fully yet, but yeah, so we'll be keeping an eye out here to see what transpires, as we have heard rumors, but these rumors were more so for something years out, so I can't see it happening already, so it would be really confusing if that's the case, it'd be way too big of a project. With that being said, I don't think there's anything else to report on, again, subscribe to our Patreon to support the channel and get these construction updates a whole day in advance, and yeah, that's it, thanks so much guys, have a good one, bye.


First question is from Ren Amy Maya. How do you make your biceps appear thicker when vealed viewed from the front? I've been doing wide grip curls and pull-ups, but only the peak seems to grow. Okay, so in some cases close to a camera like so That's an arm right there Justin always hide those bad boys So here's the here's the issue with some muscles. You can Target and shape because they're large and because they have attachments that are a little further away from each other for example Theoretically you could work more of the upper chest and the lower chest just because of the attachments here on the sternum The biceps it does have two heads to it So there's two main segments of the bicep, but they attach it kind of the same Place so thicker more peak really you're just developing the bicep now that being said yeah But what about like the brachialis? Yes the brachialis which runs underneath and then that's on like the side of the bicep I was just gonna say develop that it'll give you that illusion Yeah, cuz theoretically you could train the brachialis, which is also a flexor that Could theoretically give you a little bit more bicep thickness now. Here's what I like to tell people I like to tell people try doing hammer curls it works that muscle also a lot of people don't do hammer curls Hammer curls reverse curls if you don't have those in your routine would be a great exercise or even Zopman curls throwing those in there Yeah, so those three movements. I would do that, but this idea of changing the shape of a muscle with exercises is Something that people that love clickbait and to sell you things will try and do and the truth is if you already do Reverse curls and hammer curls all the time and you're very consistent with doing that then actually picking an exercise That technically isn't for that that's for the bicep that you never do will probably show more thickness in your bicep So if you're really into arm training and bicep training in particular Here's the things to focus on right? So you can focus on curls with varying hand position because the hand and you can see here I was moving my bicep affects the bicep so supinated versus pronated that also will involve You know break your radialis which is here on the forearm break your alice Which is under the bicep and then here's the other thing you can focus on elbow placement when you're doing the exercise so Exercises with the arm next to your body Preach your curls or spider curls where the elbows in front of the body and you can even do These chin ups. Well, that's a mass builder right curl grip right chin ups. I don't know. We're talking about. Yeah Keep talking about thickness and whatever Well, no, I'm just saying supinated grip right more bicep Here's the thing with all like as you can make a case for all these different exercises and the thing that I'm just gonna keep Hammering home is that it's the exercise that you don't do when you want to develop a muscle more, right? Yes, that we've talked about there. There's a hierarchy in in in exercises, right? As far as like these are more These are superior to others But if you do all those superior exercises, you've checked all the boxes for a long time Then simply doing an exercise that your body is unfamiliar. They're not good at yeah And you're not good at and and get good at that exercise is going to show change in development in that area And that goes for any muscle and so more importantly than you know Trying to make the case for this exercise targets this part. It's like, okay If you train these movements try training a different movement for that and that's always been my experience is I'll do this New extra. Oh wow drag curls. Let me try that. Oh my god It totally works and then you just talked about it today with your chest. Yes a perfect example of that Yeah, I bet if I just did that and I went to an incline dumbbell press to be like, oh my god. It's the same thing


Everybody I'd like to introduce John Sinoff to you and he is Going to explain to us how he ported netpst to the latest micro 32 soft CPU and how he had to modify the CPU before Being able to run netpst Good morning everybody. Thank you for attending so as Martin said going to talk about What I did as a hobby project recently and it involved Parting net BSD and HBSD which has been presented yesterday On an open source CPU called the lattice micro 32 So okay already been presented. So I will be joining M labs in a few days Which is was formerly known as milky miss community and which has been incorporated recently by its founder So that's a cool company doing interesting open source hardware and software stuff like the milky mist board if you Out of it or the milky mist system and shit. I will talk about it a bit later So okay, let's talk about this port and more specifically ported on the milky mist one board and First part I will talk a bit about the hardware. So Talk about the MMU and then a bit about software But first what's the milky mist one? It's this It's an electronic device aimed at generating video effects In which are in real time synchronized to a lot of sources like for instance audio input or MIDI or DMX events Etc. So it's a kind of a an artistic device That can be used a bit like this so you can film someone a performer and then you project against the wall at a party or at a concert and You apply nice real-time video effects like rotation zoom in zoom out or blurring, etc and you can interact with all kind of Party like devices like MIDI keyboard, etc So that's the device and it produces nice effects like these ones. Those are screenshots of the device output And the cool thing about this device is not just any close source commercial device because it uses an FPGA as its main components and most of the interesting part of the Device functions are implemented inside this FPGA. So it's not fixed in time you can play with it. You can modify you can do Your hack stuff on it and it's pretty cool. So What's an FPGA? So it's a chip basically like any other but you can Configure it to behave a bit the way you want. So it you've got logic blocks Located as an array and IO block and you can configure each logic block to do something like logic operation like add or XOR or stuff like that and then you have a switching matrix That allows you to interconnect those logic block and you decide which block you connect to Which other and then you can create basically a logic circuit and You can do this a lot of times and you can Basically implement almost for free an ASIC chip ASIC chip could cost usually thousands of hundreds of thousands of dollars So this The PCBs is an electronic circuit, which is also open source So it contains this FPGA and on the FPGA it runs the milk in the system on chip, which is also open source So it's a whole bunch of Function blocks which are controlling I don't know for instance the USB or the sound or MIDI DMX the UART etc rather DRAM or the frame buffer so all All those controllers are making the device work and then on the bottom left You've got the micro 32 CPU. That's the the main component. I will be talking about it's the the soft core CPU Which is running on this FPGA and which will be running that BSD So okay micro 32 so so little bits have architecture. It's a risk Big Indian that six stages. This is fully bypassed. So it's Pretty okay in terms of performance and you can put caches or not you can disable them and they are Up to two way set associative and it's using the wishbone on chip bus, which is an open source Specification bus which is used by all the open cores community of open source hardware cores Devices so you can easily find a lot of open source cores which are Talking this protocol this own chip bus protocol got tons of them So okay, and the CPU actually works on pretty much every FPGA because it's device agnostic It doesn't use a specific blocks forms one specific FPGA vendor for instance So you can for instance run this CPU on the D zero nano, which is running an Altera chip or on the milky mist one running Featuring an a spot and six FPGA from Xilinx Papillopro Or the mixio, which is a new project from M labs video mixer also open source and Or even the new Kintex 7 series from Xilinx So hey all of these could run and can and run actually run those days the micro 32 So why is it interesting? It's small. It really really doesn't take a lot of resources inside the FPGA It's all as I said, it's portable and it's pretty fast A lot of CPU open source also that you can find on the internet usually they run at 28 25 megahertz and This one can run up to 200 megahertz on recent chips It actually works and there are a lot of CPUs There's a hobby project that you can find on the web, but most of them a lot buggy if they actually do work in the end So it's one of the few that works and it does an okay software support We've got the GCC be new to GDB even QMU, which is really handy And it's open source No to the bad points It had no memory management unit, so I couldn't port net BSD But that's one of the few that was the first thing I I fixed I worked on this with the Mikael value. It's a two-man job and CPU is used in some closed source commercial asics, so it's pretty solid. It's been implemented in real Real chips and it can achieve a pretty decent frequency in quite all cell process so It's a standard Risk pipeline So just get the address computations and you've got the instruction fetch then decode execute loads turn right back So I think all of you already understand all of these So, okay This is a simplification drawing of the CPU The original CPU how it was working. So we've got the pipeline on the left Then you've got the two instruction and data caches on the right and then main memory is just off chip and originally there was no MMU and No notion of virtual addressing so physical address was used everywhere but directly talking almost to the RAM chip and There were no translations, so it was okay to run UC Linux for instance, which does run some extent, but As I understood it netBSD couldn't work in such a system. So We modified it so we added this memory management unit so that on the left pipeline only Manages virtual addresses and then it gets translated and goes through to main memory as physical addresses so in this way we were able to To have a something to run that BSD So, okay, but a bit about the MMU's job So it basically translate a virtual address into a physical address and it gives you also memory protection so For instance, it can allow you to say, okay, the stack is not executable and okay This is data so you can write through it or maybe this is read-only data so I don't want to allow you to write through it and It gives you this kind of security so We explain how it was implemented in this case. So This is the same diagram, but even more simplified so we've got CPU pipeline Giving some virtual address for memory management unit and it gets translated But okay, how does the MMU knows how to translate this virtual address into the physical address? How does it work? It's using okay, it's using the page table, but the page table as you can see is located in the DRAM so We couldn't access each time The DRAM to do a translation that will be terribly slow. So What was down, oh, no, okay first, sorry, why are we taking talking about page pages and First before I was talking about addresses. So why do I switch word? Because in reality You're not just translating addresses because if you were just doing a translation from one address to another Addressing like this. So for instance the address 4 to the address 1 and all 0 5 to 1 Or 1 you would need to remember all those translations as you would there would be a Lot of data to remember you would need to remember all All those translation line and for to translate for gigabytes you would need like for for billions of lines So it can't work like this. So To be more efficient use pages and then you take big chunks of memory and you translate those big chunks by what we call pages so And in this case where it's implemented as four kilobytes. So By just having one line you can translate four kilobytes from and it's more efficient like this That that's what everyone is doing basically So, okay So we're accessing this page table in RAM, but we're not doing it all the time because it's really slow Dram is really slow So we are going through that what's called TLB translation look aside buffer And it's just a cache like the instruction of data cache, but it caches Translation of virtual address to physical address so this way if TLB contains information you're looking for in just one cycle you get your Physical address translation and you then you can play with it. So it's pretty handy and this is a non-chip cache So it's really fast access Okay, and Then Okay, and then in theory The one when you are accessing the TLB, and it doesn't contain the information you're looking for because as a cache is just a subset It's pretty small You should have a way to then access release a page table in the slow DRAM to get the information and refill the TLB and Usually hardware does this so the TLB should directly go to main memory and Do the refresh but here in the design which I was chosen only for to be to be easier to implement and for simplicity We're not doing this. So the TLB won't fetch from page table directly. There is no hardware page through Walker So we'll need the help of the operating system here so in fact if the information is not in the TLB it will trigger an exception and For trapping to the operating system and the operating system will have to fix the stuff so to go read page table find somewhere to do it even if The MMU is off etc. And then once information is there To refill the TLB to sort of a date to put the mapping inside and then to resume What's what was going on? So it's a bit like what most mixes are doing or even poor PC bookie and It's not the the most efficient way, but it's easier to implement like this So the TLB is entirely managed by software Okay So as I said the features of this MMU Only use four kilobyte pages So it's not configurable. You cannot say okay. I want one megabyte page size and some so Quick question. I've got a 32 bits physical address If the page size is four kilobytes, how many bits of the address indicates the offset within a given page? well Depends what you mean by this Yeah, if you do a byte access you can Yeah, to to indicate enough set within a page. So it's the lowest bits There's alignments when you access DRAM Yeah, is there is byte addressing inside micro 32 and since it and you can be An align just for the byte addressing can be anywhere. So here's the usual case There's no trick in the question so So yeah, it's as usual. It's it's 12 bits for the for the 4k So I've got the page number on the left, which is 20 bits and the offset within the page Which is 12 bits So there are two TLB is to turn two software sort of caches for the translation one for instruction one for data and they are pretty big but it's to cope with the slowliness of Having to do it in software each time information is missing so we want TLB to be really big So that most of the time for lucky information is not missing and it's in the TLB so There are 1024 entries or ten bits to index those it will be Useful information for later So as I said no hardware page three workers, so it's still to be assisted So okay, we are something a bit like this So we feed a virtual address we feed if it's a load or restore a string share of data and then the MMU will answer okay, this is a physical address and Okay, I grant you the access for instance or I deny it and But here in this case since it's still be assisted We need a way to say to say I don't know because maybe the MMU doesn't know the answer And then that's where the software operating system part kicks in So okay now let's have a look inside how the TLB works a bit So I just walk you through it by Translating a virtual address So let's take for instance this virtual address a triple zero one double zero four I've only showed the three first line of the TLB so line index is just handy For the for the talk, but it's not actually any formation inside the TLB It'll be only contents of tag physical page number a read-only bit and a valid bit So how does it work? So first we split our virtual address into the page number and offset in the page The offset is not really useful for now. So we're going to put it apart Okay, so we've got our virtual page number that's That's the thing we are going to translate so we need to process a zero zero zero one How do we do it first we? Write it in binary and then before I said the TLB is undecided by ten bits. So We took we take those ten lower bits of the virtual page number and Those will index the TLB. So that will be So that will choose the line that is interesting To us. So here it's one. So we will choose the line Number one. So that's where maybe the information is so then Okay The valid bit is one. So okay, it's at least a valid information Then we've got also the information that it's a read-only mapping. So if the access is writing it will be denied for instance Then we've got the physical page number Okay, that could be our answer and then we've got this word information. It's a tag Okay Why I'm say I am I saying maybe it's information we're looking for because You saw that I only took the ten lowest bits part of the virtual page number to address this TLB so whatever the value of The first bit here, I would have chosen anywhere the same line the line number one. So we have Kind of a fight for this line. It could be a translation for a lot of Virtual addresses. So we need to check that that's will really the one we're looking for And to do this we're using the tag information here those ten bits. So we're taking the value 280 I would translate it in binary and when comparing we are doing the tag check and Okay, we see that it's the same So that's how we know that it's really the formation we were looking for and then the physical page number column will contain really our result so The physical page number is the P triple zero one and then to get really is the entire physical add-on we append the Page offsets So, okay, we've done the translation. That's basically how it works At the moment in the micro city to CPU plus MMU Except that now I needed to add an address space ID to the TLB to to make it work so But that's basically it It's okay enough for the Hello parts. No Just report to you how I am Managed to progress and Actually running that BSD kernel So first a very cool thing that I enjoyed is that Everything is kind of cross-compilation So I could work for instance on my MacBook and macOS and There was no issue with it. So I could just run this command And it would just generate for me a cross-compilation toolchain that runs on macOS But targets like this micro city to architecture and it was really handy Build.sh is doing it for me. It's pretty awesome too. I was really surprised. So I had to Hacked some make files there and there to to get it to to work because the architecture was not supported in the tree yet, but it was really Really useful Okay An issue I had was that The kernel is not linked against LibGCC So I had a lot of missing symbols when I was at the end trying to link my kernel And I was wondering why and in fact, okay, and it's too late. It's not linked with LibGCC So every time there was a multiplication or division of modulus operation was in instead of doing Inline the code was inserting calls to those utility functions and they were not linked with it So instead the kernel is linked with LibGCC and okay I learned that I had to go to this C-slip LibGCC and add the directory for my architecture and put it put there the The utility functions to do the mathematical operation Okay, then My first goal was trying to at least get a binary image to get it to link Even if it didn't work because first to to get some binary to try to run it and then try to debug So I just tried to to feel the include and configuration and Directories by copying a bit. What's what did what exists in the other architecture trying to understand how it works Sometime at first I didn't understand exactly What I was supposed to put so I kind of copied and and then for all the missing symbol I'll just put stubs because I really wanted to be able to to get this elf image And okay try to run it and then debug one by one all the issue and write the missing pieces so first up everything run this really simple command and Then in the end when I got the elf I could try to really implement something First thing I needed to debug. So I needed to print something on the console. So I did the very basic Console driver only for early prints and it's really not difficult to do You just declare a struct where you put your callback function for reading a character of writing a character to the UART and Then it will be used later on So I needed to implement of course the exception handlers that's the the first function is the reset handler which is executed in this case and then for That's the code executed when you've got a nioq or TLB miss exception or this kind of stuff And then At startup it's called the milky miss startup C code Then it initializes the console driver by using the previously written structure. So if you remember this milky miss com comes And it's pretty easy.  So by just assigning this structure, it's okay Then you can do prints and you can do easy debug So that was convenient Then pretty early in this milky miss startup function You need to initialize the virtual memory subsystem. So you do this by calling the machine dependent PMAT bootstrap function That's basically where you register The physical run which is available in the system. You say to UVM, okay I've got that much pages and starts here and it ends there and Okay, you can deal with it and you can do a location on this pool And then when basically those are very simple Initialization is done. You can call main function, which is machine independent and it's basically a very long list of subsystem initialization calls So then you're in the in the usual net psd kernel so But then you need to implement a few stuff like P-Map the P-Map subsystem as I said is the kind of the virtual memory system It's not so straightforward to implement But the really good news and the really good surprise I had is that I didn't have to do it and that's pretty cool The less you write code the less bug you have so since Mico 32 is software managed tlb and there is already code for that thanks to Matt Thomas I all I only had to use the files in ccv and P-Map which are already used in a poor PC bookie and Okay, the P-Map system system was done I only had to do the the first function which is just P-Map bootstrap, but All the rest was working fine so far So that's a good surprise then You've got some stuff like copy and copy all which are basically taking data from the user space process Copying it to kernel memory and the other way around Then you've got to implement atomic operations There is no atomic instruction in Mico 32 so and I by reading Basically the kernel code of NetBSD. I learned about this technique to the restartable atomic sequence for implementing the basic comparison swap, which is the Core of all the implementation of a mutex spin locks and all this stuff like this and so It works a bit like this. That's the actual code. I have implemented for the restartable atomic sequence so The interesting thing is that since there is no atomic operation in Mico 32 Obviously it takes here five Assembly instructions, so it's really not atomic at all. So how does it work? if I'm ever interrupted for instance by an exception or most likely an IRQ for in the middle of this supposedly atomic operation then in the return path when I will return back to What I was doing. So in this case this is a major operation. I Check if the PC return PC is in between the cast rest start symbol or the cast rest Cast rest and symbol and in this case instead of just returning to the PC to where I was Executing so for instance may be returning to the store world R1 Plus 0 R3 then I choose to return to the cast rest so I rewind I Go to start and restart again So I found it a pretty cool trick and I didn't know about that. So that's what I use Then you need to indeed at support for interrupts to handle them But even to let the driver register the interrupt callbacks So you need to be to do a bit of code about this and then you need the running system clock to Let the system be able to to schedule LWP is a thread stuff like that So so you need basically on the milky me system if there are two timers so far I'm using one of the two timers who do the system ticks and And there is a global symbol you need to implement. It's called CPU in it clocks and Its main goal is just to set up this so to active to initialize the timer to activate it to register the clock IRQ hundred and and do this kind of stuff and Then another thing is that your clock IRQ hundred needs to call a machine independent Symbol which is heart clock and that's how it plugs all together to to go from the machine dependent code To then the rest of the system to do the time accounting and the scheduling stuff So you need to plug to the heart clock system Then there is a whole bunch of all the functions that are not detailed because it will be really too long But you need to implement CPU switch to Which is basically the machine dependent function to switch from one LWP to another so LWP is lightweight process if I'm missing and Then a k copy to copy data Setful to save current context and the fork operation to create process and Really cool stuff I enjoyed about the NetBSD environment and ecosystem is that the small nine in the parenthesis Is that for kernel code function you've got man pages and you don't have this on all operating systems So I found it really cool I wanted to know how to implement CPU switch to what it was supposed to do. I don't have any clue I could okay read the other architecture But I I just had to type man and CPU switch to and I had my answers and a bit of how it should work So that that was pretty Cool So then other function I had to implement so SPL is to block interrupts So that in some critical sections, you don't want to be bothered by any interrupted so you can block them and then re-enable them afterwards and Okay, CPU start over basically a lot of machine dependent function. Okay Then I wanted to try to boot user space when the kernel was fully booting So to do it I had to create a dummy RAM disk containing only one binary which is in it Then I built the kernel with the MFS option, which is memory 5 system which allows you to embed inside your kernel image a RAM disk Basically, so a file system, which is which will be in RAM You insert in the kernel and then you try to boot it. That's the approach I had and That's basically where the process the progress of the project is right now It's the kernel boots and it's booting and it's running the init Which is a really a small handcrafted statically linked crappy init just printing a low world, but I find it Cool, it's at least runs. So, okay time for the demo as I'm saying it runs. Let's try to see if it's true Okay, so let's just compile it for the fan Just touch one file and then so I'm it's pretty easy to compile or just use build.sh and Should just compile all of the bugs Okay, so then You've got the kernel here Then you can use QMU which is really handy to debug to run this kernel So you say you want to run the milkimi system on chip and you select the I8 maybe it's a bit small Okay, is it better like this? Yeah, okay, so Select milkimi system and chip You say you want to use LM32 with full divider and multiplier and with MMU You don't need a graphic and you select where the kernel is so Select this kernel going to happen gdb and Okay, let's run it with gdb and That's really handy because that's basically how I debug debugging All the problems I had is that there is a gdb server inside QMU You can just attach it and look at all memory registers, etc So let's attach Okay, that's more what okay I forgot to do something. I just build the kernel, but I didn't put the ramfs inside So if I just hit continue it should just okay, so you see at least the kernel boots So you've got Okay, I've put a lot of debugging prints, but basically at the start it does the pmap bootstrap and it registers Using UVM page feed load it registers Available RAM So you could ram size you say it registered a bunch of pages in the sub system and then it calls Okay, then it It initializes a pmap module, etc. And then you call main. I've put a few prints of the main What's the main doing so basically it initializes all the subsystem and Then it's trying to initialize all the drivers so for now there are very few rival only the timer Then the UART and The clock which is the other timer and it's doing the clock ticks then you turn on interrupts and then it tries to find a rootfire system which is Not ah, it's not managing to find rootfire system. So it's not booting So now let's try to put a ram disk so so here I'm creating dummy fire and I'm putting Dev console character device I'm creating FFS and I'm compiling my init statically linked in it inside the ram FS and then using MD set image I'm embedding the ram FS inside the kernel. So now it should boot a little bit further so Let's run it again First let's see what's inside. Okay So you've got the main our may be in C. Okay, that's basically what's the init is doing It's really really almost nothing. It's just opening dev console For STD and STDL, STDR and it's writing a hello world. So for now, it's really a simple init stuff And it's statically linked so all the write and read and open functions are just in line assembly inside the init So that's what we are going to try to run. It should print hello world so There is a bug where after init it crashes so to prevent this I'm going to put a breakpoint on the last instruction of main Just for the beauty of not having the crash printed. So So this is a virtual address And now if I show QMU and I do continue Okay, so then you see what's more is that it found a Root file system which is of type FFS which was in the RAM file system then a bunch of warning And then it's loading init and it's running it and you've got zero That's it and then if you really want to see it crash you do continue And now you delete the breakpoint and you do continue and then you've got it Okay, so Okay So that's just a memory layout Which is I think it's the pretty usual one so three gigabytes for user space of virtual memory and one gigabyte for kernel space So on the milkems board there is only 128 megabytes of RAM And the interesting thing here is the RAM window I will explain a bit what it is So basically All the physical RAM has been mapped In this RAM window which is in the kernel address space So, okay physical RAM start as physical address 0x4 or 0 and basically all the beginning of The kernel virtual memory so starting at 0x4 all these beginning so from C0 to C8 It's in fact a direct mapping to the physical RAM So by using these visit physical addresses you can basically access all the physical RAM And that's really handy for some stuff So for instance here is how I'm Managing the page table. It's the pretty again the standard usual way. So you've got a page directory And a page table So if you want to try to again see where these virtual addresses you to composite and got You split it into the 10 bits 10 bits and 12 bits. You take the top 10 bits Which are three so you index and you use the third Row in the page directory. It gives you the address of the page table. You see C4. It's the same order and then you take the following 10 bits and Okay, one zero. It's it's true So you take the second line and this will give you we will give you the the physical Address and again you happen the offset Okay, this is a this is a data structure in the kernel to Really manage and remember all the mappings all the virtual to physical mappings but The kernel is running with memory the MMU on so it cannot Access to which to a physical addresses it could it can work So Okay, you think okay, I will put only virtual addresses in this structure this structure Okay, if put C4 to point to my page table and it's I am okay But I need also to be able to access this data structure from the TLB miss under and in micro 32 It's running with MMU off. So I cannot do the reference is kind of pointer and that's where the RAM window kicks in because Since it's a direct mapping. It's really easy to translate from one to the other by just doing a subtraction on addition so So I choose to put Virtual addresses inside the page table and everything so for for the kernel it's okay, but When in exception handler, I just do this computation So I can walk this page table from the exception alone with no no issue because I know how to translate So that's that was the interesting part about this Okay, it's pretty much done. So if you want to follow progress about this part It's really just the beginning since it's just Loading small in it and there is no leapsy port yet, and it doesn't run bunch of user space stuff So this is a static web page and giving the the progress But I try to update also the wiki page on HBSD website Which shows you all what's working what's not working and how you can bootstrap the stuff So get the the correct git repository Compile the tool to the cross-toolchain compile the gdb compile the q special QMU port etc So everything is explained to directly start developing on it Then I'm using I'm using git basically so I'm all my code is on github But I imported all the netbsd source without history. So it's not an awesome way of working So a better thing is the head HBSD git repository which has all the history which is based on your Mirror and that's there is a Short you ever about this So okay, thank you for listening and also thanks to all those people and More that I forgot who really helped me a lot in this not so easy task, but really cool and full of learning stuff Thank you Any questions? I think your TLB is very large. Yes compared to others Is this do you notice did you experiment with smaller one and see how the performance is affected? No, I didn't try actually that's a good question Yeah, it's pretty large, but what's cool with FPGA is is that I get it mostly for free Because when you're implementing this kind of cash is you're implementing basically SRAM inside the FPGA and You have special hardware blocks usually to implement those SRAM and anyway You're usually not using them. So when you do logic a Usually you do logic Multiplexers etc and you don't use the SRAM block. So they sit there unused so anywhere you can use them and You can get bigger caches and And yeah, it's cool other questions Could you explain the process of synthesizing the CPU core and world-building process from the scratch very long source or VHDL source to Building the kernel and a bit stream Is it possible to build a world soft CPU with your MMU extensions using only free tools? for example without Xilinx ISA or other proprietary Verilog synthesizers Okay Well indeed FPGA world is a bit sad because there is no full A to Z Toolchain which are open source So if you want to do real FPGA development You are forced to use a closed source vendor dependent a toolchain like as the Xilinx ISC or the Altera one etc So now if you cannot synthesize your system on chip Well, okay, you cannot get to the bit trim to the bit trim file with only open source Toolchain, but there is a bit of work in toward this direction and actually You can synthesize so only the first part of the of the work because you need to synthesize and you need to map to the Block technology and then you need to do the place and route etc. So the only the first step Of the pipeline it can be done an open source You can synthesize that is micro 32 or even all the making is the system on chip using your uses Which is an open source synthesizer so thank you and Another question Is it possible to run that CPU in simulation in for example a car was very low? Yes, it does work and it helped me a lot to debug the MMU when I implemented it I used several simulators, but I very low also and Okay, it's simulated also by QMU as you could see but it's not at the gate level But yeah, you can run it with the carries. Okay. Thank you so So how much work was it doing the MMU implementation? well In fact, yeah, it's in theory. It's pretty straightforward to design an MMU since it's just Well a cache with a bunch of logic around but in practice since I was really a beginner It's kind of my first FPGA project So took a year to implement and to test and debug etc But in theory, it's not that complex, but I was doing this project to learn. So I had my own learning curve about this Other questions, okay How much time did it take you to do the NPSD port? Well Well, as I said, I'm only doing this as a hobby project. So in my spare time and it took really a hell of a long time Maybe I'm not really good, but yeah, I started beginning of 2013 I think so it's one year and a half. It's yeah, it took a long time It's just one hour there one hour there in the night or in the subway and Yeah, it's really long to work like this. It's not an eight hours straight hours a day It's easier to work to work when you've got like two or three hours straight because then you've got to remember what you did before Etc. It took a long time More questions. Okay. Thank you


Mets and then managed by Terry Francona. Let's do Pedro Martinez, yeah. Okay, let's do Terry Francona, Rockies, Nolan Jones. You wanna do a Med Rosario? I was gonna say Manny. Easiest parts are out of the way. MLB All-Star for the Mets. Let's go Mike Piazza. Rockies, All-Star, we'll go Charlie Blackman. Dodger's All-Star, I mean that's... Who do you wanna go with? You can pick 900 million different names. Mookie Pets. This is where it gets a little interesting. Mactress or played for the Dodgers, right? Yes. So we'll do that. I'm done, I don't have anything else. How many teams has Tommy Fan played for? Played for the Messi, played for the Red Sox, played for the Diamondbacks. He played for the Reds. Do Tommy Fan. Yes, he for sure has. Don't have anything for the Rockies. Rockin' my brain right now. Let's try Matt Holiday. Oh, man.