it's my pleasure to introduce the next speaker of the day, Peter Claes, from the KU Löwen. After career steps in Löwen, Melbourne and Oxford, Peter is now associate professor KU Löwen and he holds a dual appointment at the Department of Electrical Engineering, Processing of Speech and Images and the Department of Human Genetics and that's a particular flavor and a particular expertise in his research that he knows about this boundary between imaging and genetics. He's an honorary fellow at the Murdoch Children's Research Institute in Melbourne and he organized two intentional workshops on facial genetics, so exactly this boundary I described. He's also a member of the Löwen Institute for Human Genomics and Society. So we are very happy to have him here to learn more about medical imaging and analysis in the era of big data, which is a fundamental column and part of our networks topic on machine learning for personalized medicine. So thank you for joining us today and we're looking forward to your talk. Okay, thank you Carson for the kind introduction and thank you also for inviting me here today. As you mentioned, the topic of my talk will be about medical imaging and analysis in the era of big data. Just a brief outline to begin with, I will talk about why I consider imaging to be big data. I will talk about different imaging modalities, the analysis and if time permits I will also dive into some particular applications that I myself have been focused on in the past 15 years or so. So imaging and big data, it's all related to the idea of deep phenotyping, which is quite popular today. And when you look at deep phenotyping, you actually are looking at three distinct aspects. First of all, deep phenotyping involves intensive, so a lot of measurements of the same thing. Extensive, many different things are being measured of a person and large scale. You're trying to measure these on a very large population of individuals. So deep phenotyping definitely arises from intensive, extensive and large scale data collections. If I'm looking at images, images by themselves are big and they're basically underpin the intensive phenotyping of deep phenotyping. So an image is collecting tons of data in a single shot. And in fact, images have been increased in data size over many, many years. As you are all familiar with the cell phone in the early days, you had a one or two megabyte pixel camera. The cell phones of today's, they have 15 megabyte pixels and many more. So the resolution of these images is increasing. So the amount of data that an image is collecting is more and more. So the images are definitely big. They've always been big and they're just getting bigger. That's basically the feedback here. At the same time, I also want to reduce another kind of concept that is related to imaging, which is basically feature extraction and pattern recognition.  And this is done by feature extraction here simply as an example. If you want to classify apples from peers, what you try to do is you try to extract some features based on shape and color, for example. And those two features are allowing you to basically recognize apples from peers, apples being red and round and peers being green and a bit of an oval shape. So if you combine feature extraction and pattern recognition with big data, well, basically, those are the two key ingredients for the era of deep learning as we are witnessing today. Now, deep learning itself is not new. It is a subfield of a larger field known as artificial intelligence. And artificial intelligence has been popular since the 80s and the 90s. However, the way intelligence is being approached has changed over the years going from the implementation of knowledge, expert systems to shallow learning, as we say, from imaging descriptors to deep learning from raw images. Nevertheless, image data has been part of AI from the beginning, like image analysis have been using AI since the 50s, 60s and 70s to do the first imaging processing and analysis pipelines. And of course, a lot of deep learning today has become successful, particularly by working on image data. The only distinction of AI in the past and deep learning today is the access to large scale data. Large scale data is a key ingredient for deep learning and machine learning today to be applicable on data. If we simply look at these deep learning networks, I think we're all familiar based on this summer school that these consist of artificial neural networks and in itself is just a connection of very simple units that on themselves are a bit naive. But when combined in multitude, they really can solve complex tasks. And often the comparison is being made to the human brain, which also consists of primary elements of neurons that are being connected into a very intelligent system being ourselves. And artificial neural networks are trying to mimic the same kind of architecture that is observed in the human brain. The key difference of deep learning today in machine learning, let's say 20 years ago, is the following. If I look at an image in the past, we spend a lot of time thinking about what kind of features would I be extracting from the image knowing that these features could potentially solve a problem. So you would do something known as manual or engineered feature extraction. In the past, if you were doing a PhD in engineering, basically if you found a new feature, you finished your PhD. That was the way to get your PhD in the past. Today with deep learning, we have an end-to-end situation. So basically this engineered feature extraction step is skipped and the deep learning network is going to learn automatically to what extent features need to learn to optimally solve a particular task. So now if you want to obtain a PhD in engineering, it's a completely different ballgame and it's all about getting enough data, sensitive data, and to control your data for these deep learning networks to be applicable. And they're very successful. We have seen this especially on image data and has progressed into many other different fields. But I think deep learning has really gained a boost by solving very complex imaging tasks. And the first and the most well-known imaging task is the one of ImageNet where millions of images are being collected and categorized in groups. And basically deep learning has cracked the ability to, given the image, classify the object that is seen in the image as you can see here, the identification of flowers, elephants, boats, and so on. And this is where really the feature learning of deep learning has excelled in solving these tasks. So in other words, if you look at large-scale imaging data sets, these are the new gold in data science and imaging analysis today. And I'm just listing here a few. There are initiatives of collecting a lot of data on individuals, including imaging data. And I think the best known here might be the UK Biobank, which has also been collecting MRI imaging on participants. But essentially these kinds of collections are providing the next generation of machine learning to be applied. If I look at imaging modalities, the second part of the talk, there are quite a few. And if you're interested in more details, I'm just going to go briefly over them because I won't have the time to dive into much detail. But for those who are interested, I do recommend the following book on the fundamentals of medical imaging by Professor Paul Sutons, who also was my supervisor for my PhD. But just to go over a few of these modalities, I think something that is familiar to all of us is simply the two-dimensional photography. Two-dimensional photography is still very used in clinical routines, especially in craniofacial surgery, for example, where images are being taken of the patient before and after treatment. If you move from two-dimensional photography today, we can also do three-dimensional photography. And this is done by three-dimensional scanner. And essentially, it is like multiple cameras are looking at an object from different points of view. And together, they can triangulate the image back into 3D. It's much like how our two eyes are working. We have two eyes of two different cameras, slightly deviating apart in distance. So together, we can basically perceive depth in the surroundings. And these cameras work in the same way. But a 3D image is essentially a point cloud of the collection, which is being connected into a wireframe. And then using computer graphics, you can render this into a surface. You can put some light, and you can really observe a continuous surface. And typically, you can just simply add some texture to make it more realistic and more interesting to look at. If we move from 3D, oh, that was one button too fast. My apologies. So today, we've also moved into the era of four-dimensional imaging. And it's the same again. You have these spots that are looking at the object. But now the object is moving, and the pots are essentially recording movies. So this is a dynamic 3D or sometimes called four-dimensional photography. And this is quite interesting. Again, like in surgery, like in facial palsy, so you have some part of the face that lost its nerve function. And the surgeons try to repair it. They can then reassess the movement in the face or other parts, like in orthopedics, if your knee is bend it properly and stuff like that. So this is kind of an interesting new imaging modality that is finding its way into the clinical practice more slowly. If you look at, well, the more medical modalities, the first one that is best known is X-ray imaging. X-rays are essentially electromagnetic waves that emit a photon-based radiation. And that were discovered by Wilhelm Konrad-Röntgen in 1895. Basically, what happens here is like X-ray vision. And like you see also in airports, like X-rays are scattered, and they move through the object, and they're either absorbed or scattered by the internal structures. So that an X-ray collector behind the object records the intensity of the X-ray. So if you have a substance that absorbs all the X-ray, you will have a darker region. If you have something that scatters the X-ray through, you will have a light. So you have intensity changes like grayscale value. And here, for example, you have an X-ray of the chest. So you can look at ribs and the lungs of the person. So you really see through the body, essentially. A nice innovation into X-ray imaging is X-ray computer tomography. And it's also using X-rays, as it says in the title. But the clever thing is the manner in which the X-rays are being imaged. So there is a whole setup now involved with a patient lying on a table and being pulled into a rotating device. And it's basically accumulating X-ray imaging. And together, thanks to computer algorithms, they are able to reconstruct a three-dimensional image, slice by slice. So you see here a movie where we're actually going slice by slice through the patient's body. But instead of having a planar two-dimensional projection of a three-dimensional structure, thanks to the rotating movement around the patient, you are actually able to reconstruct the full body in three dimensions. And here, depending again on the tissue, like heart tissue, it's absorbent and soft to less. So the intensity changes are going to display different types of soft tissue and heart tissue in the of the internal body. And a similar kind of adaptation in this technology, but more broadly applicable to orthodontics, is known as combing CT. Again, it's an X-ray imaging device. But instead of having a rotating slice by slice, you are now beaming a full cone of X-rays at the same time. And the result is essentially this, that you need less acquisition time and hence less radiation to make a full three-dimensional image. And this is one of the reasons why this is so popular in orthodontics, because they're like the the risk of exposure to treatment gain is not so high. So you don't want to have a system that gives a lot of X-ray radiation to the patient to be imaged. But essentially, these are all X-ray based imaging. But you can do more. You can also say, you can also add some contrast agents to the to the patient's bloodstream, for example. And typically, these are then radio opaque material. So if they get, they really absorb the X-rays. And hence they really, the contrast of these structures is being enhanced in the image. So here, there is a contrast agent being admitted to the blood flow to the bloodstream of the person. And as such, you can really highlight the blood veins in the hands on the left or in the neck and head and brain on the right. This is what is called as CT angiography. So it's by administering an additional contrast agent into the bloodstream. Other types of CT scanning are dynamic, like imaging the heart is not an easy, easy thing to do. Where there the scanner is essentially aligned with the rhythm of the heart. And you kind of take pictures at a certain stage of the heart cyclists. As well as if you're using the profusion, you can continuously image the patient and actually follow the flow of the contrast agent through the veins and the blood vessels. And as such, you can investigate if certain parts in the brain in this case are being under, if they have enough blood, yes or no. And in cases of a stroke, there will be regions in the brain where this imaging is then revealing that there is no blood flow to that part in the brain. And this is then to localize the obstruction and then operate or trying to treat it as such. So it's known as dynamic CT. So basically it's like a movie of CT scans. That's another way of looking at it. And it gives you some dynamic information. Another type of imaging modality that's very popular. And while often using complement to CT scanning is magnetic resonance imaging. It's very different. It's also non-invasive. And it's using not X-ray beams, but it's using a magnetic field. And basically, so the the idea is for when when different tissues are being are being exposed to a certain magnetic phase, they're out there at the magnetic phase of the atoms of the tissue will change under the influence of this external magnetic field. And this change can actually be visualized. And depending on the tissue, there's going to be a different reaction to the magnetic field. And again, you have different intensities. And hence, you can discriminate different tissue types from each other. But what is more interesting as well is that in MRI, you can also play with the magnetic field sequence properties. And hence, you can put more or less emphasis on different tissues. Here are a few different MRI sequences. And I'm just going to highlight the first one is a T1 is known as a T1 MRI sequence. And that is basically used to image the brain anatomy, because in this kind of image, we can really differentiate well between the gray matter, the white matter, and the CSF in the brain. A flare image is another example in which the magnetic field sequence is different. But that's an image that is typically used to emphasize tumors. So different types of tissue of tumor tissue will be emphasized using flare imaging and other pathologies. And that's the nice thing about MRI imaging. You can really play with the sequences and hence emphasize different tissues of interest, either the anatomy or some pathological aspects. Similar to CT, here as well, you can also add contrast agents. But instead of having a radio or park contrast agents, this would be something that is paramagnetic. So something that is highly influenced by the magnetic field and hence lights up in the image of the MRI. Here you can see again, this contrast agent is again administered to the blood flow. And also with this contrast agent, you can highlight the blood vessels and the blood stream as such in certain parts. Another really cool thing for MRI imaging is something known as functional MRI imaging. And both imaging or blood oxygen level dependent imaging fMRI is one example of this. And essentially what it does is it's following the following assumption. So when when certain parts in the brain get activated, they're going to require some oxygen consumption. So there's going to be an increased blood flow to these parts in the brain and it's going to as such going to be an oxygen concentration. Now, what is of interest is that the parts that have a high level of oxygen have a different magnetic property and the parts in the brain have a lower level of oxygen. So hence, you can emphasize the parts in the brain that are being activated during certain tasks. Here in the slides, you basically see the activation of the brain when a person is holding his breath. So basically getting an oxygen deprived situation. On the top, you see like holding your breath after 14 seconds. And at the bottom, you see, well, the blood trying to compensate for the lack of oxygen after 27 seconds. So you can really see that the whole brain is being lit up by this imaging. Now you can use this in the following setups, for example, you can actually ask people to do something or to look at something. And then you can monitor the activation of oxygen and hence the blood flow in the brain as such. On the left, you see an example of the activation of brain regions that are involved in speech and language on the top. But interestingly, it has shown that the same regions in the brain are also activated by gesture language. So not spoken language, but gesture language and spoken language turn out to be processed by the same regions in the brain. And this is visualized thanks to the bold functional MRI imaging. On the right, you can see activations of the brain for the motoric and sensory skills. And so the person is trying to perform a motoric exercise. And by at the same time, imaging the brain, you can actually see which regions in the brain are responsible for your motoric and sensory skills as such. So this is what is known as functional MRI. The next modality down, which is also very interesting, is known as nuclear medicine imaging. And it sometimes sometimes sounds like, like dangerous at the nuclear aspect. And in fact, it's not so dangerous, but it's quite distinct to the other imaging modalities, because here, the imaging source is not being from the outside. But essentially, what happens with nuclear imaging medicine imaging is that there's a tracer molecule that holds an unstable isotope, like a radionucleotide that is being injected again into the body. So it's another form of functional imaging techniques, but it's using radio tracers to visualize and measure changes. And for example, metabolic processes and other physiological activities, including again, blood flow, but also regional chemical composition and absorption. As I mentioned, the tracer module is administered with an unstable isotope. And there are two types of isotopes, positron emission tomography like a PET scan, or a single photon emission computed tomography aspect scan. And basically in the latter, for example, the isotope is emitting gamma rays, and it's the gamma rays that is being that are being emitted. And essentially, if the tracer is moving down the body, and it's being absorbed by certain metabolic processes, then there the tracer will light up. And basically, that's why you see all these, well, apparently little lamps lighting up in the body and exposing, well, the proper functioning of a metabolic process or the malfunctioning of a metabolic process, of course, as well. And this kind of imaging is typically combined with an anatomical imaging. So for example, MRI, you can combine with a PET so that you can also localize the metabolic processes. So this is then known as a PET MRI, but you also have a PET CT or a SPECT CT. In fact, PET CT is a very useful in imaging the location activity of cancer and metastasis of cancer. But it's typically a combined imaging, the PET or SPECT together with an MRI or CT to gain some anatomical localization of the activities.  And the main difference between them is where exactly these are situated on the electromagnetic wave spectrum. If I go from the shortest waves with the highest frequency we have, on the left here, the gamma rays used in nuclear medicine imaging. And we have the X-rays used in CT and combim CT. Then there is the aspect of visual light in the 2D, 3D and 4D photography, but also endoscopy, something I haven't discussed here. And then there is thermal imaging, which I also haven't discussed, but at the other end of the spectrum we have radiomagnetic waves as used in MRI. So all of these modalities are basically electromagnetic waves and they're basically different from each other in the wavelength of the imaging modality used. But there is one more imaging modality that is not an electromagnetic wave, and this is known as ultrasound. Ultrasound is based on the time of flight of ultra-high sound waves. So in contrast to the other imaging modality, it's a sound wave and not an electromagnetic wave. Sound waves as well, they undergo reflection and refraction depending on the underlying tissue. And again, for example, if I have a sound wave and it hits on bone, it has a perfect reflection. But if it hits off tissue, it's going to be refracted. And if I basically send down a sound wave and it's being reflected and I measure the time of flight between sending it out and receiving it back, you can then form an image. And I think every one of us or the most of us are most familiar with ultrasound thanks to well, the ultrasound imaginations of fetus imaging during pregnancy and many parents and take the very first image of their child and show it to the rest of their family. And ultrasound as well, much like other modalities, have really known an expansion in technology. And we can go from the traditional two-dimensional ultrasounds here to now today, we can also do a 3D ultrasound. We can also have a 4D ultrasound, so a movie of the baby, and then the high definition ultrasound. So here as well, there is a lot of technical advancements that took place to increase the use and value of these images. Independent of the imaging modality, the complexity of an image data is mainly defined by three factors. And the first factor is the resolution as illustrated on the top. A resolution, this is an aspect also where each manufacturer of an imaging device tries to improve their systems with. As I mentioned in the beginning, the cell phones in the early days, we had a one or two megabyte pixel camera. Today we have cameras with a much, much higher resolution. So with lower resolutions, the problem there, we basically not always see what we want to see. For a, the resolution is too low, like the five by five, we don't see that the letter R is actually displayed. And the advantage of increasing the resolution is that you have a sharper vision of what is, well, for example, the R in this illustration, but essentially also in the medical imaging at the better delineation of the heart or the liver or other organs are becoming possible if the resolution increases. Another aspect that is clearly involved in imaging is the signal-to-noise noise ratio. So any ray or wave detector introduces noise into the signal. And if we look at the imaging modalities we've covered so far, examples of high signal-to-noise imaging, so where there is a lot of signal and not a lot of noise, we talk about photography and CT. They have a very good signal-to-noise ratio. In contrast, when you look at MRI and ultrasound, it's the opposite. The signal-to-noise ratio is low. So there's a lot of noise in these images. And the last aspect that is contributing to the complexity of image data is imaging artifacts. And these are specific for each imaging modality, but they always introduce some artifacts. You can see some strict artifacts here. Essentially, this is a CT scan of a mandibular with teat fillings and the amalgam fillings, the metal fillings in the teat, are basically creating artifacts as they interact with the x-ray. They really scatter the x-ray beams, essentially. This is known as metal streak artifacts. So these are the three aspects to keep in mind when you work on image data. So the higher the resolution, the better. The higher the signal-to-noise ratio, also the higher the quality. And the lower the artifacts, the better as well to do some analysis, which brings me to the next part of this talk. What about the analysis itself? So we've covered some modalities. If I look at imaging, processing, and analysis, I make a distinction between, well, simple operations on imaging, segmentation, registration, representation, recognition, and generation. I will go by these one by one to explain to you what it all means and what the purpose of these can be in the medical setting. First of all, simple imaging operations. A key example here is what is known as histogram transformation. It's a very basic operation. And I always say, lucky, you can even open Adobe Photoshop and apply this kind of transformation to any image you would import there. Histogram transformations are quite popular in Photoshop as well. It basically involves a transformation of the gray value histogram. This is to enhance and emphasize different gray value intervals. In this example, this is the original range of gray values. If you really want to focus, let's say, on the parts of the lungs and have the details of the lungs, you can transform the histogram to emphasize that part. And you will see at the expense of the other surrounding tissue. So you cannot really discriminate anymore here, but you really see the details of the lungs. And the opposite, you can also transform the same histogram differently by, hence, emphasizing the surrounding tissues but having no detail left in the lungs whatsoever. So histogram transformations are often used in radiological examinations. If I want to see lung damage, for example, as a radiographer, then I would emphasize the structures in the lungs much more, especially for COVID investigations today and the damage that they do in the lungs, you would perform such a transformation to emphasize any damage done to the lungs. But they're very simple to do. But there are some other simple things that we can do. And they already generate some, well, very interesting applications and aspects. And the other operations that I'm talking about here are known as linear image filters. So a filter is something like this. It's a small window. And it has some weights. And this filter is being convolved over the image, as you can see here. So the blue one is my image. And the filter is being basically slided up from top to bottom and to below. And essentially, I'm applying this filter to do certain operations. If I'm using, for example, the filter here in the middle, which is basically 111. And essentially what this does is it's counting up the value of nine pixels and taking the average. So this is a smoothing filter. If I apply that filter to this image, I get a smoothed version of that same image.  If I apply multiple of these filters together, I can basically run an edge detection onto the image. And then you can also start playing around with the filtered results. You can add one, the smoothed to the edge detector. And then you have something what is known as image sharpening. So the result here is a sharpening version of the original image. And this is the key of image filters. And in the past, again, this was basically what engineers were looking at, trying to find out the filters to perform certain tasks. And I think you could have guessed why am I saying that? Because today, this task has been taken over by deep learning, but deep learning in essence is again, simply filtering deep learning on imaging using convolutional neural networks are nothing more than filtering images in the same way we have been filtering images for 30 years or more. But the main difference is instead of trying to come up with designed filters, the deep learning is going to learn the ways to extract features from the image that are deemed necessary or useful to solve a certain task. And the other difference is instead of running a single level of filters, you're going to continuously do this operation so that we can extract simple features and make more abstract, more complex features by increasing the levels or the depth, in other words, in the deep learning network. But essentially, this is how deep learning extends simple linear filtering on images, which has been done for many years. And because deep learning is so good as it, it doesn't come at a surprise that deep learning is also overtaking some typical operations on images. Here you see, for example, an operation that is often wanted is super resolution. So you try to increase the resolution artificially of an image from a lower resolution image. This is, of course, very interesting if you're dealing with a low resolution camera, you can still artificially increase the resolution and hence the detail that the image is displayed. And this network is basically trained on a database of images of both low and high resolution. And it really learned to transfer, to upgrade these low resolution images into a higher resolution equivalent. And once learned, once the filters have been learned, you can essentially deploy them on new images and increase the resolution. And this is a Google project, by the way, that is being shown here. That brings me to the first, so operations are one, that brings me to the first more semantic image operation, which is known as segmentation. So image segmentation, if you want to have a definition, is essentially the process of partitioning an image into different meaningful segments. You're trying to delineate that the parts that are of interest for you. Here, for example, is an anatomical segmentation of the brain into gray matter, white matter, and CSF, so the filling, the fluid filling between the matters. Or here, a lesion segmentation of a tumor, that's your segmentation outcome. Or WMH stands for white matter hyperintensities, which is often, which is brain damage inflicted by multiple sclerosis. And by segmenting it and hence measuring the effect of these lesions, you can also make some clinical interpretations of the condition of the patient that you can, for example, measure if the tumor has been growing in the last few weeks, or if you're giving a treatment to a person, you can measure if the tumor hopefully has shrunked in size, so that you can see if the treatment is actually having an effect. And so segmentation is one of the core analysis that is being done in medical image analysis for patient monitoring. Segmentation can be done in different ways, and the first way you can do so is as an object delineation task. And basically, it's trying to find the contour or the surface of the context of the object of interest. It's no more than that. There are quite some algorithms that are basically trying to model this, successful ones as well. But essentially, you're trying to delineate this. And in the past, this was done also manually by the radiographer on the screen, essentially. Segmentation can also be done by partitioning, where you take the image as a whole, and you're trying to define neighborhoods by looking at, for example, similar intensity values. This is a very naive segmentation, as you can see here. But that's the idea. I'm just going to try to group pixels together into coherent regions. And the most promising way of segmentation is segmentation by classification. Here you're going to try to classify a pixel. So an image is a lot of pixels, a lot of voxels. And for each of them, you're going to try to burn with machine learning or any other kind of tool, a classifier that's going to predict the class to which the pixel belongs to. And this is essentially where deep learning has taken over. Deep learning is now doing basically a classification based segmentation. So it takes in the image, like you can see here, even for autonomous driving, the same happens. If the car is looking in front of the street, there's a deep learning network that is trying to classify each pixel into a category. What is the street? What is the moving people? What are the trees? And so on. And in the medical imaging analysis, the exact same thing happens. So the image goes in. And for each pixel, there is a label being put out as a probability of it belonging to, for example, a tumor. And I can see here the unit is one of such AI systems against an expert delineation. It's really getting close to achieving the same results. Another kind of operation on images aside from segmentation is what is known as registrations, the process of spatially aligning multiple images into a single coordinate system. And typically in medical imaging, this comes from different imaging modalities. So for example, a person had undergone a CT scan and an MRI, you can then try to align both to superimpose and fuse the information of both. Or for example, a person has undergone surgery and you basically want to see changes based on aligning two images. I give you a geographical example of image alignment, but here you can see a surface-based image alignment where essentially the alignment consisted of a facial template, a wide mask, that is being deformed and shaped into the face of a person. So basically it's like indicating thousands of points on an individual face. So you can do that for large databases and that's interesting for reasons I will show you later on. But these are other examples of why registration is of interest. So if you have two CT scans, the first one is the original, the second one is an angiograph, so CT with a contrast agent added. If you don't do a non-rigid registration of a normal CT scan and the one with a contrast agent added, you can really segment out everything except for the blood vessels. So that's the kind of application of image registration where you can then emphasize the differences between two images, one with and one without a contrast agent and then you really segment out all the blood vessels and you can really focus on that. Or intra-patient MRI CT scans. So this is the same person scan with a CT and MRI and as you can clearly see here a CT scan harvests or shows different kind of information and actually has a lot of information on the heart tissue. The MRI scan on the other hand has a lot of information on the soft tissue. So if you combine and merge them you have a good vision on the heart tissue as well as the soft tissues in the human head for this example. So that's also done through image registration. And in fact you can also gain segmentation by using a registration for example if you have an atlas or if you have a brain template in which you nicely indicated certain brain regions according to functionality or whatever. You can register the atlas to an unseen image and as such transform or transfer the labeled segments as such. So this is basically another way of getting to a segmentation but now using image registration. And today again deep learning has kind of overtaken the problem of image registration. I will spare you the details but there are some really nice auto encoding systems that are able to given two images to map one image onto the other in a single step. Like most registration algorithms so far up until the point of deep learning were kind of iterative in nature. You had to find a solution and then improve the solution in the next iteration and keep on iterating until the solution didn't improve anymore. One of the benefits of deep learning equivalence to registration is a single step solution so the network gives you a mapped version of one image to the other in an instant. And the nice thing about registration is that you can also then work towards image representation or embedding. An image representation is basically or often a transformation into a low dimensional representation that still retains meaningful properties of the original data. As I mentioned in the beginning of the talk an image is really big in data so there is always an interest to trying to downscale the amount of data you need to retain. For example if I look at the full image here of healthy and diseased people I would like to be able to map them into a same lower dimensional representation and hopefully at those mappings will then also tell me how the diseased images are separable from healthy images. And so I can then use this representation as a diagnostic tool. I now have a new image coming in and I'm mapped that image or I embed the image into the same representation space and depending whether it's closely to the diseased or the healthy individuals I can make a diagnostic outcome and say okay this this configuration is definitely to the diseased or the healthy situation. So a lower dimensional representation is always very interesting for image data and in many for many years the following concept has been quite strong in doing so it's known as a statistical shape model. And I'm going to illustrate this based on facial images also because I've been working a lot with them but essentially in such a representation a single point represents a single facial image. So it's a high dimensional space but of course I'm just illustrating it as a two-dimensional graph here because of well illustration constraints but each point is a full face and if you represent your data as such it's really easy to construct for example the average face with basically the gravity point of your point cloud. Or you can also look for modes of variation in this case it's showing me differences of faces due to age on the one side if I would move along this direction I have very young small faces on this side of the direction I have very older more matured facial shapes so this is the mode of variation that you can then extract from your images. You can extract more like a second mode of variation and for faces this is really the difference between elongated small well tall and narrow faces and small and broad faces on the other hand. And then here you can still see some others like this is a third mode of variation and all these modes of variations combined are known as an active shape model so you can you could really model the variation that is present in images and you can use that model to determine things like for example a boundary within which you say okay if I'm within the boundary I'm definitely dealing with a plausible face outside the boundary I'm going away from the facial configurations and I'm creating anything but a face so it gives you a probability model of what facial probability is given a face. And today as well active shape models have now been replaced by deep learning alternatives and the key network to use here is an auto encoder so here typically you have an encoding stage of your shape and then a decoding shape and at the middle you have your lower dimensional space that is then replacing the active shape model and here essentially what I'm trying to what I show here is I trained an auto encoder on 16,000 faces and then I did some kind of TSNE visualization of the faces and as you can see the auto encoder is able to encode faces that look alike closer to each other so it's clear that the auto encoder is learning what facial variability is all about. So that's the next step of representation learning but now using deep learning. If you do some representation learning it's also very easy to start to do some recognition like image recognition is basically the ability to identify objects places or people writing in actions in images and in medical imaging recognition is often related to finding imaging biomarkers for diagnostics and prognosis for example if you're doing some tumor delineation but you have different tumors in a dataset where patients have been followed up longitudinally and you have some outcome of the prognosis whether the survival yes or no you can then train machine learning networks that given the image trying to predict the prognosis so what is the chance of survival of the patient given the current tumor segmentation here or if I look at faces again if I give at a facial gestalt can I can I diagnose a certain syndrome yes or no for example this is the average face of a control group and this is the average face of the canto syndrome the idea is to look at specific features in the face biomarkers in the face of the image and I can then end concluding a diagnostic outcome yes or no or a prognostic outcome. Essentially recognition on image learning yeah again here deep learning is is is excelling out of there as I mentioned in the beginning of the talk image net is all about recognition from images and like objects but also officially here now I'm showing you an app that is actually in place so it's commercially available for clinical genetics geneticists to just take a picture of the of a patient and to have some kind of facial interpretation of the potential underlying syndrome that is presented and it works really well it works really accurately so recognition again from images is something that is commonly done and very interesting for machine learning to be deployed an interesting network that I always like to use when I try to recognize aspects in this case syndrome classification is a triplet loss network and the reason I'm saying this is because in in medical conditions in contrast to image net which is on mainstream image data which can be gathered by by the millions in medical settings you often don't have very large data sets so you really need to look into mechanisms and learning paradigms that allow you to optimize the use of the data and the triplet loss is one of those instead of trying to classify an image given the image only the triplet loss is going to use triplets of your data so triplets of images together and basically what typically does it gives you an anchor this is the this is the standard image let's say a positive example so this is an image of a person let's say in this case of the same syndromic group and a negative example so it's another image of someone from another group and what the triplet loss is trying to do it's trying to learn a new embedding of the data such that the distance between the anchor and the positive example goes down and the distance with the anchor and the negative example goes up so it's really trying to cluster your data based on these labels but the advantage basically is that you can use triplets of data and and each anchor can be combined with multiple negative or positive examples so you're reusing your data in a very efficient way and such a network is really interesting for medical applications and medical data sets as such and if you do so for facial shape for example I learned a triplet loss to separate males from females here and it's also nice to see that the embedding indeed is putting really extreme males on the one hand and really extreme females on the other hand and there where the two cohorts are touching upon each other indeed the distinction between the male or female face is less clear so here is basically you can have some confusion whether it's a male or female and on the right I've also learned something but not more in terms of age so the yellow are the older faces and the blue ones that are shown here are younger faces so again you can learn to recognize age by just looking at an image or learn to recognize the sex by just looking at an image that brings me to the last typical operation which is generation and generation essentially is a task of generating new images from an existing data set and this is typically done for simulation purposes so if I have a known condition I would like to generate something that I want that I think is going to happen so if I if you look at the animation here oh that was one too fast Peter if I look at the animation so I'm I try to plan an orthognatic surgery by moving the chin forward so I'm asking myself what would the facial envelope look like if I do this and that is basically an application of image generation you're trying to generate something that hasn't been given yet so synthetically generate images well again if you have your models it's quite easy to generate images by simply sampling within the boundary so here you can you have your active shape model if you simply sample within the boundary you can generate new faces that has not have not seen been seen before and I think most of you are familiar with this at least I've seen it in the media a few times now again deep learning is really good at generating images and the well the best network to to start engaging with this kind of work is the generative adversarial network training so you have a generator of images and at the same time you have a discriminating network that tries to say whether the image is correct fake yes or no and then the generator is learning to improve himself to fool the discriminator it's like a police versus thief kind of game where the one tries to outsmart the other and that comes down to the adversarial training but what I what I what I thought was interesting to show is for example at the latest results obtained by Nvidia on the left here they've basically trained a network that is generating faces but all of these faces are synthetic so none of them do actually exist and in my opinion the quality of these images is very realistic but also it has also medical applications and medical applications for example is shown here let's say if you train a network that is able to generate a synthetic CT scan from an original MRI scan so doing intermodality changes so from MRI to CT you could then generate CT scans from all the MRI scans you have and it sounds silly why would we do this well you should know that a CT scan does not come without a cost CT scan is known as an invasive imaging technology with a lot of radiation so if I can generate a CT scan or something that is equivalent to a CT scan by using a non-invasive imaging technique such as MRI of course it's a less burden for the patient to do so and so it has a lot of advantages in trying to solve this question the earlier publications on this topic are promising but I think there's still a long way to go that brings me to the last part but I'm just going to check I think in terms of timing might be more interesting we are a little bit flexible I mean at most you could do another five minutes to leave five minutes for questions but up to I would say up to five minutes of talk yeah okay then then I think it's just interesting to show a few applications and again as I mentioned these are basically where I've been working on so there are many many more of course but I just wanted to show you something I think the third one is extremely exciting for us if I may choose here okay I will I will try to jump but the forensic one is definitely like a virtual autopsy where all of the imaging modalities are being combined and used to really determine the cause of death and to solve criminal cases we've also been working extensively on gunshot trajectory look here also if someone's being shot in the head the purpose of this work is to really highlight the damage done and it sounds a bit cruel but if multiple people shoot at you the outcome of this kind of examination is to determine which bullet caused you to die we said the bullet going through the heart or the bullet going to the liver and that's that's key information to convict one person yes or no another kind of forensic imaging is bloodstain pattern analysis here we basically look at the shape of blood stains and we do a segmentation here and then registration into a statistical model that then allows us to deduct the point of origin where the person basically was hit or cranial facial reconstruction is also a combination of registration and statistical models to learn the relationship between the skull and the face and hence if I give you the skull the question is what does the face look like that's another application in forensics clinically people have a lot of questions like the surgeons for example have what are the changes that I induce on faces before and after what's the asymmetry it's clear an asymmetrical case if I improved it by how much and what's the lack of harmony and again here by registration by taking multiple images and superimposing or registering them to each other you can make these measurements I know I'm going fast but I'll try to make it to the imaging genetics asymmetry is the same and harmony is basically trying to assess the patient in the context of the larger population and assess as such create something like a normal equivalent so this is like the patient without the asymmetrical component in the face and that can be used to plan virtually a complete treatment such that the time in theater itself can be reduced from one day to an hour so that's where imaging comes into play this is diagnostics growth curves syndrome classification triple loss I'm just going to skip okay to the imaging genetics imaging genetics basically is trying to also understand this complex puzzle of going to the genome and to your phenome so hey how does it come that it that certain variants lead to certain facial shapes in this case the difference basically with bioinformatics is that we really focus on the phenotyping aspect so I try to improve the phenotyping through the imaging and I'm basically using tools from bioinformatics and others to hence deduct some information and that generates a lot of kind of investigations that we can start doing like developmental aspects like for example what's the effect of alcohol during pregnancy on facial shape how does a face grow from one to two 25 years of old what is facial heritability so here also just by collecting imaging and by register and images images and by bringing them to lower dimensional representations you can really yield the power to find correlations between in this case heritability genetics and facial shape but also evolutionary questions can be answered like I finished did some investigation of climate adaptation to nose shape and essentially their colder climate in Europe is responsible for smaller noses here but also population genetics can be correlated to facial shape and hence we can generate my brothers from around the world I always say so same person but a different population background and investigate other questions like is there a relationship between heterozygosity and facial masculinity I can answer that the relationship was absent and last but not least I guess yeah we have the whole genotype to phenotype correlation analysis it can also be done we're basically here like the GWAS paradigm and here we basically we generated some kind of imaging technique that was able to divide and conquer the wealth of data in imaging by looking at segments in the face and breaking down the image into correlated segments so it's a data-driven segmentation it's not a manual segmentation that really allowed to optimize the power of a GWAS onto facial shape and hence identify many more low side and will previously identified and then I kind of like to play with the idea of you have genetic correlated aspects then yeah you might try to model even an image or image generation from DNA which is then referred to as DNA phenotyping or the opposite an opposite paradigm where I basically don't try to generate a face from DNA but if I give you a face I try to classify to see if the face fits to a given DNA so it's a it's a rephrasing of the problem but instead of image generation you're doing image recognition and again here in the last part of recognition deep learning and learning machine learning in general it's like a classification really excels at solving this problem so when I contrasted the results of DNA phenotyping to facial recognition from DNA the latter proved to be much stronger in establishing a link between a face and DNA and the former basically I think that's that's the main overall message that I still wanted to get contained here today so I'm happy to hear any questions if there are any if not then thanks again for having me but thank you Peter for this great overview that was very educating and very exciting thank you my pleasure are there are there questions from the audience people that are sending applause to you for this for this talk that was great that's nice thanks so I have one question here in the chat by Leslie for generating CT scans from MRIs do you think the biggest driver for improvement will be due to more or better data or better mod or are better models needed yeah there's certainly the two-fold yes there is more data needed there are only very few databases that have both MRI and CT scans of the same individual so it's really hard to get like paired data so that's definitely an avenue to increase on but it's not easy because CT scanning is not done without any ethical concerns so I strongly believe that the unsupervised paradigms where we see more and more of today like there is this thing of image to image conversion where you basically can simply collect large databases of CT scans and large databases of MRI scans and really try to generate one from the other and this is all GAN based essentially and then you're using the the smaller data sets where you do have paired data to optimize the well the conditional learning input because it's a conditioned network where the conditioning is an MRI image and you want to generate a CT scan that fits to that MRI so it cannot be completely unsupervised but in short it's an increase of data and well we need we need to investigate a new new learning paradigms essentially it's a creation of both yeah then there's another question by Giovanni on ESR in the network talk to a student in the network Giovanni please hello thank you for your talk I have a quick question regarding these very last part that you described I wanted to ask you to expand a little bit on how you use genomics essentially for predicting like generating phenotypes and so on like do you focus on specific loci I suppose you don't use complete genomic sequences because that would be a little bit too much for example in the previous slides before I think this one or the one before with the face but yeah I wanted to to ask in which format you use specifically DNA that's a good question so essentially the information that is of value to creating faces from DNA but before I answer I also have to admit and be upfront that the prediction of a face from DNA is far from accurate and possible and I say this because I know there are some commercial companies that pretend that it is possible there but they're never really validated their work but the information that is of interest is are a few aspects first of all is the sex which is basically determined by the X and Y chromosome and secondly there is the aspect of population or ancestry and so far I've been working mainly with simple principle component analysis of SNP data to generate these as you would correct for the confounding in a GWAS today we are investigating variational autoencoders to do the same thing and to see if they don't code for ancestral variation better than that but that's like the genomic view and then aside from that we're still limited still at individual SNP data so it's definitely not full genome data it's more like the low side that we've identified to be associated to facial shape we don't have something like a polygenetic risk or yet although and I can maybe I can send it around later we do have a paper currently already out as a pre-pint but it's on the review on the review still that is well we call it a polygenic shape score and for the first time try to really condense a GWAS into well some kind of facial phenotype prediction but there I can also tell you that that is again limited to kind of cross-population shape differences we found like it was interesting we did a GWAS on an Asian population and an Asian population only and if we combine certain genetic loci into these shapes scores we were actually able to generate a European face from a from an Asian cohort that's kind of the the thing we saw but aside from that it's not going to expose an individual within Europe or or anything like that I see thank you that's really fascinating thank you Giovanni we have time for one quick question by Lukas so Carson and thank you Peter for the talk I was wondering regarding the imaging modalities that you presented there are some of them that have to my uh small limited understanding a lot of heavy preprocessing attached to them such as functional MRI for example are there any solutions proposed to automate uh this heavy preprocessing using deep learning for example yeah definitely yeah but you're right uh like I just gave a brief overview but a lot of these imaging modalities really come with a lot of noise um and hence the the post-processing is important I can tell you that the manufacturers of these devices already spend a lot of energy in in updating the post-processing of the imaging that their their clientele in the end is is a medical doctor a radiologist who has no knowledge of post-processing so they try to basically relieve the burden and and get the best image out but I always find it a double coin like if you do a lot of post-processing you're not really sure if you're actually not eliminated the information that you were looking for in the first place so post-processing also comes at at a cost I would say but here you're right also a lot of these manufacturers are investing in deep learning to solve many of these tasks and if I can say if I look at my experience there is a lot of traditional algorithms that are able to solve a problem but deep learning is so is to do is able to do the same but it's it's better at dealing with noisy and more differentiating situations so it basically expands the applicability of some previously uh already well-working algorithms thank you very much Lukas and Peter and thank you Peter again for this talk and also for taking the time now to meet our doctoral students in the breakout room we are also grateful for that the general audience the PIs and myself we say goodbye for now and thank you we are grateful that you that you joined the summer school and we invite you to to meet the doctoral students in the breakout room and the general program here continues in half an hour with Jennifer List Garden's talk see you all then at 4 30 Central European time thank you Peter


This is where the ayah that says بسم الله الرحمن الرحيم ألا بذكر الله تطمئنون قلوب Indeed in the remembrance of Allah The heart's right tranquility Peace Positive thinking Think about the positive things And if nothing is there Think of your iman الحمد لله I have the iman The love of Ahlul Bayt is in my heart That is the greatest ni'ma That's one thing Thinking positively That's one Second Gratitude What is gratitude? Some researchers They went to a school They divided the students into two groups One group they told them everyday Just write something about your day Anything about your day Today for example I had a lot of homework OK write it down Write anything about your day The second half of the students They told them Write down Something positive Something you are grateful for Something you are thankful for What are you grateful for So they started writing down After two weeks They gave ten one dollar bills To each group One dollar bills This is the United States So one dollar bills Ten of them To each group They said you can spend it in any way you like You can spend it on yourself Or you can give it to charity You can give some of it to charity You can keep all of it to yourself Give it all to you It's up to you Interesting what the researchers found Is the group that wrote Their gratitude journal Everyday For two weeks They gave up two thirds Of their money for charity And they kept only a third for themselves The other group The other group That did not write a gratitude journal But something happens in their day They kept two thirds of the money Or more than half of the money They kept it to themselves They did not donate Not even half of what they had They kept more than half to themselves And only donated a small portion The conclusion they came with The researchers Is that by writing everyday Something you are grateful for You can improve your mental status You can be happier You can help others You can become generous So gratitude they feel Can be taught Gratitude Being grateful And that is something important What does the Quran say about gratitude وإن شكارتوم لأزيدناكم If you are grateful I will increase you This increase is not necessarily Financial increase It's general Allah can increase you financially Allah can increase you psychologically Increase you health wise Increase you etc So therefore Being grateful Gratitude is something that is very important That's the second thing that we need to practice Gratitude Being grateful


So Mike showed us an astonishing diversity of eyes and discovered a lot of exquisitely well-adapted novel and optical mechanisms. And of course, as we've heard from Daan, all of these have evolved in order that animals can have the vision they need. Now, evolution is driven both by the benefits of a particular adaptation, but it's also influenced by costs. So, tiny, lots of animals have very small eyes, because Daan Eric says every animal has the eye it's needs, and they're very small. They have grotty little eyes because they don't need very good vision. And so the costs of a good way of the benefits. And so I think a lot of people would agree that if we're to better understand these adaptations, we should be looking at eyes through a cost benefit lens. And Mike was very good at making new instruments that enabled us to see new stuff we've never seen before. And so I'm offering as my tribute tonight a new instrument, which is a cost-benefit lens with slight resolution. I think Mike would like the idea that this is totally new stuff that hasn't been published. So we start with what makes a good eye. The New Testament says that there are three factors. The first is the point of thread blanching, which is what you would call is the blur. You want that to be as narrow as possible. So you get a nice sharp image. The second is that you want your photoreceptors spaced close together at small angles so that you can resolve the difference between nearby objects. And the third is that you want to catch lots of photons to make the photon noise. The variance in the photon catch goes up as the number of photons. And it's photon noise which makes things look horrible and speckly at very low light levels when you don't have very many photons. So you want to get a good signal to noise ratio by increasing your photon catch. And we know partly through, we go and look in the Old Testament revival, you'll find all the equations you need and to work out that these three quality factors depend on the physical dimensions of the eye. So if you want the photoreceptors to subtend smaller angles, then you want to make either the radius of a compound eye or the focal distance of a lens eye make that longer so that now the angle is reduced. If you want to make the lower circle narrower, you want to make the lens larger in diameter because that's the equation for the ultimate limit to the resolving power of a lens which is departure. And if you want a better signal to noise ratio, you want to make a longer photoreceptor which catches more of the photons that come in because that will increase as the square roots of the number of photons. So everybody knows that there is an expense, a cost to having vessel creation. It is that you need a much bigger eye and that's very good for a lot of purposes. And in fact, people have used eye radius, eye focal length or axial length length from top to bottom or the area of the corner of the eye as measures of cost because they reflect bigger equals better. But these measures have rather poor resolution because many eyes are, for example, are not spherical. This compound eye, although it is part of a sphere, it's in fact a shell only an outer shell of a sphere and this inner part of the eye isn't occupied at all. And many so-called and many simple eyes are actually tubular, for example, or distorted and we're the multiple man. And the second and maybe even more serious problem is that when an animal invests in an eye, it invests in both photoreceptors and in some optical mechanism for focusing the light into the photoreceptors. And in these simple measures, the costs of the optics and photoreceptors for all that together. And so we're not really seeing very much about costs and benefits. And so I'm going to introduce a new measure of cost, which is quite simple to derive. In principle, there are some equations, some geometry called specific volume, which is the volume of devoted to an angle as per angle of visual space, not the total volume of the eye, but the volume per angle of space that you're looking at. And I think that improves and we'll see that it improves the cost-benefit lenses resolving power. So the best way to introduce this is just to show how it's done. So we'll take the acquisition compound eye of an insect, which has an array of lenses, and under at least focus the light onto photoreceptors here, which then absorb them. And you can see that this eye is locally spherical. And the volume of the optics is simply the volume of this outer shell. And the volume of the photoreceptor array is simply the volume of this next shell in here. And there are equations for geometry, which give you those and the volume of the entire eye. Now the interesting thing about these equations is that they contain the factors R and D, which is and delta pi. R is D and delta pi. So they contain the radius of the eye, D and delta pi, because that's R, and the focal distance focal length, and the lens, and the length of the photoreceptors. And those are all the factors which determined our three quality factors.  So these measures and costs are directly related to the ability of the eye to form and capture an image. So they can, they link the volumes directly linked to the R3 quality factors, the blur, the signal to noise ratio, and the angular density of photoreceptors. And these equations given by physics and geometry, and by physiology, and they're very, very simple. This equation here, that the signal to noise maximum is a quarter times the square root of some constant times the length, needs a little bit of explanation, because it may be unfamiliar to you, to many people in the room. But this is a discovery made by photoreceptor physiologists, that the number of microvilli that a photoreceptor makes in an insect eye determines its signal to noise ratio. And the principle is quite, that sounds really scary, but the principle is quite simple. So the photoreceptor absorbs light using a column of photopigment with photosensitive membrane.  And Roger Harding colleagues showed that each microvillus processes the signal from a single photon when it's absorbed. But it takes time to process that signal. So a single minus microvillus can only process as a maximum about 100 photons per second. So a very bright light. So as you increase the light intensity, so this is the rate at which photoreceptor, this is measurements that I made with John Anderson, where we were able to estimate the rate at which a photoreceptor was absorbing photons. And then looking at the noise, work out the rate at which it must be transducing these photons, converting them into little signals. And you can see at low light levels, the rate increases with the rate at which pigment is absorbing in these microvilli. But then you get to higher light levels because it takes time to process a signal, some of the microvilli are not able to process the photons that they've just absorbed because they're already processing much. And so the rate at which you're transducing photons falls below the rate at which you're absorbing photons. And it turns out that this means that the signal to noise ratio stopped climbing. And in fact, it goes through a maximum, which is this value of the quarter of the square root of the number of microvilli. And that comes from both the theory of the binomial statistics of this absorption process, but also has been shown in experiments and by modeling. So this is a hard limit. And it's a real limit. And this makes it very convenient because this is a very simple equation here. So for our first model of this cost-benefit lens, we can use this very simple equation if we consider just live photoreceptors, which are what these are, which are operating under very bright light conditions when this happens. And you'll see that in this experiment, we looked at two photoreceptors, one of which is a long photoreceptor, 250 microns long, and another one is shorter. And you can see that the longer one does indeed transduce more photons in bright light. So it all properly works. So the other question, the obvious next question is, well, how well does do our two measures of cost, the volume per solid angle of the optics and the volume per solid angle of the photoreceptor array, how well do they represent the real costs of the eye? Well, the first cost is space. And of course, this isn't exact. The second is mass. It's heavy. So it's reasonable to assume that the density is about one because everything in biology is made of water to a first approximation. So that's a reasonable approximation. Materials, it's a bit dodgy to assume they're all the same. But the cost is probably relatively small compared to the other ones. So for a first task through the problem, our lens won't have perfect resolution, but it might still show us some. The real problem is with energy. So there are two energy costs. One is the energy cost of carrying the eye as you're moving around in flight. And that's proportional to mass, especially when you're flying. That's okay. But these photoreceptors consume more energy per unit volume of photoreceptor than most tissues in the body because they have to use a lot of energy to convert all those photons into electrical signals. So this is not going to work. We're going to have to find some way of accounting for the extra energy consumption of photoreceptors. And the way in which we do this is we adopt a method which was used by biologists who are interested in weapons. So these are the weapons of stag beetles. So the male has this giant jaws here, which it uses to fight other males and to display to females. So how much does that cost? And what they simply did was say, okay, when the stag beetle flies, it's going to cost energy to lift this thing off the ground and carry it around. And so you take the specific metabolic rate. That's the energy per gram of the flying insect. That's the cost of flying, energy cost of flying a gram measured in the laboratory. And you simply multiply that by the weight of this weapon. And that gives you the energy used to fly that weapon around. And so we apply this to the eye. In the eye, people have measured the specific metabolic rate is oxygen consumption per gram. And these people here did fantastic experiments where they measured this for the eyes of a blowfly. So we can work out from their values what the oxygen consumption per microvillages. And then if we divide that by the specific metabolic rate, like in flying beetles, we get an equivalent mass. And then we convert that because the density is one to a volume. And so we've converted energy to volume through this trick. And we calculate that from very little data on this, but our best estimate is that the energy ranges from between 0.05 and 0.5 cubic microns per microvillage, which is quite significant because the volume of the microvillage, the true volume of the microvillage, is only 0.2 microns per unit. So the total specific cost of our eye is going to be the specific cost of the optics volume, the specific cost, the photorescent of volume, and this energy surcharge added on here, where this is a factor the energy per microvillage is an equivalent volume. Okay, so now we have all of the equations that make our lens work. Let's apply it and see if we can see anything new. And we looked at published data on the compound eyes of dialing flies because this is the best dataset available. And we made some simplifying assumptions and they all perform roughly the same function. So that's kind of what we're going to do. And so what we're going to do is we're going to model eyes and we're going to on the computer and we're going to make eyes of different configurations, but at the same cost. So sorry, they're the same configuration that all opposition eyes, but they have different dimensions for their components, that they all have all of these eyes that we construct at the same cost. So believe it or not, these two eye regions have the same cost. That's because when you go out and everything gets longer than you would in the center of the sphere. And this one has invested heavily in lenses and less heavily in photoreceptors and vice versa. And the result is going to be that here you're going to have very sharp image, but it's going to be very noisy. But here the image is more blurred, but it's much more reliable. Now it's quite difficult to look at these two things and say, well, which one of those do I really prefer, which is best? So we want a measure of performance and you could use lots of measures of performance because we're talking about these universal factors that influence everything in the quality of the image of the eye. You could use some of Dan's new measures or you could use small target detection or something. But we chose to use information capacity in bits of information per solid angle per second because this is a general measure of all of the information that's available for you to see. And in fact, it's the one that Dan used when he was free of paper and I would read it if we had time in my preference. So okay, so we make all of these eyes all have the same cost and then we work out how much information for each eye is going to be able to capture what its capacity is, and then we plot them out and compare them. And what you get is this rather grand thing which in other areas of physiology is called a performance surface, that's just surface across morpho space where morpho space is just all of the possible configurations where I didn't make the same cost. And you can see that there is an optimum, a best configuration which maximizes performance, but the optimum is rather broad. These red areas performance is 95% of maximum. Now this is bad news for people who like to write papers saying that everything in the world is optimized, but good news but insects because it means you can specialize your eye for better signal to noise ratio by sacrificing optics or by crisper images by sacrificing signal to noise ratio without losing more than 5% of your general performance vision. And one of the things that people have always observed is that in these compound eyes in particular everything is exquisitely fine tuned. There are loads and loads of yet minor adjustments and this makes sense because in fact if you make a minor adjustment for one function you're not going to suffer greatly for another function until this makes minor adjustments much more profitable. Well does it tell us anything useful about the structure of eyes? Well we can apply our method to work out how much volume is given to the photoreceptors in a compact fly eye as a function, as a percentage of the total volume of the eye. And you can see that in these flies between 50 and 80% of the volume is photoreceptors. And the theory says that you want a zero energy cost, you want to give a very large volume of the eyes to photoreceptors because they're cheaper. If you increase the energy cost the volume you devote reduces but it's still high and the data falls somewhere between these two theoretical predictions. So we conclude that in fact lots of volume is being given to photoreceptors because this is an efficient way so this bit makes an efficient eye. And of course the photoreceptors, the volume of photoreceptors is long because the photorus is high because the volume is high because the photoreceptors are very long. They get to be something like flies 300 microns long. A single cone in your retina, the equivalent structure is only something like 20 microns long. So these are really long photoreceptors and in fact, as Don Eric has shown, these acquisition insect eyes have the world's longest photoreceptors even though they operate in purely bright light. And this helps to explain why. This is an efficient way of using more resources. And so what you see is that as you invest more in the eye you have larger lenses, narrower angles between omitidia and that gives you better spatial acuity and the length of the photoreceptors increases. So in fact the investment in photoreceptors is matched to the investment in optics. So acuity and length increase in step. And this is seen almost everywhere you look in acquisition of our eyes. So if these acquisition, if this is a particular character of acquisition eyes that they have really long photoreceptors compared to simple eyes, then if we build an equivalent model of a simple eye, which we do just in this very here, you should see that the, you should find that the simple eye photoreceptors, which are the solid nine, the theoretical predictions, they should be much shorter. And indeed they are zero energy costs. The difference is from there to there. And for high energy costs, the difference is from there to there. But you see another thing. You see in the theory that when you impose an energy cost, a small energy cost, it has a tiny influence. I've got these two the wrong way around. That should be point for, oh no, I've got the right way out. It has very little influence on the optimum length. But when you apply the same energy cost to the simple eye photoreceptors and imply an even smaller energy cost, you find that there's a very big chain. So the simple eyes are much more sensitive to photoreceptor energy costs. And that's simply, and that's basically because as you see from this comparison here, in the simple eye, the photoreceptors are more closely packed. So in the compound eye, the volume costs of the photoreceptors are much higher relative to the energy energy costs. And so this has makes a profound influence on the design of the eye. And you can see this when you take the photoreceptor energy costs in our models as a function of eye size, where this is photoreceptor energy as a percentage of the total cost of the model eye. And in a compound eye, it starts off quite high. And then as you make the eye bigger, it becomes smaller and smaller. And that's because these photoreceptors are getting spaced further and further apart. In the simple eye, but high energy costs, even though the photoreceptors are very, very short, the energy cost is now the dominant cost in the eye. It's over 60%. And that's because the photoreceptors are really tight in the simple eyes. So what have we seen? So I hope we've seen that this very pompous microcosm is deriving performance services over more space is useful. We actually do see stuff with this lens that we haven't seen before. And we see that photoreceptor costs shape the entire eye because they are competing for the resources that are invested in the eye. And this, so what this means is that if you spend more on photoreceptors, you have to spend less on optics. So the optics are all smaller. So you can't really look at the two in isolation. And we also see that the impact of the energy costs depends on eye time. So you can't come along and say, well, people have found this thing about energy costs in an in-second line. And it's going to have the same effect on the lizard's eye because they're in geometries. And finally, we see that when we look at our eyes, we find that the investments in optics and photoreceptors are very carefully matched to maximize efficiency. And this is a new principle of the design of eyes. Now, looking ahead, I think our cost-benefit lens can evolve to improve its resolution. And we know how this is done. You have to adapt it to get rid of all sorts of defects and artifacts and actually. And so to begin with, it can be adapted to view eyes of any shape or size. We have formulated this approach and shown that it's useful using these compound eyes where the geometry is very simple. But the principle is that the shape and size of a component in the eye and its constituents determine the eye's performance. So in principle, you can apply this approach to any eye component of arbitrary shape or size. And we now have fantastic methods for imaging the structure of eyes within the eyes and wonderful modeling techniques which can tell you how weird blocks behave optically. The second thing is we applied it to very bright light levels to simplify our proof of principle. But in fact, my co-author on the study, Bran and Andeth, and I, Heras, and I, he's the Andeth Heras. So anyway, Bran, Heras, and I developed a very simple simulation in which you can work out how many photons are given a ray of microvilli is absorbing at any light level. So we're not just stuck to looking at bright light, you can look at dim light too. And I can give you a practical example of how you might want to apply this. So this is Darn, Heras, and Helmut's famous paper from 1996 where they tested the Darwin's theory of the evolution of the lens eye that you start off with a plate a little.  So you end up with a little pinhole and that works like a pinhole camera. And then you introduce a lens here and that starts to focus the light and that makes it much, much better. And Mike Land famously, when talking about this, famously said that this is a critical stage and any lump of stuff put in there would improve the performance of the eye, even the lump of snot. So we could apply this method to Mike's lump of snot. And I think it would work very well. So I have lots of thank yous to say. So first thank you to Bran, who in fact did all of the modeling and formulated the project with me. And also to lots of people I've worked for, some of whom are in this room on the visual ecology of photoreceptors and photoreceptor costs as an important factor in our design. And I also have to think thank all of you in the room who are visual neuroscientists who share the same interests because we are, it's such great fun to work with you all. And the reason is because we're proud in our work but we're not proud in people, proud as people. We like to share things and we like to work with each other. And we just want to discover stuff because it's important and it's fun. It should all be fun. And I think we have to give a big thank you to Mike for that. Years and years ago we were discussing, I was discussing with a group of colleagues rivalry in scientific fields and how dreadful it could be.  And he said it was because the first two people who discovered the technique that got this field going were bitter rivals and really jealous of each other. And that tradition has just carried on. And I think Mike has played a seminal role in creating a tradition in our field which has made it more productive and emphasises something that science should be fun. So thank you, Mike. So there are any questions? Any questions? Yeah, hi, good to see you. This is a political construction, of course, I suppose, or building complex structures and so on. And the other is the primary structure. Yeah, the second part of it implies wide enough management in terms of provision of energy and collection of waste and so on. So it's, my guess is that we were talking about the command at the second. We were talking about the retinas energy bill. So the current feeling in the field is that you make this eye and you build this brilliant optics and then you build this brilliant retina which is adapted and you put them together and you don't actually care about how much you pay for each of them. And the energy bill is just something annoying that you have to pay on top of everything else and it's going to be big but that's, hey, that's vision. The answer to your second, so you make a very good point that photoreceptors are not only more active from second to second as they are transducing, they cost much more to maintain because stuff is continually turning over, particularly the visual pigment and stuff wears out. And of course they're more difficult, they're more expensive to build because they're much more complicated cells. Now all of those costs are accommodated, can be accommodated within that energy factor. It's just a question, this is one of the improvements to the lens that we would like to see. We need a much better understanding of actually the costs that are involved.  So I hope people will start looking in fact more critically and in more depth of a lot of these costs and some of our preconceptions could well be overturned. Okay. I wanted to give you an example of something in the world which is such a bad, you know, and you're greatly missing big leadership in this kind of thing. Oh, I haven't really, yeah, that's a really good question. It would have been that you're still evolving, what does that mean, your maths? Well, there's plenty of things, the fits are not perfect, it's a trend, because there's plenty of things we haven't taken account of. The maths is right, it's the numbers that you put into the maths, which could be wrong. And there might be a better way of actually dealing with the costs in the first place, but until you take this method and break it, then it's a start. As I say, it's a lens, it's not terribly good, but it's better than the previous lenses and it, given time and effort, it will evolve. Oh, so. Okay, I have one question. I think about the signal to noise ratio in photography, right, with the extra net exposure on that, and is it possible that some of the animals live in a slow world? This is another thing that actually we worked on, and that is that fast vision is more expensive than slow vision, because the photoreceptors have to use more energy to change their signals very quickly to keep up with what's going on. And we actually know a lot about the biophysics of that. And so you can equate the time result of the photoreceptors, which is a factor in this information capacity with energy consumption. We didn't build that into the model because it just makes the microscope that much more of a lens, that much more complicated. And we really wanted to show that this was a reasonable way to start moving, but yes, that would be an obvious next step. And I think the method would deal with that, the equations can deal with it, and we know the mistakes. The problem is when you're building in more space, every time you add a new dimension, everything goes up as the power of whatever it is you're looking at. And so you end up with some very, very elaborate models. And probably what we need is Mike to come along and boil it all down to some very simple equations that you can write on the back of an envelope. And once upon a time, I did this, but it didn't work. Yeah, I want to say one more thing about Mike making science fun. He always said, you don't start your research without hypothesis. People haven't seen what's out there, but he went out on the ship, he went out to have a good time, see what was in the water, and then you started getting new ideas. That's what's fun science is when people are something new and not discovering or even knowing something, really it's true. So, thanks. So I've got a question. I think it's naive, not bondage. I couldn't really think much of it. It's pretty good that as well as you. So that jumping spider with different sized eyes, it's using different cost-benefit systems. I'm just wondering what those why did what kind of function with these different sized eyes? So does that relate to the jumping spider's eyes and different sizes? We actually have a pretty good idea about either the sizes and why they have eye movements. So the frontal eyes, or as they're medial eyes, whatever you are, get confused. They have great big lens, fantastic resolving power. And they have very long focal lengths. So the interceptor angles are very small. So they have very fantastic social discrimination. Their discrimination is about one-fifth as good as ours, which is amazing. But is that about right? I went to a lecture on this. But it's also a very small part of the vision here. And it's for the thing is that eye is as long as the spider's body, almost. So it has this tiny little patch of focus, which are attached to the lens by this long sock. And as Mike showed, muscles move this little end of the sock around. So it looks in different directions. And just as the icing on the cake, before the light reaches the photoreceptors, it's refracted by a depression in the photoreceptor layer, which is equivalent to the foveal pit that you find in birds and in birds of prey, and also in comedians. And McIntyre and Williams show that this acts, this makes, converts the eye. It acts as a diverging thing. So it converts the eye into a Galilean telescope. A pair of opera glasses increases the magnification. The thing that all these three animals share in common is that their eyes touch some parts of the body. So you can't make them any longer in length. So they develop this telephoto pit. And then the other eyes get progressively smaller and smaller as they go more progressively. And that's the equivalent of your visual impurity falling off as you move towards the periphery, because you need those other eyes to centralize what they should be looking at. So they do optic flow and all of these sorts of things. And in fact, the casual observation suggests that in the longer focal length, well, in fact, there's a very nice paper just come out recently on the development of these eyes and spiders by Elka Bushbar.  And as the focal lengths get longer, the photoresence gets longer, which is what our model predicts. And also within a single instar, the eyes with the bigger lenses and longer focal lengths at longer distances. So that kind of fits on what we've got. I didn't pay him to ask that. Thanks, everybody for listening.


Hello, I'm Vikash Kumar Singh from Guru Nanak Institute of Technology, Kolkata. I have been placed at ITC Infotech. The course that I had taken from Naptal was Database Management System, the reason being any company that visits your campus will ask questions related to database and the structured query language. It is the most important subject. We generate so much of data and there exists so much of data, so we need an efficient way to like access it, manipulate it or control the data. So database is the most important subject and from NPTEL like they will teach you everything and literally speaking there was no interview that I couldn't answer. I answered A to Z whatever the interviewer asked. If you focus on NPTEL tutorials then you don't need to go anywhere like they teach you everything and literally that's the best material that you can get anywhere on the internet. Thank you. Hope it helps.


What do Corbin reindeer and taxis in New York all have in common? Sex they're horny.  What was that all about? She heard Alexis say it and then she comes running and she goes Ashley Wow, and I was like no child. I am mama Yeah, did you put her in the pen and lock it again? Yes, great 100% perfect That's what you do to correct the kids put him in the pen lock it and leave him. I saw a great Where do you keep your pen?  Oh, cool And obviously have you seen a red beer film? This is yeah, yeah, I don't guys. I don't remember any of the names. Oh, you know what? I don't think you've seen to Marsha What'd you see of his barfie barfie? Yeah, yeah, I can't ever remember the names guys. Yeah. Yeah Also known as Ali Abat's husband. Oh, yes, baby daddy baby daddy. She's prego impregnated her that that happens with his sperm I hope it was his sperm. We don't know. You know what that means. We don't know. You do what you gotta do I don't know means they're gonna have a baby Yes, it does You are good at science. Thank you Wow science Yeah, mr. White science Name that name that show Game of Thrones. Yes. Here we go. He's two films coming in pretty big ones, too Was that a question? Yeah, I guess so My most I'm being useless there is one is then I do a lot of silly magic tricks what I do the most of is Procrastination just sit on my back me and I don't think 90% of the time I have no thought in my head a lot of people say I'm very lucky But then that makes you very lazy because you don't get you're a guy But I can just sit on it. I don't get it This is one of the best parts about being a guy. I Can't I can't do that. Just turn my brain off 250 Then I got Great picture No feet and she looked at it. She started crying. Oh I hate being wet Rejected because they were under water A lot of scenes in the rain. I just don't like getting wet. I don't know what's wrong I even if I wash my hair, I have to immediately dry it. I just don't like being wet. Wow. That is so random See the thing is I don't post and I don't know followers. So what's the point? Like I just have the handle so I can follow mashable But other than that, I have nothing, you know, I so there's no there's no point You know, I could come I could make my account public but as of right now, I'm okay Yes, no social media Craziest I wouldn't say because it usually is used in a negative way, but I remember like when I started like my early years There was a girl that I never met her but my my my watchman told me that she came with a panda Then she married my gate. Yeah, the bungalow I used to live with my parents and There was some tikka on the gate and there was some flowers and all so that was better I was out of town that time. I think so that was quite crazy. I haven't met my first wife. Okay So I look forward to meeting at some point This must have been free alia. Wow None yet. Hopefully soon, yeah, the eight or something. I don't know. Maybe in my Going to be children's names. Oh Most embarrassing moment. I don't know if I get embarrassed too easily. I think Complements embarrass me more than anything else because I don't know how to handle take a compliment So, yeah, I think being highly complimented makes me a little embarrassed Probably You know just your home Dal, chawal, bindi, some simple chicken curry and hachar and pyaz and that's it. You know, that's all you need And some ghee on top of that Always When it was I really don't care about like money has been a Driving force to me. I've grown up in luxury. So that's never been something that I just tried for But I think over the years I've realized my network is just equal to the amount of love I get from the audience You know when I meet my fans like But it's millions But he is a millionaire folks. I think that's really let's google it Aliabad. Aliabad Oh, yeah, he's great Uh, but I think if I say he just said shamshera, so this is okay I think uh, we're really close friends and we annoy each other a lot. We keep fighting and we have a good creative energy between us So shamshera is uh, is a story about shamshera who's the leader of his tribe the comedians. It's his fight against Oppressors who have jailed his tribe and it's it's about freedom and like he says that Azadi Wanted some more. I'm just I'm just confused. I know Because this was I don't know was this Well, how long has he been working on it? I don't know a lot. I think he's been working on this was posted june 24th Uh 2022 and he my future wife. He was married to alia by that time And also impregnated or by that time he I mean the public didn't know but They may have been they may they may have been in a position where he or the production people said Because he mentions the film wait till the trailer is released before you release the interview How long ago did they get married this year? Oh, okay. So it hasn't been that long. No, no, no I think they got married back in I want to say march. Yeah, it was early this year. Yeah Mowage Mowage newlyweds. Yeah, you need to see I'm realizing a couple things and obviously You're realizing how you failed me. No Our buddy times and other things have been you've had some other stuff going on in your life Popping children out or popping children out starting your own, uh, mama baby business. They're kind of invasive. Yeah Children never and uh, what are you talking about? You really need to see tamasha I um, you're gonna love that and You really need to see the trailer for his new movie It's a massive superhero superhero marvel here for it. Um, and then this is a different one It's another epic, but I think it's more of a you can see this It's a teaser. Isn't it that we saw no, we told it was a full trailer comes out the 24th of this month Oh, that's right. We saw both. There was a teaser and then there was the full trailer. Yeah. Yeah, uh, we saw in both What's your favorite rend beer film? Me Yes, tamasha you over barfie Barfie's my favorite prionka film That's true I'm not talking about performance. I'm talking just the film. He's a part of Because that's actually a rend beer film. He's the star of that one I feel no Favorite movie that he has been a part of yeah, it's without prionka that film isn't what it is for me And I think tamasha would be what it is irrespective of the cast Um Barfie's very dependent on that actor doing a good job in that particular role. I mean rend beers role is critically important, too but the It's funny because if I mean If I had to choose between those two films, which of the two is my favorite Um, do you need to pray about it first? Maybe. Okay. I just I love the message lord tamasha I really do Barfie for me is all about prionka's performance tamasha for me is the story and since movies are storytelling I would say I was gonna say it sounds like you've made your decision. It's it's tamasha. Yeah, tamasha is a great one I love tamasha very very true. Yep. And you? Yes Nice Anyway, let us know uh, obviously besides the new ones What should be our next rend beer film that we should watch and ashley? She's coming to do a review with us sometime. Uh, let us know what it should be down below