Welcome back. My name is Steven Seltzer. I'm the radiology department chair at Brigham and Women's Hospital in Boston and what a treat to be here with you today to celebrate this very special occasion and what a great morning we had with such an amazing diversity of provocative and exciting topics of conversation and the afternoon will not let you down either. We'll be hearing from some of the young investigators in our field, winners of presidential early career awards, we'll be hearing from a distinguished radiologist, we'll be hearing from a Nobel laureate, we'll be hearing from a winner of the National Medal of Science and most importantly we'll again hear a patient story. So I wanted to thank Dr. Pettigrew and the NIBIB team for putting all this together and appreciate very much the chance to moderate the afternoon session. Now let me introduce our next speaker who is Carla Pugh. Carla is a winner of a prestigious presidential early career award for scientists and engineers, the P-Case Award. Carla is a associate professor of surgery at the University of Wisconsin, she's vice chair of education and patient safety and she's doing very innovative work in a topic that's very relevant to the NIBIB portfolio which is the use of sensors and simulation technology and we'll hear about her exciting work. Carla, it's nice to have you. Well thank you very much for the opportunity to share my work with you, it's truly an honor to be here and last night we heard some of the history about NIBIB and I have to say I'm pretty sure I speak on behalf of all of the researchers that we are extremely grateful for all of the hard work perseverance and innovation that went into bringing NIBIB to NIH as an institute so we are very thankful. Palpatine is the use of touch in clinical practice for diagnostic and therapeutic purposes. The motivation for my work comes from my own personal experiences as a medical student and a surgery resident in the clinical arena trying to learn what it is that my esteemed professors were experiencing and doing on a daily basis and many times as I watched them perform surgeries and physical examinations I would find myself wishing and hoping that somehow their level of discernment could magically be transferred to my fingertips but obviously that was not possible. There's one story that I experienced that's worth telling and I remember being a second-year resident and spending time on the urology service and there was a patient who was having a procedure relating to the prostate and here's a picture of a normal prostate and here's one of an abnormal prostate where there is blockage of urine and this patient has a problem voiding their bladder. This particular procedure was called a suprapubic prostatectomy and as a second-year resident this would be my first time seeing this procedure and I was very excited and I stayed up all night reading in the textbooks and looking in the anatomy text to see what was going to happen and much to my disappointment this is what I saw for 30 minutes in the operating room the back of the surgeon's hand and as I stood there quietly as a second-year resident intimidated by this very intelligent surgeon and the senior resident fellow who was there I found myself bursting with curiosity as I looked at the senior resident and wondered how is it that you are learning what he's doing and you can't see and so I asked the question and in this day it was a little different I trained on the East Coast and junior level residents aren't supposed to talk in the operating room but I had to ask the question I said how how are you learning I said can I feel what you're feeling and the room went silent and he said sure and he took my hand and put my hand inside the patient's bladder and I felt something that was sort of mushy and I had no idea what it was but this was one of the one story that's pretty similar to some of the other experiences that I had during my training and I actually found it pretty frustrating but I still had the desire to learn and that's what led me to go to Stanford University and get a PhD in education after my surgical training and while I was there I took classes on human computer interactions and learned a lot about sensor technology and thought of all of the things that I could do to change and help medical education one of my class projects would actually ended up being the technology that I used for my dissertation was using force sensing resistors or sensors and placing them on teaching mannequins and I actually chose the pelvic exam not because I'm a OBGYN doctor or a gynecologist but because there was a professor emeritus in the lab who had a model and so I put sensors on the model and this is a picture of a normal uterus most of the mannequins have interchangeable parts and this is a normal cervix and uterus and this is an abnormal uterus with vibrate tumors and so there was no textbook or any information to tell me where to put the sensors but I sort of stood back and looked at where people touched them where they weren't supposed to touch and I put sensors in those areas and you can tell this was a long time ago based on this computer but basically this was the this was one of the models that I built and I used it in the classroom and it was very interesting basically the gist of it is that you can see where someone is touching and with how much pressure and instantaneous immediate visual feedback when you allow them to look at the computer screen and along the ways you know in using this both as a teaching tool and an assessment tool I have these stories that I call the epiphanies and I only tell one there was one student in London I did a visiting research fellowship there who was at the end of her rotation and we did an assessment of their knowledge and we had three or four mannequins that had different anatomy on the inside and after they all went through and did the examination without looking at the computer they did the exam as if they would normally do on a patient we collected their data and they wrote down what their clinical findings were and then we turned the screen around and did the exam and told them what the findings were one of the students was very angry she said I've been doing this exam wrong for two months why didn't I have access to this before I did an exam on a patient and we didn't really have an answer we said well you know this is we're glad that we know we're teaching you how to do it now but it was sort of a shocking scenario for the student and that realization became clear that when you're trying to teach an internal examination to someone who can't see what you're doing they're trying to mimic a behavior that they really don't see and then when you're observing them you can't really see what they're doing and so obviously we then went the next step in terms of building all of the epiphanies and things that we knew that people were doing wrong into the curriculum the other one which was interesting was that there were some students that were naturals meaning they were good they had not had much training they could put their hand inside and they knew that there was something abnormal this doesn't feel like the picture there's some fullness on this side and there were other students who would touch the mass which as you see here there's a mass here in the right at NUXA and I could I know they were touching it because I can see the sensors going off on the screen but they had absolutely no idea that they were touching anything and so we sort of coined the term that there are people who are haptically gifted and unfortunately there are some who are haptically challenged but you can't tell who they are just by you know looking at them this is an example of this is another after I had some success I started putting sensors on every model that I could get my hands on and this is the students at Northwestern using the digital rectal exam sensor simulator and you see they're all looking to the right because they're looking on the screen checking their hand movements the other source of research for me really has been the data that comes out of these technology I can as a surgeon learn how to write a little bit of C++ code just enough to be dangerous much to the chagrin of my engineering students and basically this is a you know less than a second worth of data that comes out and all the sensors are sampled at a rate of 30 Hertz which is faster than any human being can do anything so I didn't know what we were going to do with this data I didn't want to miss anything and this data has mainly been the source of all of my research questions since I was a graduate student this is the data plotted in the line graph formation and for my dissertation the assessment variables if you will in terms of performance that we use with the time that it took to do the exam the number of areas touched during the exam the average maximum pressure used when touching each of those areas and frequency within a sort of narrow mathematical definition when you look at this graph each of the different colors represent a different area of the anatomy and so you can pretty much see what someone was doing from beginning to end and tell a story in terms of what they were doing during the exam now the next slide that I will show you is a slide of a line graph of two medical students are during the course of my research that wanted to use the simulator to teach they were fourth year students that wanted to teach first year students using the model and by that time they had used it and then went from the model to a patient and found it to be very useful so they wanted to use it with students and then just to the pelvic exam is basically you have one hand inside the vaginal vault here and you apply counter you apply pressure on the cervix here and this pushes the whole organ towards the anterior abdominal wall and then you also palpate the other the fundus of the uterus with your opposite hand and so basically trapping the organ between your two hands and these are the two exams of the students I can tell you the the ratings here on this scale are a little different because I had to squeeze this exam to fit it on the screen compared to this one it took twice as long but it's plotted as amount of pressure over time and you can see that right away the student went in did touch several areas on the cervix and then began to do several course palpations on the uterus while holding constant pressure on the left posterior part of the cervix whereas this student touched several areas of the cervix here held constant pressure on the left was not able to find the uterus to palpate it switch positions went on the right then did left and right back around anterior and so on and so forth and still was barely able to touch one portion of the uterus at the very end and we could not tell the difference of what was going on before in clinical practice when you're standing watching so on but now I'm looking at this data so clear rather some students who could fill the uterus and some who didn't it's just part of their technique and their palpation was not there they don't all come one-size-fits-all everybody does something a little different and so the question remains you know should the second student fail or how do we grade this or you know what do we do with this information I then sort of even went out of the realm of Pat palpation any procedure that you can do that doesn't require cutting otherwise you would cut the sensors I can still put sensors on a mannequin and watch other procedures and this is one for intubation and when you look at this the top graph is an experience intubation and the lower graph is one of someone who's inexperienced and if you look closely again I had to squeeze this graph to fit it on the screen it took almost three times or two times as long as the one on the top what you see here most importantly is the sensor this brown one is the center that's on the tongue and it's not sort of intuitive but that's where the laryngoscope rests and so this person experienced anesthesiologist put laryngoscope in the mouth and held pressure and then it passed the breathing tube past the vocal cords and this first blip is the tip but the second one is the balloon and it took me a while to figure it out because I'm looking at this graph and then I go back and watch the procedure and as I start to collect this data all of the procedures that we do have a signature and so my students in my lab could show me any random graph and I can pretty much know whether it was a prostate exam or an intubation or something because they all have a signature but when you look at this performance here this person did several move manipulations of the laryngoscope held pressure wasn't able to see the cords and the other thing you start to see your other sensors here and this one was one of the first ones where I was able to make it clear to everyone that it it doesn't mean that you're doing a better job if you touch more sensors in this case when you start to see since additional sensors going off that's because the person is really struggling and they have severe tilting of the neck and pressure on the teeth and areas that they shouldn't be touching and then they finally were able to pass the ET tube when you could see they really didn't have good control of the over the airway so I'll move on to another exam the digital rectal exam and this one I have more more data but this was a quote that's on the web page for the American Cancer Society and you can read it and I like the term subtle prostate abnormalities and I think this is great because the performance expectations are pretty high but we have absolutely no way of ensuring that any clinician can perform at this level and so this was my interest in doing this work I buy several mannequins off the shelf I also modify them to try and get a realistic range of clinical presentations and these are diagrams show if this is sort of a view through the rectum in an anatomical diagram of the prostate here we built two models one that had a fairly large prominent nodule on the right lateral lobe of the prostate and then another one that had enlarged lobes bilaterally and also a rectal mass and so this is actually not uncommon when you have an elderly gentleman who may have a lesion and these were both pretty subtle and so this was meant to be a easy exam and a difficult exam and so for the first pilot study I had students from a probe 30 medical students 25 residents and 30 faculty examine both of these models as unknowns and when you look at the difference here there's some interesting things going on the medical students don't spend as much time as the faculty or the residents the number of sensors they touch are about the same no different both the students and residents don't apply as much pressure as the faculty frequency is lower and you know the difference is here both on the rectal findings and prostate findings where the students were way off on the more difficult simulator and this was a sort of encouraging one of the things that we found after looking at what's going on with the students usually you would think the students would spend a longer time because they don't know what they're doing ironically what we found is that they actually spent less time in this case because they didn't know what they were doing and this is the scenario I have no other way to show it except that when they learn how to do the the procedure they watched a video and this was the video the same thing the back of the hand and this is all they saw now what they were doing after we looked at the data because we couldn't figure out why they were only touching three or four sensors in the center they were just pronating and supinating their finger in the anus whereas the instructors were doing this maneuver and the students never knew and we never knew that they didn't know and so once we looked at the data and mapped it out and we figured this was going on then we took that back to the classroom and made sure that the students now using the simulator can touch all of those areas both laterally and they understand and many of the students say yeah I would have never figured that out the textbooks or static pictures and you look at the video and it's the back of someone's hand so we titled the paper lost in translation and so we also then got really encouraged and then did a bigger study using the same models but we basically over a two-year period collected data from several hundred urologists at a American Urologic Association meeting colorectal surgeons at their annual meeting and then 200 students that were third-year students during their pre-testing before going into the clinical years and so there were no differences in their performance on the easy simulator they all pretty much figured out the normal rectum and the large prostate nodule not a hundred percent but but not bad and then when you look at the more difficult simulator is pretty interesting so for the prostate exam the urologists were the most accurate compared to colorectal and students but then when you look at the rectum it rectum the urologists were actually less accurate than the medical students and when we got into the details of this and look back at the surveys many of the urologists almost 40% of them did not complete first they admitted that they don't do the rectal exam so they they focus on the prostate and they never supinate their fingers some of them openly admit it but the other part they just didn't even even though the form is said to complete the findings for the rectum and the prostate about 50% of them left the rectal findings blank and so that gave us a quantitative number in addition to the anecdotal comments by some of the people that say they don't do an exam and I think the problem with this is that you know you sort of know that this happens here and there as people become more and more specialized but it's not really clear that it's public knowledge and so the scenario where you have a a patient who goes and sees a primary care physician and the primary care physician says well your your history sounds like you've got urologic problems I'm gonna send you to a urologist and I know you you know the rectal exam is uncomfortable I'll spare you the exam and because urologist is gonna perform it well this has happened more than once it's happened to me where I've had a patient and I'm a surgeon who had gone to a urologist and then came to me and really did actually have a rectal cancer so it happens we don't have a way to quantify it the study that I've done which I'm actually just in the process of publishing is a little window I believe into what may be the truth out there that the specialist focus on the organ that they are a specialist for and we sort of need to understand this is a new evolution in medicine and what the implications are for our patients we then got really sort of bold and decided well we've gotten some luck with the with the prostate exam the digital rectal and the pelvic exam let's try the breast and I say this is bold because the breast is a different beast of sorts if I can use that word the anatomy is has much more variation in it than the prostate or the pelvic exam and obviously a lot wider variety of both normal and abnormal clinical findings in addition the technique you're looking here you some people use both hands some use one there is a broad range of degrees of freedom and so it's a little more complex if you're trying to look and understand the breast using sensor data and this was the project that we were funded for our R01 for an IBIB and it was to validate a set of sensorized breast models for high stakes clinical skills assessment and the first thing was to develop and evaluate these models to see that they can be used to quantify breast exam skills and then we take wanted to take it to the next level in terms of validating it as a high stakes assessment and I'll talk about that again I looked on the market at all the models that were out there and many of them unfortunately and I'm sorry if any of the vendors are in the audience but they're quite awful and some of them are too hard and some of them don't have they have homogeneous material with a big lump and that just is not good enough for medical students or residents or physicians that we want to say you have the minimum competency or even mastery in terms of performing the breast exam so we buy parts and we build you know the layers that we need to represent a wide range of patients these are sort of our basic scenarios in terms of a firm mass which could be either a fiber add no more I cancer we can put it any location make it any size I mean we do no masses we also represent the wide range of normal the breast of a normal 80 year old woman feels very different than the rest of a 15 year old and so we try to also represent that range the same with cystic masses our early excitement was that just in terms of placing sensors on the models and you see the sensors here in color that we can actually differentiate between the two most common types of breast techniques that are used the first one you see here is called the strip technique and you can see by the colors the person started in the upper inner quadrant here and then proceeded with the rest of the exam on this one the person used a circular motion starting at the breast and here you see yellow and coming out and there's you know the variations there so we were happy at least that with one model and the sensors that at least we could tell some difference in the technique the other excitement for me was finding the work of these two researchers who have done over 30 years of research on touch and they have found some very interesting things regarding object recognition and that human beings use reproducibly and subconsciously six specific maneuvers to decipher an object and many of you may have been the subject of some of their experiments or at least a relation where someone asked you to reach into a black bag and pull out the nickel and how do you know it's the nickel there may be a dollar a key or something else in there but you know it because you felt it before you know what it feels like but what they've shown is that what you do to figure it out is subconscious and it's reproducible in terms of if you you know you can tell by it by the weight by the pressure the hardness of it and so some of these maneuvers we were able to actually see in our data the ones that were usable in our clinical exams where the lateral motion the pressure and in sort contour following obviously we're using human body parts and patients that are still alive so we were not we can't do unsupported weight with the prostate because we can't unless it's taken out so these three were the most common ones that we used and so we we actually purposefully just perform those maneuvers on our models to see what the waveforms look like and the first one you see is when you're sort of tapping we didn't have a language for it when you're tapping in a single area without releasing full pressure the second one is actually tapping in within a small radius and then rubbing which is related to their lateral motion in a single area in a multiple areas and you start to see that actually some of these waveforms look differently depending on what you're doing and that gave us some encouragement one of the difficult ones is that you can't really tell what the sensors we were using which only are used for direct force you can't tell the difference between lateral and circular motions because it's it only detects direct force so we then collaborated with a group of engineers that develop biologically inspired hair cell sensors and 3d sensors and these sensors then can detect directional forces and now that we compared the hair cell sensor with also 3d sensor and at this point because of the data is easily discernible we are initially calibrating our mathematical algorithms with the 3d sensor because it gives us a little more differentiation with the lateral and circular movement so this one is vertical rubbing and you can see the blue line is just the direct force waveform but the purple and the red one on below show the x and the y axis on the 3d sensor and so instead of having one signal we now get two signals that help us differentiate that from the horizontal movement which you can see the purple and the red sensor on the right side here look differently so now we have a means of being able to tell the difference in directional forces and that can give us a little more information about the details the other part with our grant is that we have some consultants from the national board of medical examiners in ecf and g and these are the people who actually design licensing examinations and these are the metrics persons who are helping us to understand how we can use this data to decide cutoff scores and for and for performance so we've got a really good team and really excited about this work and I'll just end what's making a little comment about how this relates in the bigger picture with respect to quality of care I think some of you are already thinking about that but traditionally and I am old enough to remember that physical exam used to matter and we used to do physical exams and specifically then order a test that was of relevance based on our physical exam findings and so you would either get blood drawn or go to a CT scan or get a biopsy or a procedure right now what we're starting to see is a wave of scenarios where you don't touch the patient you just send them to the CT scanner and I think that's problematic I checked with my radiology colleagues they think that's problematic as well and when you look at it it actually is market increase in the number of CT scans that people are ordering per year and sorry this doesn't show but they're starting to be a lot of talk about it as well imaging for whom is the patient of the physician and when you look at some of the research studies what you start to find when people get into the details the physicians who aren't as comfortable with their physical exam skills tend to order more studies and that's a problem risk overuse of CT avoiding you know and so there's a lot in here about that and I think it does affect how a patient navigates through the system how be it if you have a cystic lesion that somebody should be able to stick a syringe in and draw the fluid out and send it to pathology but instead you go to radiology you get one exam and it may have been the wrong one because you really never really touched the patient and then you get a second one and then that patient ends up spending a lot more money and spending a lot more time getting to the point of diagnosis and I think that's a problem and we need to do a better job at striking a balance so I think that when you're looking at this it would be great that we could now find a way to assess and certify people that they're competent in doing the exams and get them some actually you know motivation to to do the exam and feel good about what they're doing because we have measured their competency and then they're ordering the right test for the right reason and I could say I have in my conversation at dinner last night with Dr. Seltzer that you know there are lots of groups that are now looking at physician or decision support system that you can't now just order any test without really putting some information of what you've done to show that this is the right test for this patient at that time and again I think that we can sort of add some things in the process that can help certify that clinicians know how to do the right exam and I think that this will definitely have an effect on the quality of care thank you so correct technique that is that's definitely the topic of my research we don't we've never we don't know what it is and so now I finally have a way of of collecting that data that's exactly what I do so anytime I build a new simulator for example three or four of them I take it to the specialist meeting and I have several hundred of those clinicians do the exam and basically it's it's always it's never going to be a hundred percent perfect because we're not perfect in clinical practice but what it does is it gives me a range of clinical scenarios that I know the experts know and I know how they do it I know what they do to get to that answer and you can see that the people who get it wrong they're not using the same maneuver so there are commonalities in what the experts do the ones who get it right and so that's pretty encouraging and exciting to to move forward with respect to the I think your second question relates to mastery if you will and can you can you can you teach someone you know those persons sure I think so I think that in the goal of my my research in the end would also not only just to be to be able to find and define a set of simulation scenarios that would represent minimum competency that you expect for undifferentiated medical student but then another set of clinical scenarios that are for the higher level trainees and the experts who want to maintain a certain level of mastery or even reach mastery and a lot of it comes within their description and what they will do so it's one thing for me to put a lump in something and say you know what is this it's another thing for me to say describe it define it and tell me what you're going to do with this patient and so at in the end when you're going from a specific level of basic competency to mastery it then goes to decision-making as opposed to just your skill in that you have a high index of suspicion that something's not right and you need to do something else to investigate so that's what we're building into the scenarios as well probably so well more an aphorism in the quality improvement literature if you can't measure it you can improve it and you made a breakthrough now that we can measure our performance on how we do things like thank you


I've been using bamboo toothbrushes for the past few years now and feel great comfort in knowing that when brushing my pearly whites I'm not contributing to the ever-increasing piles of landfill. Plus, pairing my compostable toothbrush with my DIY all-natural toothpaste is like a double whammy. No chemicals, no plastics. But there was a downside. I knew you could compost these brushes, except the bristles. But the question was how? Could they simply be thrown in my backyard compost? Would they break down there? Could it take years? Or were they only for commercial composting facilities? I went delving into the depths of the internet and discovered the ins and outs of how to correctly dispose of these bamboo toothbrushes. Here's how to properly dispose of your bamboo toothbrush to ensure it breaks down and returns to the earth as quickly as possible. A fresh, shiny, brand new toothbrush is a much better tool for cleaning teeth than an old, well-used, half-bristles-amissing kind of toothbrush, so it's important to replace your toothbrush regularly when needed. Bamboo toothbrushes are usually said to last up to six months, however doctors recommend replacing toothbrushes every two months, so it's up to your discretion how regularly you want to change your toothbrush. I'm a little more on the relax side of things with my toothbrush, and tend to change up brushes every six to eight months, or when they're looking a little haphazard. Whether you use a plastic toothbrush or a bamboo one, the result tends to be the same. The bristles wear down and bacteria can build up on the brush, even though the bamboo handle is naturally antibacterial. So, when your toothbrush is ready to be recycled, you want to start by removing the bristles from the brush. While most bamboo toothbrush bristles are recyclable, they're the only part of the bamboo toothbrush that isn't biodegradable. Most bamboo toothbrush companies use the industry-standard Nylon 6 bristles except Brush with Bamboo, who are the first company to create a 100% plant-based bamboo toothbrush. This means that we bamboo toothbrush users must resort to removing these bristles by hand and dispose of them separately from the toothbrush handle. There are a couple of different ways you can do this. The first is to pull them out with pliers, which is my preferred method, or the other option is to snap the whole toothbrush head off. To make the process of pulling them out with pliers as easy as possible, don't try to pull out too many bristles at once, and use a slight rolling action to help wiggle them free from the handle. These will help to pull out the bristles and the little metal staples. Once separated, the bristles can go into your plastic recycling bin, but not in the mixed recyclable bins. But as they are so small, they can be easily lost, so it's recommended to first put them inside another plastic item that you're going to recycle, like a plastic bottle. Step 2 is to compost the bamboo handle. The handle of a bamboo toothbrush is 100% biodegradable and compostable. It's recommended that you either put it in your Bioway spin to be disposed of in a commercial composting facility where it will break down in a few weeks, or put it in your home compost bin if you have one. If you do end up just throwing it in your normal rubbish bin and the toothbrush goes to landfill, it will still biodegrade, it will just take a few years. Bamboo is a material that will break down entirely over time, but it's quite robust and not resistant. It's hard to say how long bamboo will ultimately take to break down, as it really depends on the conditions, such as is it in soil or on the soil, in warm wet conditions, or a dry cold climate. If it is simply thrown into your garden, it may take as long as 5 to 10 years to fully break down. If it's buried horizontally in the soil, it may take around three years to completely decompose. If it's placed in a home compost bin, it may take around four to six months, depending on the composition and size of the bin. You can help speed up the process a bit by breaking the toothbrush into smaller pieces with a hammer, saw, or other tool. In a commercial composting facility, it may take only a few weeks for it to break down, as the conditions are much hotter and more active than in a simple home compost bin. And that is how you recycle a bamboo toothbrush properly. I hope you enjoyed this video. Let me know how you go on recycling your bamboo toothbrush. I wish you a wonderful day and I'll see you soon again.


Multilayer packaging or composite materials are widely used in food storage, distribution and consumption. However, their use has raised environmental concerns due to the potential for plastic waste to cause harm to the environment. One method of dealing with this issue is through the recycling of multilayer plastic waste, MLPW. This study examined the effectiveness of ultrasonication in separating layers from MLPW. It was found that ultrasonication is the most important factor in layer separation, followed by nitric acid concentration. Temperature and width were also found to be important factors, but not as much as the other two. These findings provide insight into how to best separate layers from MLPW and could lead to improved methods of recycling. This article was authored by Ayni Sainiut, Jintara's Dinafos and Tamari Mumlats.


notessecurity.com. How many of you have actually been to this presentation before? Last year? I remember you. Okay, we're going to be talking. If you've seen this, saw this last year, and you haven't taken any of the changes we've talked about, stick around. If you went back and you took all those suggestions, recommendations that we talked about last year and implemented those, you're probably alright. What we're going to do, this is very similar to what we talked about last year. There's a couple of new things. There is one big difference with this, is that we're announcing one week from this weekend we're going with full disclosure. So last year we went with limited disclosure in order to make sure none of you guys got hurt. If you didn't pay attention, you didn't go out there. You got one week to get your act in gear. Lotus has done nothing in regards to this, except try to downplay this. So this is why we are doing this now. They've had over a year to get their act in gear and they haven't done it. My colleague Chris Goggins unfortunately is not here today, so it's just basically going to be me. Bill's up here. This isn't Chris. So I'll be doing the whole thing through here. So we'll try to give you a break. When we get halfway through here, I'm going to ask you if you guys want a 5 or 10 minute break. I know it's really hot up here, but hopefully we won't take this long of what we've actually got up here. We kind of ran over in Black Hat. Hopefully we can get through everything this time. When we did this presentation at Black Hat, we were kind of short on time. We had a whole bunch of questions. Feel free to ask any question you want, but let's try to make it kind of quick. Actually, if you have a question, because we've got fans and everything else, what I'd like to do, something different, is come down one of the aisles. Just get up and come on down so that I give you the mic. You ask the question and we'll go from there. Does that sound cool? So, general introduction. Trust but verify. We said this a year ago. You need to go through here and verify everything that you've implemented in Notes and Domino. A lot of the stuff you're going to be seeing today, you've probably already heard about. You've already seen what the likes of Exchange, Outlook, a lot of the things we're going to be talking about, very similar. It is the same type of functionality you can actually do in Lotus Notes. Unless you're running on this guy's network over here. Go ahead. After our initial presentation last year, if you saw that, Lotus had actually agreed. Do we actually have any Lotus people? One? All right. I'm going to be picking on you all day today. I don't know where you work, but I'm not actually trying to pick on you personally. I hope you won't take it that way. But a lot of the stuff we talked about last year was supposed to stay up on their security zone part of the website, which is actually lotus.com slash security zone. And it came down, I think, about two months after our talk, which was really disappointing and I think it was an injustice to the community, both the security community and the Notes community. When we go full disclosure, there's actually, let me tell you something, after we did, we went with limited disclosure last year.  We didn't really publish the exploits. We didn't want to arm a bunch of script kitties. I had about five people email me who were able to understand what we were talking about and they were able to duplicate a lot of the exploits that we had presented. And a lot of them have been very kind of antsy to go public with this, some of the stuff they're working on. There is a guy, his name is Coaxil Karma. He has a website called landofsilence.com. He's actually will probably be publishing a brute force attack. There's another company, or an individual within a company, I don't actually have permission yet to mention who he is, but he's also working on a brute force, password brute force attack. I believe landofsilence, while probably be publishing that within the next few months, their brute force attack. So you need to really make sure that, yeah, it's okay, we're going to go like this. Yeah, that's all right. So we're going to do this presentation one more time if you didn't catch it last year, here we go. All right, so how big is notes? It's really big. Last year when we talked about this, I think it was 50 million. It's now 70. It's above 70. 70 million corporate users. There's a new version, our next. There's a couple of new beta version releases which are out. You can download them. Supposedly the problems with the password authentication we talked about last year, we're supposed to make it into this. I've yet to actually see it. But in regards to what Lotus Notes is, if you're not familiar with it, many people are not familiar with Notes a whole lot. Okay, the Lotus Notes is a totally integrated group or platform. If you're familiar with Exchange, Notes is probably a bit more integrated than Exchange or a lot of the competing products out there. You can do all sorts of the stuff that you see up here. You've got the formal language, you have the Lotus Script. Lotus Script is very similar to Visual Basic. You've got JavaScript, Java, C++. You've got the APIs in there. Yeah, you can't actually compile C programs in there, but you have C and C++ APIs in there. How big is it? Who's using it? Everybody. Big six accounting firms. Half the big six accounting firms are using it. Most of the financial sector is using it.  Multinationals, pharmaceutical companies, it's all up here. Why are they using it? Because Lotus Notes actually has a really good reputation historically for being a secure platform. It's actually based on a public key infrastructure for both terms of authentication and encryption. You can encrypt at the document level, at a field level, at the network protocol level. It's pretty powerful. It actually was a good goal to shoot for, but there's still a few problems with it. In terms of access control, you can actually set access control at the field level, the document level, and the database. And historically, until last year, there were extremely few vulnerabilities. There's been some new ones, and I'm not actually going to talk so much about that. I've mentioned a couple of the guys who've worked on it. There's been some guys who've actually, if you've gone on bug tracking, they've actually been working on analyzing the network protocol that it's using. They made some interesting discoveries there. Release 5, we saw the Lotus drop the support for the Unix platforms, which I feel was unfortunate. But it runs on just about everything, and that's one of the strong features of Lotus. With Microsoft, you've got basically one choice. With Lotus, you have lots of choices, and there's a lot of companies, and there's a lot of organizations that make use of that. Same thing with the servers. So, do you want to see... I may get in a little trouble here with somebody, but do you want to see Bruce Schneider's talk at Black Hat? How many of you agreed with him? How many disagreed? Okay, I saw only a couple of hands. I don't agree, but I didn't get the opportunity to go up and tell him yet. I've got some friends that actually work within his company, and hopefully I'll be able to make my objections known to him. One of the reasons I disagree on this is that to me, security is much more like a vaccination. We don't try to control AIDS after you've got it. It's too late then. I don't try to control other diseases. I don't think you should be trying to control security. I don't think you should be throwing money in the street and relying on detection and response. I believe prevention is crucial. I'm sorry? That's what a lot of us got. Prevention is... He stressed it was more important, if I'm not mistaken. Prevention was lesser of an importance than detection and response. Yes, but he used the excuse or he used the judgment of good detection and response can make up for bad prevention or no prevention. I don't agree with that. The detection and response is a lot easier and faster. I understand that, and it sounds good, but it's kind of like viruses. Detection and response is possible, when you know what you've got to detect against. And you'll see in some of this demonstration, like in terms with lawless notes, how can you detect things that you haven't yet witnessed? When we presented this last year, there was a lot of things. One of the new things we started investigating is that notes actually support direct translation to the XML from the web server. So you can actually start pruning through a web server running on domino and requesting XML documents. And most companies, which use the logging facilities of dominoes, filter out only what they want to see. HTML documents like that, such and such, and other things which look like attacks. XML would get discarded in their logs. So if you're discarding things like XML requests, or you don't even know about XML requests, how many people actually know that you can pull XML straight from the web server? The rest of you are victims for this. So how could you justify good detection and response on something you don't know about? I mean, prevention is really key here. You have to rely on all three. But you really first have... It's like peace accords. You have to have peace accords with other nations to keep a war from starting.  So all three are really important. I don't think that you can just base, go off and leave prevention out of the picture because you have good detection and response. Well, this was my example. I've already said it. But using the farming and vaccination, because this is what the U.K. did with the foot and mouth, detection and response, and how many animals were killed. Does that mean it's right? Okay. I mean, how many animals were cold? I live in the Netherlands. I watched all this. Fortunately, the government of the Netherlands looked at this and said, this is bollocks. And they immediately went to prevention. They immediately started vaccinating everything. And the whole lot of animals were saved. So prevention is, you can't just totally throw it out the window and rely completely on detection and response. So that's the point I'm trying to stress. Because we have the same thing in security. How many servers do you actually want to lose? Because if you don't know what you're looking for with detection and response and you're not using strong prevention, you'll lose a whole bunch of systems, servers, data, before you can actually integrate that into your detection and response. So strategically, and this is one of the things I said in blackout, we really have to stress this, because already we're starting to see some of the repercussions. People don't trust e-commerce. People aren't using e-commerce. Some of it may have to do with the fact that there's a lot of dot-coms out there that should have never been out there to begin with. But overwhelmingly, there was actually a poll just done. And this was actually while blackout was going on, and it was tech TV, and I may get the percentage slightly wrong, but I'll get somewhere from the neighborhood that what they were saying, they'd actually just done a poll of users on the Internet. And currently right now it's something around 80%, 85% of all these users that they surveyed were looking to see government regulation of the Internet for security and privacy. Now, of all these 85%, which were wanting to see government regulation for terms of security, privacy, something like 75% of those surveyed had incomes of 30,000 or less. So I'm not trying to pick on you if you make 30,000 or less. That's not the point. But typically, generally, I'm generalizing, and whenever I'm generalizing, I'm lying. Generally, if you're making 30,000 or less, you're probably not a hardcore IT employee. You're probably not familiar with everything that the rest of us are. And in the security marketplace, we have three different, got a couple of things to consider here. We have the supply side, we have the demand side, and we have threats. On the supply side, I would say of all the security consultants out there, and again, I'm not trying to pick on anyone here, we have 80% which really stay on the surface. A lot of these are some rather big names, unfortunately. But they're general security consultants, and they understand the principles. They're not hardcore technical security consultants. Probably 20%, if that are really hardcore. On the demand side, we have 80%, which historically, up till now, I've been considering security as a non-issue. And if you don't agree with me, then you're probably not a security consultant. I've been out there, I've been bashing people, I've been bashing companies trying to get into the door, talking to them about security. After we did this presentation last year, the company I had founded two years ago, we did this presentation last year, and we probably had five clients, five to ten clients, for Lotus Notes, Security and Domino. We probably, in the first month, we probably caught over 200. So that percentage of the clients, maybe somebody else got them, okay. But I don't think that there's a whole lot of experienced Domino security consultants out there. And right now, with the failure of the dot-coms, we're seeing a whole bunch of motivations which are increasing. We've got international conflicts. I was talking with some of the guys from ISS, Taiwan. They were saying that they see 40% of their attacks are coming from Asia. Another 40% is coming from Europe. About 10% are only coming where they're located at. Only about 10% is coming from the United States. So they're seeing a whole lot of activity over in Asia and Europe, which some of you may be able to relate to. There's a lot of corporate espionage going on. Tons of ex-employees with nothing else to do. They're disgusted. They're bitter. We all know that the stats at 80% of most security breaches are come from disgruntled employees or from the inside. And everyone's got increased bandwidth. I've got a one-megabit pipe into my house where a year ago I didn't. I'm sure a lot of you probably even have more. So how many of you have actually are running 508? Lotus Notes and Domino. Okay. 507. Six. Who's not running 503 or greater? Okay. Are you running a web server? Are y'all? Do you know about the directory traversal attack? Okay. And I assume that everyone here has a security PSPG implemented. We've really got to take this stuff seriously. If we want to see this economy to turn around, you know, this isn't technical, I know, but this is something I really believe in. If we want to see this economy turn around, we've got to start taking some of this stuff seriously. If we want to see the whole everything on the Internet, all the growth, spur more growth on the Internet, e-commerce, we're going to have to take this much more seriously. If you don't identify your problem, somebody else is going to.  I didn't create the problems. I'm helping you identify them. One of the things, and if you're a consultant, I'm throwing this out just, you know, this is why we're using the vaccination as an example. I've found that I've had a little bit better success trying to explain this to people, for the people who look at me as the creator of security problems in the notes world. I didn't create it, like I said, I didn't create any of this, just identifying it, just like the doctor identifies the virus. Go ahead. So we're going to talk about the client side for this first half, before the break. And there's the four things which you probably remember if you saw this before. If you haven't, we have these four things, the stored forms, the execution control list. The execution control list, if you're not familiar with it, it's to prevent potentially hostile code from running within inside your notes environment, or being at least launched from there. We have the password hash problem. We've still not seen any resolution to this, really. There is a small something that you can do, and if you haven't actually done it, which we'll get to it, you need to get done this week. And then the last half, we're going to be talking about the process control list of databases. One of the things that we have been doing in the Black Hat, the Win 2K, and February Black Hat Asia, we actually did some demonstrations. They don't have the network up here, and I have some ethical problems about doing it. We did a couple of demonstrations while we were in Asia, but one of the things is what we called creative surfing. And this is very, very easy under Domino. You can start extracting information from notes databases, running on Domino servers. If they're running a web server. Most people, which have either developed internally, internal applications on Domino, or third-party applications, have not undergone a serious security review of the design. And this is causing serious problems with those databases. Did anyone see the California Democratic Party news story? They were running a Domino server. They actually were accepting campaign contributions. Bob Sullivan on MSNBC had covered this. The California Democratic Party was running a Domino server. They did not conduct a thorough investigation into the design of those applications on their Domino server, but they were accepting credit card contributions, campaign contributions on their website, and people could go in there and just retrieve all the credit card information, which I'm sure was really interesting to the Republican Party. So, we have the stored forms. How many people do not understand this? Because if nobody raises their hand, I won't skip this. So how many people do not understand the notes database structure? Okay. All right. Well, with the notes in the database, you have three different parts here. You have the data, you have the forms, and you have the stored forms. In regards to the data, it's structured data. In release five, we had the addition of HTML before that was just rich text. In release five, you now have the native. It can be stored as native HTML. You can include Java and JavaScript in that HTML. On the forms, since the data is arbitrary, it's just structured data, but it's not very convenient for people who are interfacing within notes to be able to look at a document, pull out the data that they need, or for entering data. So you have forms, and this is similar or akin to a HTML form that you fill out. It's actually how it structures that data to how it presents it to you for you to use. Wait, back up. And you actually have something else called stored forms, where you can actually put the stored form within a notes document for a transmission to another database somewhere else. So that if that database does not actually have that form that you need to view the data, since the form is integrated into the document, when you open that particular document, you'll be able to pull it up and see it the way it was intended to be viewed. One of the reasons I have this, the stored forms is a lot of document management applications. Make use of this. I'd say probably at least 70% of the third-party applications you can pick up on the market from business, Lotus business partners are using stored forms. And this causes a big problem when it comes to the security, because the stored forms can actually contain executable code. So when you pop open that document, the code that's in the stored form will actually execute. And stored forms, this is an important point. Stored forms, by default, are enabled on almost every database, especially your mail files. By default, it is enabled. And if you're running a mail server, and the mail server is actually has internet access for internet mail, we'll do a demonstration here and you can see where you can have a lot of fun with that. So we're going to do this demonstration here real quick. So to explain what I've got here is, unfortunately, since Chris isn't here, and I'm only running on one laptop here, I actually had not configured this with advanced services in time. I had expected Goggins to be here and thought we could actually set up an SMTP environment that was going back and forth so you could really see this. Those stored forms can actually be transmitted over the internet. And this is where I'm talking about. You'll have to bear with me and trust me when I say that this is actually possible. We actually have a server running here and I've got a client which I'm going to use, switch back and forth to simulate several different users. The user I'm actually going to be using here is, you can't really see this, but it's actually called Dutch Locked Offer. And since I'm in Netherlands a lot, we've been using Dutch names for some of this stuff. So to give you a quick example here or let you know what the configuration of this user actually is, this is his execution control list here. And you have a couple of settings here which are important. If you're not familiar with this, you have a default setting and a no signature. And the no signature is for any embedded objects which have no no signature attached to them. Typically these are things coming from the internet. Since things like Outlook until previous versions of Outlook and everything have not been using the same type of, the words have forgotten me, same type of signature that notes is using. Typically this will come across an SMTP server. It has no signature actually associated with no note signature associated with it. Default is anything that's within your own organization and has not been explicitly defined. It's the lowest level within your own organization. So I just wanted you to see that all this stuff is turned off here and you have a couple of different settings here. These are, as of 503, these are pretty much the default settings here. Before these things were coming with everything enabled, and I believe it was in 503 that they finally, Lotus finally did change this, but there's still problems associated with this. Let me switch to the attacker. What we're going to do is we're going to send across a stored form. So I've actually got already a menu option here which creates the stored form with the code in it. So I'm going to use this to create, which I'm really glad I did because using the quality of these projectors is just nasty. I wish we had something better so you could actually see this. I'm actually just filling out the address for a Dutch slug offer and I'm not going to pay anything else in this email. I just say you can identify it. Those are all sevens. So we'll shoot this off and if you look at the bottom of the console screen down here towards the bottom, you'll actually see this go across and get deposited into the victim's email account. So we'll switch back here now. So is anyone familiar with Bubble Boy? Okay. This is basically Bubble Boy recreated. What I'm going to do, there's no actual attachment in this. This email, although the user can't see it, it actually has a stored form within the email and this is similar to what Bubble Boy actually did. So the second I hit enter, what happened? Where'd it go? Now, maybe you would look at it as a security expert, somebody who attends this conference and we'll have gone, hmm, better call the security guy. I'd be willing to bet $100 that your secretary wouldn't and that's all I need. I don't need you to open it. I don't need to email it to the administrator. I just need the secretary to do it because there's already, if you watch the bottom here on the console, a message has been delivered back to the attacker which has the name Inbreaker. So unless this user was actually watching the console, probably just would have thought it was a buggy email. True. Yeah, what he said was that there is a bottom little display bar here and actually I didn't have time but you can get rid of this too. This is possible to get rid of. No, but this is true. This actually shows, this is kind of like a command shell history almost in notes. Everything that actually happened. I'm sorry? Yeah, secretary still wouldn't know. I've actually gone across when I used to be a administrator. You can do a broadcast on the console. Messages will pop up there. I used to tell people, log off. Please log off. Please log off. God back it up. Finally you'd have to walk around the office and hit them on the head. So switch back to the user here or the attacker. So I've got an email here from Dutch sluggedoffer and it says I have created world access. I can't even read that. Basically what it's done, the stored form had executable control, executable code which was executed just like in the bubble boy virus and what it did, it caused that user without him knowing it, him or her knowing it, to change the access control list on the mail file. And I'm sorry I forgot to show you that. Actually I'd gone off and said that. Does anyone need me to actually verify that the access control list for anonymous was completely shut off? I see a nodding head back there. Okay, let me do it again. I apologize for that. Okay, so we're setting default to no access and we are setting anonymous to no access. So we'll go back to the attacker and we'll send it again. Okay, now I'm switching back to the victim. So just to show you, doubly sure default still set to no access and anonymous is set to no access. We open it. It disappears. So default is now, if you see that, if you can make it out with the bad projectors, default is now set to manager and anonymous is set to a manager and I just heard my old heckler friend from last year say show them the log. So you can see there that at no time did anybody else, other than Dutch slot offer, change the access control list. So since that code, come up front because I can't hear you way back there and I don't think anyone else will be able to. But basically that code was executed by the victim.   The question is just cracking. The question is just what rights does the user have to have to their own mailbox or to that particular mailbox if they're opening a floor to give the manager access? Because sometimes you might have a situation where the secretary has got read access to their boss's email and you might prefer to send them an email which excuses code rather than the boss. Yeah, that's a really good question. To change, it's just like if they were to change their own ACL. They have to have manager access to it. But it's very easy to create this. I'm sure it takes no leap of imagination to figure out that you could actually check what rights do you have as the attacker causing the victim to perform this for you. What rights do I actually have to my own mail file? So we have actually done some code on this where you can actually have it do look ups in the database so if they don't have access rights to their own mail file to change it, it would automatically forward itself to the next person. It's possible to simply send it to the admin who will most likely have access, manager access to everybody's mail file and either update the ACL on the name and address book or on the user's mail file. And if you write the script correctly rather than deleting the form which is really obvious and will tip off an administrator, you can simply delete out the stored object so all they see is a normal form that says hey let's have lunch and they can't tell that it's a stored form. One of the other... Somebody else actually had a good point as far as like sending this directly to the email. I believe it was you that actually... I may not be right, okay. Somebody at Black Hat just the other day had asked the question, in notes can you actually forge the header so it looks like it's coming from someone internal and if you're using notes, if you're an experienced notes user, you've probably seen spam people do this so it looks like it's coming from somebody else. You can forge your SMTP headers and looking at the view you won't really know the difference unless you're using a heavily modified mail file. So you could send and by doing that you could actually send the email to the administrator have it appear as a normal user and the administrator will trust his users to some extent and open the email. So I'm going to switch back to the attacker here because this is also interesting. I know that there's not a whole lot of people that actually run mail servers and web servers on the same box but quite often web servers will often have mail-in databases and I know of quite a few companies that actually do run they actually have a commercial web server running Domino and they have a separate mail server which is running the HTTP process so that people can actually access their email using the web-based mail. So if we actually pop this open as the attacker we actually get a URL back which points directly to the victim's mail file and we can just click on that pop it open and go straight into the user's mail file without any authentication or access prompts and there we go. We can now read his email and do anything with it we want over a web browser. Did we not touch the stored forms already? I guess no we didn't. A lot of the stuff with stored forms was reported by Oliver Burger. I think of my slides are a little messed up here. Yeah, one of my slides got out of order. This was reported by Oliver Burger back in 1996 in their Spiegel magazine and I forgot to actually change that but there's still very few people who are actually using Access Control List and everybody is still using stored forms. They're allowing the use of stored forms. What I would rather recommend and I know administrators are not going to like this but if you're accepting internet emails coming into your server and you need the use of stored forms create two different mail files for the user and have one which is sitting there for internet email that uses no stored forms. I know this is a little bit of a pain in the butt but this is really about the only way around that I know of. Turn off the stored forms let internet emails come in there. Have an agent so whenever new emails are deposited that gets sent to their internal email. They're automatically forwarded from that point or at least they have some sort of dock link that goes to their internal email that they know they've got an internet email coming in on the other email box. That is a clutch but it'll work and keep you secure from stored form attacks over the internet. Oh, go back. Go ahead, I'm sorry. We have the Execution Control List which is supposed to prevent a lot of this type of stuff. One of the things that we started looking at was that another type of an attack was that the notes, API calls, the CAPI calls into notes are not intercepted by the Execution Control List. So you can use Visual Basic. You can actually put an active X control into an HTML file and send it to somebody and ask them to launch it and that active X control will make a call into the notes API and bypass the Execution Control List. So I'll give you a quick demo on this. Okay, I'm getting ahead of myself. I apologize here. Before I do that, let me just say a couple more things about this. On the Execution Control List, you really need to make sure this has got down and tight and that you actually have a corporate enforced, essentially enforced ECL. Does anyone not have that? Or does anyone have that? Let's ask that one. Yeah.  I know. A lot of people are just not familiar with the Execution Control List. The ECL settings are really stored in obscure location. There's not a lot of documentation on it. It's not real clear until 502. If you're running 502 or earlier than 502, you have world access to it. There's also a couple of ways you can actually use either of these two options to reset an Execution Control List. There's a command in the Lotus command language that the refresh ECL and just leave the options blank that will actually force the ECL to be reset. There's other ways that you can actually add something into the notes.ini file. And it's remove ECL set up equals three from the notes.ini file. So you don't actually add it. You remove that setting. Yeah. I'll give you a website at the end and all this stuff will be available there because it'll be available within one week. As I mentioned before, the notes API calls are not intercepted by the ECL and since OLE and COM uses notes API, that makes it very fun. Sorry. And our next? I actually have not investigated that yet. So I've only started dabbling with our next. I'm sorry. Yes. And will those be... I'm fixing to give you a demonstration on it here real quick. So the question he's asking, I'm fixing to answer for him. So here is the attacker and what I'm going to do, let me show you what I'm actually going to send him. I'm having a real problem seeing this screen, so I came and see the icons. Bear with me. I can't see the icons off to the right of my desktop. Oops. Okay. I had to move these two over so I could see them. So I've actually got... I'm just going to pop open Notepad here and I'm going to give you an example of an HTML file. And in here we actually have an ActiveX... we have an ActiveX control, which is using VBScript. Just so you can see it, that's all that's in there. So what we're going to do is the attacker, we're going to attach this to an email. So this HTML file, which is actually called prettywomen.htm. Yeah. And we know everybody wants their porn, or at least all the men. Okay.  You know what I'm talking about. It's not getting the semantics here. So we're going to switch back to the victim here. And as the victim, now I have a new email. There's my HTML attachment. This could really be anything that you could... that you would put the... an ActiveX control in. And we just launch it. Now, since we're launching this from the environment, you would think that the execution control... we're launching this from the notes environment, you have an execution control before I do this, which does not permit really anything. So again, as I showed you before, everything is turned off here. And now that there's also Java and Java, but we're not using Java or JavaScript, typically I have a shirt mic trying to do this with... one hand is really difficult. So we go ahead and we click to launch it. So now, something I want to point out here, we do get actually an alert. I'm not trying to hack Internet Explorer. My purpose is to show you about the note security. There's plenty of other good material out here, and I've never tried to incorporate any of that, just simply as to not confuse people with what I'm trying to actually do. This is purely about Lotus Notes and Domino. So we do actually a little prompt here. It tells us we're about to get an ActiveX warning. And when we get that, please click OK. So we'll click OK on this, and here's our ActiveX warning. Warning of a potential hostile code. I believe it was Loft that actually did the latest thing on Office 2000. Is that correct? Nobody knows? I'm sorry? George Kodinsky? Could be. I don't actually get into the Office 2000 Internet Explorer hacks as much. But I know that there's some good material out there on the net that you can find on this. So we'll go ahead and click Yes. Now, what we've seen is we get a listing in HTML. We have this ActiveX control, which has gone off, and this is now not going to the Domino server. Don't be confused by that. We're actually doing an HTML page directly through the notes client. And if you look back here in the view, that matches this. We've got a listing of everything in his inbox. So this is executed. We did get the ActiveX warning from Internet Explorer. As I said, that's not the purpose of this. We did not get any type of execution control warning from Loft's notes. That's the point I'm trying to stress here. Notes should have actually warned us on this and said we're not going to allow this to execute or at least given us the option to abort. I'm sorry. Can you come up here? I can't hear you. I don't think anyone else can. You had that execute with a standalone browser and study internally with the notes client. Would it make a difference with notes look at its internal web browser differently? We could try it. Yeah. 35 minutes for the whole session. Oh, okay. If you want to see that, because he just informed me, we already have 35 minutes left. Let's come up after and we'll try it. I'll just show you here real quick. Nothing got modified in the ECL. Everything is still set to no access. If you have a question, please come up here. If you had detached that HTML file and launched it separately, would it have done the same thing? The point is we actually launched it from within the notes environment. So you would think that the execution control list would actually block any calls into within the notes environment. Notes cannot protect anything coming at it externally. There's not really too many applications that I know of that could actually do something like that. And that's actually even Lotus's statement as far as that goes. So I've gone back to the attacker here and you'll actually see we've got an email back from the victim. And we did this all with using Notes API calls from the Visual Basic Script. And we actually have the name of the mail server. You can't really see this, but on the second line, or the first one after the big space, it says mail server and the name of the server, or the file listing on the mail server where his mail file is, we actually now have his HTTP hash. And I'll get to that in a little bit. And we have his Notes ID file. And we're going to make use of both of those, the hash and the ID file in a couple of minutes. And information on his local client where is his ID file located on his local client. So we've sent this all back to us. And I'm going to have to hurry because I really want to show you that. Yeah, if you all want to get up and get some more since we're running behind, if you all need some water, feel free to get up and get it. Now we've got the ID file, we have his hash. The victim's hash and his ID file. And this is where it really starts to get interesting because the password hashes of Lotus Notes are not salted. Unix actually went through this 15 years ago. They figured out they had to use salted hashes. Windows recently went through this. Lotus is just now fixing to go through this. The hashes are static. In other words, this hash of a password, the top line here, will always equal password. I'm sorry, password will always result in that hash. Secret will always result in this. And the next one. So this makes it really easy to create a whole bunch of known passwords or dictionary within a database and build a brute-forcer. Yes, but there's still problems with it because in 503 they had actually implemented an option to upgrade to a stronger hash mechanism. And in 503, although that was implemented, it was broken. So you could actually select in your server profile document and always create new users using the stronger hashing algorithm. And in 504 they fixed it because in 503 it would not do that by default. In 504, oh, thank you. Don't give up. In 504 they had actually fixed that. But I just actually realized the other day on 506 it wasn't working again. So what you have to do is you actually have to manually go back and upgrade all your users individually. And I'll show you how to do that in a minute. But you have to go back and upgrade to the stronger hash, which is the salted hash. So where this gets really interesting is if... I'll minimize this. We're now logged in as the attacker. All right. I'm going to close this so we can see what we're doing a bit better.  I'm going to detach the ID file to my desktop, which there it is. And I'm going to switch the ID file. Oops. I forgot one thing. I'm going to copy and paste this hash if I can even see it. And when I copy this hash, I'm going to switch to the next user or switch to the ID file I just retrieved. Now, I have this hash pasted in my clipboard. I've got to switch to find the icon here. I had to move this icon over here. This is the program Open Sesame, which will be released in a week, a week from today. Both binaries and source code will be downloadable from notessecurity.com. What we're going to do is we're going to launch this. And since those are static hashes and they're in static places in memory of the client, although they do change slightly between each version, Sesame knows about where those are actually supposed to be stored at. So the hash you're seeing there is actually my hash as the attacker, but I'm going to paste in the hash from the victim now.  Now we have, you can't really see it here, but we pasted in the hash in the middle box and the bottom box actually says new user password. So I'd like for this gentleman here to type in whatever password you'd like to be able to use. Okay. I can't even see the screen. I've got to switch back here. Okay, hang on, I have a problem. I've got a parentheses in there, that's why. Yeah, do it again. I think I've got a bug. Okay, I'm trying to hurry. Hold this. Meanwhile, why doesn't everybody take this opportunity to get some more water? Ten minutes before I did a run through, and it worked, but for some reason, something's weird in my environment, it's acting a little hokey. Since we're running short on time, I'm going to skip this part of the demo. If you want to see it, come see me afterwards. We'll go sit out there and I'll show you how it works. A couple of people actually saw this last year, I can talk to one of them if you want to ask somebody if it really works, it does. But what it would have actually done, we should have been able to paste in the hash, and it would have appended that hash that we just pasted in to the new location where it is in memory. And so he would have been able to type in his password, which he was typing in clueless, and he would have been able to authenticate with the ID file using that. He then could have actually logged out of notes, and the new user could have gone back in and typed in his password, and everything would have worked just fine. So I apologize that it's not working there. So we have the ID file validation, which unfortunately don't work for this show. But it does typically actually work. One of the things after we did this at DEF CON, Lotus came back and said, you had to have physical access, so they don't consider it a serious threat. But as you saw, we actually were able to send, get the user to actually send us his user ID and his hash, which then physical access becomes a moot point. So you could actually do this remotely. Yeah. Yes, it is. Yes, it has to be the same. What I hope to have in the next release of Sesame is it will actually pull up from the session in memory. So I'll actually have some VB script that will do that for us. Yeah. Yeah, he asked if the server, or I'm sorry, the minister actually checks his mail on the server console or the server is running. If we could actually have it send the server ID file back to us, which doesn't have a password, yes, we could actually have that done as well. Yeah, that's why he was just asking, this demo actually did, but I'm working on a new VB script that will actually yank it directly from the memory of the user. Because it's a static, each minor release of domino, or of the notes client, stores the hash in a static location in memory. So if you know which version it is, then you just go to that memory location and yank it out. That's actually how what Sesame does is it knows where that hash is and it plugs it in right behind it. And we don't actually overwrite it, because that way, when the attacker can log back out if he's actually got Sesame on a floppy disk, he presses the F5 key, walks away, and the original user can come back in, type in his password, and never know the difference. Yeah, in this demonstration, yeah, the VB script, since we're pulling the HTTP password there, what he asked was does it rely, the HTTP password in this, does it rely on not having the salted password? And in this it does, but even though you may upgrade to the HTTP password and the names and address, but to having the salted password in the ID file and on the notes client, it's still unsalted. Right now in this demonstration, I was pulling it from the names and address, but there's going to be a new version of the VB script, which I also intend on releasing. And then that should be able to pull it from memory. Right, if you try to retrieve it from the name and address book. But because if you upgrade to the salted, it doesn't change what the client itself is actually using. That only affects the web, for the web-based authentication. Yeah, I'm sorry. I know this is part of your business and everything, but when you do the full disclosure, is it going to cover, like, the script? I know you said you can put the Sesame out there. Are you going to have the scripts and the possible corrections for the problems? Okay, so all that will be out there. And that's going to be at the notes-security.com. Yeah, there's also 58 Nessa scripts, which I've written. 58 for every single database of the default installation that goes through... Mm-hmm. Hey, they downplayed it. They did that with everybody. You'll be able to go, like, hey, guys, look at this. Fix it. Please? All right, we're not going to take a break since we're running so late. Okay, on the access control list, we're going to go ahead and skip straight to the server and I'm on Hurry On, so we can let you guys out of here on the access control list of the server. Like I just said, I've got 58 scripts for Nessus. Unfortunately, the author of Nessus was at Black Hat, and I've yet to actually run into him. I actually had to attend a different session, so I didn't get to see his presentation, but I've been working with Nessus for about the last year and a half. And I have 58 scripts just for the server side of Notes databases, because there's 58 databases by default, which are installed on your Domino server. And if you're running a Domino in a web configuration, you probably haven't gone through and checked every single database. And the interface, ACL Reporter is a good tool. If you're not familiar with it, I think it's IBM. You're talking about the Notes database that you run. ACL Reporter is a tool that will run on your Notes server. It will collect up all of your access controls and flag those that have changed and those that deviate from a rule set, so that if you've got a rule that says default access is always defaulted to no access and somebody changes it, it flags it for you, you can go back and see who did it and correct it. Unfortunately, I found most admins will run it and not check the logs. But ACL Reporter, I'm not sure how much it costs. It is possible to duplicate the functionality in Lotus Script, though. Yeah, IBM actually has a database tool. It's actually a database application. You can run on your server, and it will do something very similar in a database as all the current ACLs. Like I said, I've been using NASA for quite a while, and this will do the same thing from the web server standpoint. We didn't have enough problems already. Skip to the next one, I already said that one. All right, these are crucial ones to look at. A lot of times when we've actually, when I've gone out and looked at somebody's server, quite often, well, typically the name is in the address book. If it's actually ever been used as a staging server, staging server, in other words, they've done a whole lot of development on it, and they've ramped it up to a production box for development purposes. A lot of times the admins just gave full access to the name in the address book. Forgot to go back and change it. You need to make sure at least at the minimum, check the top two, name and address book and catalog. The reason catalog, catalog has, its sole purpose is actually to catalog everything on your domino environment. And that includes documenting the access control list of each database. I've actually, one of the nested scripts I've got will go off and query the catalog database, retrieve with just a couple of XML requests. Not a hundred or anything like that, just a couple of requests and we'll pull back everything that's under default and no signature. And since, like the point I made earlier, since most administrators have not created a view in their domlog for XML request, they filtered out everything else. They're only pulling back HTML, what HTML requests were made. They'll never actually see what I've just done. And I now know we're all third party and internally developed notes applications result on their server and what the access control lists are. The server ID. There's, we have still to this day in the notes documentation, there's kind of a quandary with this. Regarding the ID file, like the gentleman over here made the comment, most servers do not have an ID file, do not have a password on their ID file. That ID file for the server is technically the same as a user ID file. The only difference is the information stored in the names and address book. I can use a server ID, load it up in my client, and access another server, not the same server, but another server within the same environment or the same organization, and be able to read it just as if, almost if I was a server. There's of course some exceptions if a database actually has the type and the ACL set to only a server, they'll see I'm accessing it with the client, and it won't let me open that. But a lot, since a lot of you guys don't always have that set, I'll be able to get in and access that database. The ID file, another really good point was the directory traversal. If the ID file was actually in the data directory, then I could retrieve it from the data directory across the web server. So you need to make sure you've got passwords on it. The quandary where I was mentioning was that Lotus actually recommends if you want to use the auto restart functionality, don't set a password on that server ID file. So you've got one part of the documentation that says put a password on that ID file, and another part says don't. So here with the, this is going to get just a touch on the web server. There'll be more information since we're not probably going to get to this. This is where it actually has to do with creative surfing. The URL language for the Domino server is extremely powerful and most applications make use of it. Where we run into problems is that with all that functionality you can do things, you can retrieve information on a badly designed databases. This includes the names and address book. Here we have just kind of a synopsis of how the language is actually structured. I would ask you to consult the developers and the administrators databases, the help databases for the administrator and the developers for more information. But this is basically how it's constructed. Using this you can actually come across and give you a very, very quick demo. I need an explorer. In the example I'm going to show you to retrieve a list of users. The way Lotus actually intended you to see this information from a web browser. I've actually got the literal view name described, appended here after this. In the URL we have the host name, we have the database name, and we have the view name, which is people. What that actually produces for us is what Lotus actually wanted you to see, and it looks like this. We have the listing of the users here, so I'll just pick one of these. As you can see, since we don't actually have... Actually, I believe we do. But if we did not have... No, we don't. We do not have edit access to this, to the name and address, but... So as a result, we cannot see the internet password here. But if we actually just change calling it from the people view to the dollar users view... I can't really read it, but I think that's it. No, that's not it. These screens are just horrible. I'm sorry. Here is the dollar users view. Now we actually... This is a different view in notes, and if we scroll all the way over here, we'll see that in this view, which we've actually changed in the URL itself, will produce a listing which will actually reveal all the HTTP passwords for us, even though we do not have sufficient rights to actually read it. And there they are. They just went by. I'm having a real problem with these screens, guys. So there... You can't really read this, and I apologize again, but these are the HTTP hashes here, where we did not have enough access right underneath the viewable view under the hidden view using the dollar users view. We can now actually pull all this information up, and we can retrieve all the HTTP hashes and other information, which was not actually intended. We see this a lot, and this is what I was talking about with creative surfing by using the domino URL language. When you modify this, the URL syntax of this, you can actually obtain other views, which the developer did not continue to actually be able to retrieve data from. So we're going to skip through here. This is what I was just talking about. And here you've actually just... Since we don't have enough time, this is actually one of the nested scripts, which is running from the command line. And I'm actually retrieving all the information from the name and address book from the catalog database regarding other databases and what their access control lists are. And again, that will all be available on notessecurity.com. So to skip ahead to the conclusions here real quick, there's all sorts of multiple vulnerabilities. I'm sorry, we've only got six minutes left. That's why I'm having to hurry through here. Go ahead. Just move through about every 15 seconds. In terms of workstation security, you can get malicious code to execute. I think we did get that work in where everyone should have been able to see it. With stored forms, we can reset ECLs, bypass ECLs with OLE and API calls. On the domino server security, there's... Using the URL constructs, you can actually view unintended content. You can actually even modify that. You can upload content. You can modify the names and address book using some of those custom or creative web surfing constructs. The server ID can be stolen. Once you've got that, you can get into usually other areas of a notes network. Again, the server ID is just the same as the user ID file. So make sure you've got passwords on those. One other thing, coaxial karma has actually got a couple of utilities. A lot of people seem to be very unaware of this. The ACLs in the notes database are actually just text fields. If you're not familiar with that, I would ask you to actually check out either again my website or landofsilence.com. The utilities you have is called ACL modify and ACL enforce. Using those, you can actually turn on and off consistent enforcement of ACLs or you can actually even modify anything you want in the ACL. They're just little command line shell utilities for modifying databases. This is really important because if you get physical access or direct access to the file system of a domino server, there's basically no security left. Yes. ID can be obtained from the web server. The name and address book with malicious code and email from workstation and local drive gave you demonstrations on that. And unfortunately, I'm sorry, we didn't get open sesame to work there for you. I'll be glad to show you after this. Just come on up and we'll get that running. Again, most of these vulnerabilities can be dealt with using various workarounds. All this will be documented on my website. Go ahead. Don't store the user IDs in the NAB. This is really keen, really important. Don't do that. Store user ID file on your removable media or on an encrypted PGP disk. Don't store server ID files in the data directory. This is, I can typically, almost everyone does this and they name the server ID with the file name server ID. This makes it very easy for retrieving those. Keep it out of the data directory. Put it somewhere else, much more obscure. So it makes it more difficult for people to retrieve that. Choose different passwords. As this gentleman pointed out, choose different passwords for the ID file and the HTTP accounts. Use the strong password hash from Lotus. You'll have to manually upgrade that. Before you do it, I would ask that you check with any third party or internally developed applications to make sure it's going to work if you're using web-based applications because you may run into the problem that using the stronger hash may break those applications. Always exit Lotus whenever you leave your disk. Don't use the F5 functionality. Using Sesame, you can get around that. Enforce ACLs on all databases, but at the same time make sure your operating system is completely secure. Do not run it on a file server. Do not run it on a print server. Make sure it is completely tightened down.  We gotta hurry. Yeah, here again, here's the URL. So you'll be able to go to fallingdominoes.com. And that will basically take you to notessecurity.com. Falling Dominoes is the name of this presentation. But Notes Security, all this information and in greater detail will be available on this website. Keep checking on BugTrack. I've got security zone, Lotus's security zone up here, but don't rely on that as your chief security zone or security website for Lotus Notes vulnerabilities. They've not done a very good job on that, unfortunately. Make sure hire security consultants to come in and review experienced domino security consultants to review your infrastructure. If you're not doing that, you're taking risky, shouldn't be taken. And again, here's the URLs for landofsilence.com, counterpane, fallingdominoes, and notessecurity.com, where you'll be able to find more information later this week. And that's it. Are there any other questions? Yeah. Yes, if the target isn't in T-server? Yes, it will. Yes. Yeah? How many have I been... How many have I actually done, been hired to do? We did some statistics and probably 80% of the domino servers that are out there, which are running as web servers, are completely open. I've been contracted by no more than 10 in the last year, which is really sad. Anyone else? Okay, you guys are free to go. Thank you.