Being the mother for a young girl definitely drew me to this program. There is a lot of emphasis on science and math for women, for young women. I wanted her to get that exposure. She's been to many residential camps before and this is her favorite and she wants to come back. So that says a lot to me. Accelerator is a unique opportunity, challenging and motivating.


This paper proposes a new artificial neural network, ANN, model called ANNSFP which uses statistical feature parameters, SFPs, as inputs instead of raw data to improve the accuracy of short-term solar irradiance forecasting, STSAF. The SFPs are derived from the historical data series and include the mean, standard deviation, maximum and minimum values of the irradiance and ambient temperature. The model structure is determined by cross-validation, CV, and the Levenberg-Marquart algorithm, LMA, is used for the network training. Simulation results show that the forecast accuracy of the proposed ANNSFP model is significantly better than the conventional ANN model using historical data series, ANN-HDS. This article was authored by Hongshan Zhao, Shersu, Xinjiang Mi, and others.


This is your weather forecast for the week beginning Wednesday, January 6th, 2021. We made it to a new year. I'm Chief Meteorologist John Ennsworth for Longmont Public Media. We have now a situation where the notable phases occur right on Wednesday, the beginning of our forecast period, Wednesday, January 6th, the start of this time. We have the last quarter moon rising at noon, sitting just about the next morning noon. Our drought conditions are not changing from Christmas week into New Year's week, pretty much the exact same map. And that's due to cooler air temperatures, which don't promote as much evaporation from the soil. We are getting many shots of snow to the higher mountains and the western slopes, and so hopefully that'll start to show up in the data in the weeks and months to come. Over the west, we have pretty extensive long-term drought. Out to the west coast, most of the Rockies into Texas and up into the northern plains. We should see some relief in the plains and on the west coast, as we'll see at the end of today's discussion. Precipitation over the last seven days, very minimal, a little bit out on the eastern plains. We have missing data here, and over a large part of the west. Something must have gone wrong with the data assimilation system at the weather service. So we have just the data right here, and that shows we didn't get anything. It was a little bit in the foothills, but nothing right along I-25. By Thursday AM, a ridge that's been in the west giving us warm, comfy weather has a little shortwave trough breaking through, kind of cutting in half momentarily. Do you have a storm down in the south and southeastern part of the US that's rolling along? This is creating some snow up in the northern mountains. I feel a cold front trailing along here, and we've got some rain down in the south, and we're still high and dry between those two. Okay, the next 10 days, our high temperature remains at 44. This is the typical or normal high temperature. The lows rise a degree from 17 to 18, and our expected temperatures bounce around in the low end of this normal range. We briefly on Sunday drop a little bit below normal, but this is really normal January weather, believe it or not. We do see a chance of precipitation on Friday, and some of the ensemble runs. More of the model runs are in agreement that we'll see some snow from Saturday afternoon or evening into Sunday morning won't amount to much. Then later next week we see some unsettled weather, and let's put the next 10 days in motion and see what's causing all this. So here comes that northern rocky trough, it just kind of washes out there. Here comes our trough digging in on Saturday, a little cut off low, pumping some moisture up, giving us that Saturday-Sunday storm, then into Monday, ridges back in the west. Here comes another trough, looking at the precipitation. We have a little bit of snow dying out up here as that shortwave trough vanishes. The next system comes in, cranks up briefly, creating upslope snow for us before it takes off down into Texas, and then as we go into the beginning of next week, we're getting very high in drive, even on surface high pressure systems sitting out here, there's a storm system kind of getting its act together up in the northern Rockies. It comes down and does something around Thursday, Friday, next week. That remains to be seen, it's a long ways away. Over the next 10 days, our snow expected is really minimal, right along I-25, the GFS on this run gives us really nothing, maybe a coating of snow to an inch near boulder, and then the mountains get four or five, six inches in places. The precipitation is equally very meager, five hundredths of an inch to maybe a tenth of an inch around Denver and the lower foothills, maybe up to a quarter inch and a touch more and spots up in the mountains, just not much there. Over the next seven days, as I said, we stayed in the low end of normal throughout this time. We have forties going into the weekend and forties coming out of the weekend and low to mid-thirties for the weekend itself, night time temperatures don't change much, bouncing around in the upper teens and low to mid-twenties. We have our chances of snow, mainly Saturday afternoon and evening, still fading off on Sunday morning. We saw in the December forecast prediction from NOAA that we would have above normal temperatures and below normal precipitation for Colorado and much of the West, and that certainly came true. Now, looking at January, we see above normal temperatures over most of Colorado, I guess the four corners area is close to normal, and for precipitation we're now in the normal area for this month, and so we'll get something down in New Mexico, Arizona, West Texas is where you have your below normal look at the Midwest and Great Plains with above normal precipitation amounts expected in the West Coast, or even better with Central and Northern California actually probably getting some good rains, so very beneficial, very needed out there. That's what I'm hoping to see better drop data in the weeks to come. For more frequent weather updates and local news, check out the Longmont Leader, LongmontLeader.com. I've been Chief Meteorologist John Insworth, keep looking out.


Hi everybody it's Sam at Mixercraft thank you for watching my tutorial today I'm going to be showing you how to make this memo holder. The idea for this one has actually come from this one that I have but it's just a bit boring so I thought I would do something and make it look much much prettier. Also I have used the Artaiser scratch boards and on the website when you go on they sell them also in block form and they're three and a half inch by three and a half inch pages you know like this and they're in a very similar box to this so I thought oh I'm gonna make one so I have cut down the 24 sheets that they sent me so I done a it was a Christmas box that they sent me and I was sharing it all and this was in there and I wanted a way that obviously I'm gonna use them and I come up with this and I just think it's really good so I'm gonna put some I'm gonna split them I'm gonna do half in the one I'm gonna make and that one's gonna be for me and this one I'm actually gonna be giving to somebody else so they've got the option to have the scratch pads and you've got your scratch wooden stick there or you can you know use your pen and have the normal pads I just think it looks really nice but the one I'm gonna do today is gonna match the lovely drawers that I made a few a week or so ago now and anybody that's not seen these before I remember these as children it's basically like one big scratch card and you just scratch it away and this one is multi-colored underneath rainbow patterned sheets and again you've got two ends on the stick there so you can do thick kind of lines and then you just get the kind of the top there that comes away you can also make these I have done something like that before it was a class many many years ago and you can make the kind of you paint on this you know like paint and then when it dries you can just scratch it off but it's yes they are fun they're brilliant you know to do with children and you get some great effects and also what was in the this pack that they sent me it was in this pack they have where is it this here and these are two postcards and one has silver underneath and one has gold and those are the ones I'm gonna keep and do some actual cards with them but for all of these I just thought it's not something I'm gonna sit down and do this kind of thing with but as a memo pad definitely because you could literally it writes just like a pen so you know to do this you can see there I've just wrote that normally and you can write quite quick with it so just as you would with a pen you know it's it's perfect so I'm gonna that way they will get used and also I can leave little notes and stuff like that like it's just fun so yeah that's what I've used there for that one and then for today's one I'm gonna use these ones here these are from WH Smith I got them a while back but I'm sure they're still in there and any block ones but you want something that is three and a half by three and a half which is your size that they come so this one is going to be today's one's gonna be a little bit smaller than this as well because I've done it as four by four but I think there's just bit too much of a gap there and because you have your kind of finger part there to take everything out you don't need there to be the gaps on the side so the next one today's one's gonna be a bit shorter so let me show you also there's the pens on the back there really easy and I've got my little metal corners on and it all looks very nice so let me show you how to make it okay so for today's one I'm using the touch of romance papers by the paper tree this is the same one that I used to make let me grab it this beautiful piece here so many of you are making this now you are sharing it over on the mixed up crafters Facebook group and yours yeah you all seem to be really enjoying it so what I also thought is that you could if you wanted to stick this onto it will fit two of them side by side of these little memos that we're gonna make so when we get to that point I'll show you how you could attach it to this if you wanted to also if you have missed the tutorial for that one if you just click up here you'll be able to see it so that's the papers I'm using so I've already gone ahead and cut what I need so for the base I have cut this to let me grab my other ruler actually so this is a piece of four and a quarter squared for the base for the paper to cover that you want a piece of five and a quarter squared so I've gone for the stripe but if you're gonna stick it onto your unit if you've made that then you won't need the base all right and then you want four pieces of four by 12 but you well we will trim one of them so yeah but that's the size you need but what I've wanted to try and do with this one one of the papers is like this so that's the top this is the bottom and then this was in between so you can see there there's my 12 by 12 piece of pattern paper so I thought I use that to line the inside and then with these I'm gonna you we're gonna actually stick two of them together because you need it's about 16 inches you need a length all together and obviously we've only got 12 inches with our papers so by sticking them together then it will wrap all the way around and then I'm going to use this greyboard so the greyboard I'm using is the two mil and it's that one I've been getting from Amazon and it's really nice so but you can also use the phone board if you've got it but well actually yeah you should be able to because the length that I give you in the paper you might just have to keep yours a bit longer and maybe not trim it because that's five mil whereas this is two mil so there will be a little bit extra so you want three pieces that are two and a half by three and three quarters and that's going to be for the back and the sides and then for the front part and that's where we'll be able to kind of lift out the little post it's all the you know the memo bits then this is one and a quarter by two and a half okay so it's really easy and the metal corners I'm gonna have to have a little root around for in a minute because I want silver ones for this one so I'm not going to put mine on top of that but I do want everything to kind of match such as to get a look nice so two pieces make sure your patterns facing down the joint is going to go on the back of that and then the pens are going to cover it so don't worry too much but you want to stick about an inch over the top of it so grab my other one there so you want to just add some glue so it covers one inch make sure I've got my pattern the same way up I'm using my grid here because these squares are one inch so I can just stick it like so so just stick that down and then you want to grab your three larger pieces and you're going to stick these in the middle so you want to make sure you've got even overhang here then these two are going to go on the edge basically just going to be like we're making a mini album this is our front cover so we do it just the same way the gap you want to leave in between is about one eighth of an inch that's where you want to kind of be I always when you fold up the side of the gray board this is usually what you do you fold that up so it's next to this piece and then you kind of keep it in place and let it kind of come down just moved a bit and that's do it again that's the gap you have but I always then like to pull it out just a little bit more and that way it just means that you definitely won't get any cracking with your paper because I am using a paper this one here is hundred and yeah 160 gsm so it's got more of a paper feel but it's perfect for these kind of projects because it will you know it's just better to fold and stuff start to creep into 200 gsm and higher you have a lot more bulk when you're kind of bending on corners and stuff like that so I'm just going to I'm using the cloud glue all the way through this I've run out otherwise I would use that it's I've turned upside down but it just wouldn't be enough for this project I need to order some more otherwise I would have used the book binding but I'm just going to use my calau here so I'm just gonna pop some blue glue on the back there and then just stick this down in the middle of this paper like so and then do the same with all of the pieces and just make sure you've got the same 1 eighth of an inch gap in between them all okay so while that's kind of just drying just bring up the sides and like I said if you are using the foam board then I wouldn't trim this just yet because I would make sure you've got enough to wrap around or once you see where I trim it then you can trim yours accordingly but otherwise you should be able to use that foam board okay so just folded those pieces up and then I just want to make sure I've got the same amount kind of overhanging there so just do that with like I said with the foam board if that's what you're using keep these bits always handy scraps and then we just want to stick this over again like as if we were making a mini album so first of all I'm just going to run the glue along the side of the grayboard there because you do have you know that two mil so the paper will stick to the side and then we're going to bring in the sides of each corner so I'm just going to run some glue so it just kind of covers all of the corner so and then just bring the corner into that grayboard like that so you want a nice square shape actually stuck onto the grayboard so again around to that one you want it to you know really wrap around the corners there and then do the same like so and then I like to use my bone folder but if you just kind of push it down into start down here actually just push it into the corners just to really just get everything stuck down and then just pop some glue on the top and just kind of bring up the side and then you can just fold that right over and then do the same on the other side then again on the very ends here as well okay and then you just want to very carefully if you I like to use my other bone folder which I can't find is just kind of pushing the paper just carefully on all of those kind of joints and then it will all come around like this you want nice right angles and you shouldn't have any cracking because you've given yourself plenty of room but now that's the back but I've got nice little bits of the pattern coming up along the side there it's just going to match really well the next we want to decorate the inside so that's when you need this piece but what you want to do is we're going to stick them both together again so let's just do that a second so flip them over again stick one inch just overlapping and then we're going to stick this now on here but we need to trim it down so the 12 inches isn't long enough so unless you just want to cut a little bit and join it that way but I just want to keep that join again in the middle just so that everything kind of looks neat so that's going to stick over like so but you want this overhang here okay because we're going to use that to stick it onto this piece alright so what you want to do here is just with a pencil so I've got my join in the middle of the middle square there I've come down a little bit because obviously you've got all of that coming over there anyway and you'll probably have a contrasting color so you'll be able to see it a bit better but I'm just going to do a little pencil mark there and there let's just trim this then you'll see what I've done okay so now I've got that piece and then it will stick in there like so but you want it to overhang all right we want a nice overhang just so we've got plenty to be able to stick it down so now I'm going to run yeah I'm going to put the glue onto here but don't go right up we can go down to the very bottom but don't go right up to the very top because the paper won't go all the way up there you should come down just a little bit I'll come down about quarter of an inch one eighth of an inch something like that and then go right down to the bottom this has become a real nice strong piece you know storage or stationery so pop that one all down there let it kind of grab but then we're going to start molding it into the shape before it's completely set so I'm just going to spread out some of that glue okay so just start again with your bone folder just kind of you know pushing it into that fold just to help it don't push it so hard that you crack it like that you've got to be really careful I didn't even really need to do that it's alright you're not going to see it it's going to be completely hidden yes gone yeah so try not to push down too much but you want to start getting it into that shape like that and then you can cut up because you can see now the the creases you're going to cut up to the bottom of the board we're going to fold this all in and create a base just so you've got something to stick it to but now you'll be able to fold that all around and if you get anything where it kind of lifts just go in there and just stick that down but you want to kind of keep it in that square shape like so alright so just spend a minute making sure everything's dry and you're happy with how it looks okay so that's all drying what you want to do now is just take a little wedge off of each of the corners because we we need to fold these into a nice square shape so just a little bit just don't want it to kind of buckle at all you want it to be nice and flat what's going to happen next is you want to carefully as of why you want to make sure it's all dry but you're going to fold this back on itself so it lines up perfectly with the bottom there alright so just go along like so now we can stick these it doesn't matter that they don't join and you know that's not what we're aiming for we just want a nice surface for us to be able to stick everything but if you imagine it's all going to be like that you can stick those front pieces over the top we're just going to add glue and you want to keep it in that square shape but that means that we can just add glue to that once we've done this piece stick it onto that and then we're going to put a nice decorative piece of paper over the top of that and it's all secure so you'll need to do this piece if you just want to have it on that piece that I've showed you I'll bring it over in a moment but what we do for us is we just stick this on so you just want to add some glue to the top and you're going to cover this in exactly the same way so bring the corners in like we did and then if you've got your metal corner protectors we're going to put them on as well then again just going to add glue to just like corners and then lift them all up in exactly the same way then glue the sides and fold them all in so I'm going to do all that on high speed okay so that's all covered and I've also just covered it with a piece on the very back which is four by four okay so now we got this ready and then I've got my corners here I've got my trusty little hammer I bet a few people asked me about this this is from a very old dovecraft tool craft toolkit and inside I wonder if I still got the tin bottom of my drawer and in here you've got a little craft mat you've got pliers you've got a pokey tool you've got big needles you've got a different oh that's for hammering yeah that can do your eyelet sets you've got different hole punches there if you want to obviously hammer them tweezers and it's yeah it's quite an old one had it a long time sure there was a date 2007 dovecraft it says on there so but anyway it's just really handy and I always use the hammer so I've got some of these decorative corners here my size is 15 mil by five roughly around that so you just pop them on there I'm going to turn it over and then you just want to just gently tap more hammer actually each corner like so so I'm just going to do that on the other three corners okay so that's the base done obviously if you're not doing the base you don't need to worry about any of that but this you can now start to see what I'm talking about so this is going to sit on here it's going to sit actually just in all of those little metal corners and then we're going to cover it with decorative paper on top and that's it it's done so what you want to do first we need to start squaring this off so I'm going to just cover a corner of the back one you end up putting glue on all of it in a minute but just for the moment and then just just focus on the back to and you just want to maybe use your grid or something where you can make sure that you've got all your sides straight and just stick that down for a minute I'm just going to flip it over again keeping it nice and straight just to make sure I really got that glue stuck like so so that's a perfect square now and then you just want to do the same with these ones so I'm just going to pop some glue on the backs of those and again get a real nice right angle and just bring that in so it really doesn't matter what it looks like because you're going to be covering well this is going to stick right to the base but you're going to be covering it I'm just going to turn it over because it's easier to keep it all lined up with my mat okay so you'll now have something like that now if you would want to oh it's gone off a little bit there it's just trim that off there you don't want anything coming over there we go now if you want to cut a piece for the bottom as well as the inside you can I mean I've got this piece here so I might just just for an extra reinforcement let's cut this down to four by four and stick that underneath just because oh no needs to be smaller than that sorry so that was four and three quarters four and three quarters by four and three quarters and then I'm just going to add some glue just on all of this that would just give you a bigger surface because there's not the gap there to then stick it on to either that or the unit if you've made that and you want to kind of it you know extend it and add more to it just turn that over okay so now if I grab my unit you could sit this on the top and if I pop it on its side you'll see how that's going to look and you could sit two side by side you could extend it you could have bigger you know papers in there you can do all sorts you could put your pens on the side you know there's loads and loads of options but I just thought I would share that in case that is something you want to do but now we're going to stick this one onto here so it's all nice and square so all I need to do now is just add glue on top of this if you want you can use hot glue but I'm going to stick with my callow and I'll just put something weighty on there just to keep it down I'm going right up to the very edge of this time because obviously it will stick to all of that just want to make sure you've got a nice even little overhang there you might decide you want a bigger base you know you can you could do a larger one and have a pen pot on the side or something this there's lots and lots of you know ways to make this you know into something more but I'm just going to hold that there I'm going to pop that on there and just hold the sides with my fingers for a minute okay so I've just cut this piece to go inside but I'm thinking I'm probably just going to stick with the Y actually I should have just it was a piece of scrap this is three and three-quarters yeah three and three-quarters squared so I'll add it onto this actually and it just covers up all of that inside then and just again add some more strength to your little holder okay so that's finished now we need to do the pen bit but I'll just show you how this will look so let's just pop some of these in there I try to look for white ones but I couldn't so I'm just using these ones I've got and it will hold a lot in there this is like I said really handy yeah it's great for the kitchen as well so you can do it in the colors of your kitchen but it will fit three of them in there perfectly so the pen bit is again it's completely optional but I'm tempted to use I just not let's use this piece here I've got this piece of scrap so the width I done I doubled this up so this is reinforced so what we got one and a half so three should be able to use this bit actually so three and you want to fold this in half I'm just folding it because it's paper if you'd rather score then obviously do so and then you're going to glue that together I put my glue so this is by sorry I didn't give the measurement it will be in my blog as normal this is three by eight and a quarter it's just scrap you will trim this so just stick fold it over and just stick it together so you're just reinforcing that just making it a bit stronger okay and then on the back I'm just going to take all that back out again okay so what's going to happen is there's going to wrap around so on here I've put some hot glue and then wrapped it around the pen hot glue and then wrapped it around this wooden piece but what I actually did is I wrapped it around a pencil because I figured that I'm not always going to have this in there once this is gone then I won't be using that so a pencil is obviously you know that's a standard size you know that's in every household that will fit in there nicely so pencil and pens quite nice if you're going to use a pen I would say if you're giving it maybe you're selling these at craft fairs then go for something that's a bit more you know of a standard size you might put some pens in them to sell with it I would think that's nice as well so you know play around see what works for you but I've got my hot glue on just need to wait for it to warm up a little bit more and you can also have three in there I mean it would easily fit more you can see mine's gone slightly off to the left there I want to try and get this more in the middle this time but I'm going to let's have a little look I've got a nice pen in here that will go I think a bit better it's this lovely cat kits some one which I love it's got little dogs on it so I think yeah that look quite nice so I'm going to have that one there and then I'm just going to use a pencil again just to give me an idea but I want it to be like so okay so you want to decide that's going to be there and there she want there to be a gap big enough for it to stick to the thing so I'm going to start this one about there so you just want to kind of twist it out that way a bit I'm going to run some hot glue make sure I'm in shot sometimes I lean too far back on my chair but I'm just going to run a little bead of glue there and then that one I'm going to pop just about there so it's like halfway up if any glue oozes out you can peel it away before it really sets now that one will go like so you want to really push it up as close as possible kind of just kind of mark with your finger where it's going to kind of be so and then I can just lift that up and I can see where I need to put that glue again I'm going to pop the pen in there just so I get the right angle and then bring that down like so yeah it's going to hold in there perfectly if you want to come higher up I did go a little bit higher up on that one but that's still okay I'm not too worried and take that away and now you've got a little area for you know your pen so then I'm going to grab the pencil and that one's going to be there again I've gone a bit off to the left so you can play around a bit more I seem to just do that I guess I could fit three in quite well so maybe I will and then again just mark that one a little bit and I just thought that's actually the one that we want to cut so I'm just going to pull that glue dries just snip it so the pencil in and then just let that grab once it's at the back it looks really good that's how it looks you know you just don't see any of that now I'm gonna actually I'm gonna put the pink the yellow and the green because the green I think goes quite nice with this paper so I'm gonna have the green on top pink on the bottom it's much better that size you just you have a little bit of shake but not that big gap that that one's got although that still looks nice now just let them all kind of level off the colors go quite well with this actually and there you have it it's really really pretty I love these papers it just yeah they're just gorgeous and there's some more extra there so I actually ended up using so I said a well it was eight and that's five and a half so it was only what one two about three inches that you you needed to create that but you could certainly get an extra one in there so have a play around I mean these are just for me that one's been given to a friend who I know is gonna like it as it is anyway but now I'm really pleased with how that's come together they have it my two little memo pad holders I'm super pleased with them I think they look very pretty this one matches my other storage now very nicely and again if some of you do stick it to the drawer storage that I shared earlier I'd love to see you know what it looks like and what you've done to kind of adapt it and yeah if you do want to go for something more you know patterned and darker because this was obviously using those black ones there what I'm gonna do is I'm gonna keep I'm gonna half them so I'm gonna have some of these in here and then you know some of the papers I'll jump a little up but like I said see if you can find any white blocks I can't find any because I wanted them for this tutorial but because I've got the coloured ones I thought actually they don't look too bad in that one at all so yeah I hope you've enjoyed it I hope you like it please give me a thumbs up if you've enjoyed today's tutorial and consider subscribing so you get to see more. See you soon. Bye!


We have almost an hour for discussion. I wonder if I might just take my prerogative as moderator to try to follow in your footsteps, Kelly, and just throw out a series of questions for each of the panelists to try to answer, to try to get at some of the issues you raised at the beginning of this session. Would that be all right? Okay, and then throw it open to questions from everybody. But I thought what would be really interesting to hear from each of the panelists is given that this is a panel about relationships, if you could talk about at least one way that you know that a relationship is successful, and one way that you think would be useful to help build a relationship, any relationship. So maybe we can start at the end and go down unless somebody else wants to jump in. All right, having to go first here is tough. So relationships being successful. I think one of the things that's tricky in this relationship building is it often takes a while for the relationships to develop. And so any metric for success has to be timed appropriately. So I'm not answering your question exactly. I think for me, the primary way that we have determined whether our relationships were successful is whether we conducted research that the community felt was meaningful and important to them. And then we're able to disseminate that research in a way that was accessible to the community and had direct, we could tie a direct implication to either changes in care or changes in policy. And that was all mediated through stakeholder relationships. I'm gonna think a little bit more. I might be able to come up with some more nuance there, but really that's kind of the gestalt of how we determine success in that relationship building. And of course, if people are willing to stay engaged, that always is a good sign of relationship building.  If there was one way you could, something that you could name that you think would help build or maintain a relationship. Right, I mean, so I think primarily for us, the ways that we've been most successful in retaining those relationships is making sure that it's a meaningful engagement so that people really do have some power and some control. Because the way I've seen it fail is when you invite people in and they have a very preliminary role very early and it's quite minimal and people are unwilling to really make serious change. And so I think that that is really the key in our engagement for success is that we truly are willing to change things very significantly, change our approaches, our methods, our priorities based on that engagement. I think, you know, I mentioned before that we have patients representing over 2,300 different conditions, but we probably excel in about 25 to 30 of them. And the reason for that is that in order to be successful with building relationships is that you need to take the time to understand the condition, understand the trajectory of that condition, understand the kinds of questions you need to ask someone when they start looking at the diagnostic process versus someone who's been living with that condition for many years. So it actually takes us about 12 months to decide that we're going to fully, deeply dive into a particular condition from the point of making that decision to the point of having what we believe is a successful relationship. And a lot of that is built upon ethnographic interviewing that's done with patients in the field via Skype, with clinicians, with researchers, really trying to understand what are the tensions and the problems that we actually on our platform could begin to address. So that's actually a big undertaking on our part and actually requires a lot of investment on our part. So we oftentimes are looking for collaborators to help support some of that work. We will never get to all 2,300 conditions, but we believe as we chip away and we deeply dive into these, we will begin to start to create a framework within which others can start to think about using that as well. A few things that we've learned along the way, I think that are just absolutely critical to success in any relationship. And that is, as I think we've already heard, engaging people in something meaningful. But really, and we heard this earlier as well, that research questions in and of themselves aren't necessarily the questions patients are asking. So we need to be thinking about what are the kinds of things that matter to them that we can reframe into a research question and then find ways of being able to measure that and then learn from it and then collectively share that. So the other piece would be that we have learned one key measure of engagement for our patients is making a commitment to them that if you give us something, we're gonna give you something back. That can be as simple as telling us your age and then we give you an infographic that tells you all the other people within that community that have that same age or a co-morbidity that you might have in conjunction with your primary condition and give you an instant infographic that shows you the other people like that. So I think it's an important piece that we have to keep remembering. And then finally, the dissemination of research results. We actually don't wait until the publication. We will share with those who participated in a survey the preliminary results so that they can get some insights right away from that. And then after that, the publication might come out and be more specific about the details. So I think there's still a bit of an open question what is successful in patient contact. PCORI had at one point a must measure enrollment or engagement and everyone in the country scratched their heads on that really meant in terms of is it for now, is it sustainable? So I think right now, what I would see is being successful is from purely from a clinical research recruitment perspective is sustained retention and over time. We've got lots of experiences of patients and participants who do maintain participation in these research studies for no really good reason other than their loyal. And we need to determine how that can be really more methodologized, made methodological such that we can understand how to retain and ensure those relationships. I think it's very much echoing everything else here. How we seek to do that, I think there's two parts of the CENA project that I think are trying to address that. One is that during the patient reported outcomes which is where the participants sign up and start answering questions about the disease efficacy, we give them immediate feedback of how everyone else has answered those questions like sub-second response time. If you've got this condition, if you've got that, we tell you how everyone else has answered it. So it's very engaging to learn dynamically while you're engaging how everything else is doing. It's a very real time, active and actually fantastic environment. But as part of that, we're also establishing conditions by which we can communicate with them to give them value ultimately over time and how do they best respond? Do they want to participate actively? Do they want to just get outcomes? Do they want to merely lurk? Do what they want to know more about their communities? So we're trying to establish mechanisms by which we can supply those values back but during the process of them actually engaging themselves, we're also simultaneously telling them how their community is also being engaged. That's, I think, our two chunks. I'm really happy to follow you because one of the ways that we also measure successes by retaining engagement, it's so difficult to do with working class families that are working two or three jobs and then going home after work to take care of their children. So we do a few things to make sure, first, the organization's intergenerational, which means no one ever ages out because we believe community is defined by having everybody at the table and that we learn across generations from each other. We listen to body language. There's a lot of ways that people communicate without saying a word. We provide childcare, translation, and food. We meet when the community is accessible. And we also, one of the ways that we have been able to get people involved over and over again is by having and documenting and celebrating short-term, mid-term, and long-term successes. When people are involved in a campaign, whether it's to address a public health disparity or to gather data that shows that their condition is directly related to that condition, to that situation, it's important that they have some short-term successes because that's what keeps them coming back. It's something they can celebrate. It's something that comes out in the news. And then they stay in for the midterm and then they stay in for the long term. And so that, I think, is really important. People who are working really hard, who've got a lot of other stresses to deal with, need to know that the time that they're committing is actually manifesting in something positive that the community is benefiting from. And once they've seen that, they keep coming back. Okay, I'm gonna open it up to anybody, including the panelists, if they have questions. Yeah, thank you so much to all the speakers. Following up on sustainability question, I think we have some models out there, such as Framingham Heart Study, the Nurses Health Study, where there has been real ownership by all of the participants, which has required an awful lot of infrastructure of communication, et cetera. And I think those are successes maybe we can learn from. But a question I have, and Elizabeth, I appreciate your comments on having short-term, mid-term, keeping people engaged, but so much of research is multiple years and incremental. There is no big, oh, wow. In fact, it may be a negative wow, if that's a possibility. But how do we not get citizen scientist fatigue or citizen fatigue when the outcome is, oh, we didn't prove the null hypothesis, we got nowhere, onto the next study? I mean, it's hard to keep professional researchers engaged in those as well. So I think it's, or will only big, oh, wow research be appropriate for this. Can I just give you a brief answer? I think the problem with how you're framing it is that you're thinking in terms of outcomes and we think in terms of process. So the process of bringing people together builds trust, builds relationships, has people sharing information across the table. And those things are worth celebrating. Things change when you bring people together to work on a particular project and there are successes and those things have to be, so if you think about the outcome, and this is a country that always thinks about what's at the end, but when you have a justice framework, we know that a lot of the change happens in the process, those are the things that you have to lift. And we have a group of cancer survivors, these women who came to us because they lost their bus and we got their bus back and then they came back to us because they wanted to expand the median on 4th Avenue because there were 88 fatalities and deaths in five years and we got the thing expanded. And now they're involved in working with recovery folks and they know that's gonna take longer but because they've gotten things along the way and we've been able to succeed, it's made it, but then we're also multi-issue which makes it a little easier for us also. I'd like to just follow up because I think it is a really important question and one of the things that I think we need to be recognizing is that there's a sense of urgency amongst people who are living with illness that you just as researchers cannot appreciate and understand and that we need to find ways of being able to build in incremental opportunities for learning, one of the things that we have learned over time is that patients are actually learning and benefiting just from the process itself, I think we've just heard, but also that we can have interim outcomes that actually help to benefit whether it's a positive or negative one. The lithium study was a negative outcome but yet people learned a lot as a result of that. We also learned a lot as a company about what that takes to engage patients and understand them and listen to them carefully. And then lastly, I would say, sometimes as I mentioned before the research questions don't need to take so many years because the patient's kinds of questions may be different than the kinds of questions you're asking. So think about ways of reframing it so that we can maybe incrementally understand it better. I'll give you one example. We had one of our ALS patients who was scoring zero on the clinical endpoint for ALS, the functional rating score, so which meant she could no longer participate in clinical trial activities, said I'm still functional, I'm assisted in every possible way with ventilators and every other kind of device but I'm still a functional member of society. She actually worked with us and we actually published with her the extension items for extending the ALS FRS score so that you have three additional questions when you are using assisted devices. It did not take forever to get that done but the positive impact of having that patient participate in that and have her be a co-author on that paper was incredibly valuable. Can I just get people to raise their hands again and then I'll see if I can keep track of you. Let's see, okay. Okay, let's start here and then go this way. My question is about the process part of building trust and so on. Whether you're a patient and dealing with a for-profit medical services system or a community member who is exposed to toxins or who's denied services, would it make a difference if the trust-building process is with say a for-profit company or with a corporation that actually is funding the research that produces the pollution versus, so historically, there are interests of people in building trust in the authorities or researchers in building trust for purposes of controlling people and maybe even profiting as well as interests of people who seek democracy and economic and social justice. And I'm wondering how you parse the trust issue if trust could come from both a motivation, either a motivation of benefit or exploitation. Well, of course, in our prison projects, we are not directly channeling farmer trying to recruit patients, though indirectly we are, ultimately. We're trying to provide the environment search that there could be a trust relationship that the participant would recognize that entity as something they could contribute with, but it's an educational process. So right now our trust relationships are again, all led by advocates or of nonprofits or which are patients, but prominently, and all which are defining directly within their community is what they consider effective trust-based relationships. But, and then some of them, of course, have already given some serious thought to the more the for-profit or other conditions, but we haven't actually made that a priority for developing the trust upfront. The first part is just to go from the advocacy, go from the community, and then determine disease community by community, what their aspect, what their interests in some of those other relationships are. And some of our communities don't seem to have any direct interest in the for-profit relationships, so others, of course, might. So we're still testing it, but right now we're trying to still lead from the advocate, patient, participant as a representative of trust first and letting that community self-develop such that it can be in a position to answer the questions of secondary use or secondary engagement. So I think it's different based on who you're talking about, whether it's a for-profit or a non-profit and you can't always assume that the non-profit is always on the up and up either, right? Because that does happen. We've got a business in our community that has reached out to every non-profit with funds so that they can support and bring their members into supporting a plan that would end up in their displacement. And our organization is literally feels like, you know, like we're fighting all of them. So those are non-profits, but they get resources and they provide services. And the community they serve will be displaced by this corporation coming in and doing this. So the approach is different depending on who it is. We had a power plant company that wanted to expand its business and knew they couldn't get past us to expand and we put together a coalition because they wanted to reach out to the community. We were sort of like the first stop, you couldn't reach out to them without us getting in the way. Put together a community benefit agreement where they had to reduce net emissions. They could expand, but they had to use equipment that would reduce net emissions. That had never been done before. It was a peaker right next to our community that was dropping tons of knocks and socks on us. We meet with businesses all the time to try to get them to engage in practices where they can continue their business because we're EJ, so we have to work somewhere, but that they work in such a way that it's not killing our community. And so before they even get to the point where they need to, so we get funding so they can hire engineers so that we can look at their facility. But the trust thing doesn't come without the work being done. We can't trust unless we've had an engineer look at your plan. We can't trust unless we've gotten all of your information. And then the trust happens and we have been able to develop really good relationships with some businesses on the waterfront that have been open and willing to work with us in a way that's different. So we could support them and celebrate them as a business and the community can feel confident that this particular business is operating in a way that's cleaner and better. But it takes a lot of different efforts. Like me, there we go. I'll just say quickly. I mean, we do have financial relationships with biopharmaceutical companies and with us it's all about transparency. It's in the consent we have on our homepage. We have how your data is being used so we make it very transparent when and how we're sharing data. And we periodically survey our membership about their perceptions of the privacy and confidentiality and data usage. And we make sure that when it's necessary we alter our processes. So it's a rare disease community that's highly motivated. So I think I'll go back to this idea that it's very dependent on the community and what the community motivations are. As the one for profit company sitting here at the table, I feel like I do need to respond. I think one of the areas that we believe very firmly is we would not have a site at all if we haven't established the trust of our patients to participate with us. So we take that very, very seriously. We have an approach that really we adhere to which is patients first. So we do engage with pharmaceutical companies because actually they have a lot of money to be able to start exploring questions that actually matter to patients and especially patients who are looking for solutions to problems that we're not finding answers to. So, but we do that very intentionally and very carefully. And as I mentioned before, when we work with external researchers, whether it's an academic, clinical, non-profit environment or a for-profit pharmaceutical company, we actually vet that project very carefully. It goes through an entire team and some of our patient advisors to determine does this seem to matter to you and are we putting patients first in this perspective? And then ultimately, if we can answer yes to those questions then we'll continue to move on and move forward with it. We actually reject many proposals that come our way because they are actually trying to answer questions of interest to the researcher and not necessarily to the community. So I think it's an important way. I think it does, you do need to parse that out some. We actually think of ourselves as not just for profit and in many ways we don't believe we could have built what we have today had we not had the investment of people who really believed in what we were doing. So, a non-profit can only go so far. I think we've been able to innovate more rapidly and to be able to grow more rapidly because of the investments that we've been able to generate. But without the trust of our patients we would not be able to grow at all regardless of how much money we had. I might just add something in here which is I think that your question raises this issue of conflict of interest which we brought up before because I think this is one place where the regulations kind of clash with sort of common sense and what we think we need to build relationships with trust because the regulations really focus a lot on for example financial conflicts of interest which might help with transparency in certain areas but not in other areas where the issues are not primarily financial. So for example, if you're thinking about maybe a patient advocacy group that is recruiting its own members to be in a clinical trial and is also involved in the design of that clinical trial there might be a conflict of interest there because if a pharmaceutical company were playing that same role you would say oh no you can't do that. That's you can't recruit and design the study and fund the study and do all that stuff. So I think the regulations and our thinking needs to kind of adjust to well what makes it okay and what makes it not okay is it oversight by a third party? Are there what, how do we negotiate that? Because it seems like you can't just sort of have one set of rules and have it apply in both the financial context and the non-financial. I know we had a hand up over here but that was you Pietro and then in the back. So I found the question about, can you hear me? The about relationship success interesting in particular your answers, the spectrum of answers was interesting in that it seemed to cover different interpretations of relationship success and who the stakeholder might be in that success. So in some cases it seemed like success, a successful relationship was about the success of the individual, in other cases it was more focused on the success of the community and in some cases it was literally the success of the relationship, how long the relationship would endure. And so for me that sort of raises the question of whether or not there's some sense of obligation to study these relationships, to formally assess the impact of the relationships on the various stakeholders and then be transparent about that so folks could make informed choices. So the first part of my question is to what extent do you already do that sort of thing and or believe that you should or shouldn't do that and then the other part I think relates back to Leah's point which is that for some people if you give them enough rope they'll hang themselves. So I think sometimes because I think everything you've described is motivated by positive intent that we sort of tend to assume that this is gonna be beneficial and so are we obligated to determine circumstances and when it wouldn't be beneficial and also to report that where do we draw the line of what we let people decide for themselves in a transparent environment versus what we would never ask them to do even if they wanted to do it. So is, so to what extent do you believe that there's an obligation maybe a moral ethical obligation to assess the impact of the relationship on the various stakeholders and then report that to all the potential stakeholders so they can make informed decisions versus just assuming that because the ultimate goal is a positive one that it's okay. Take a crack at it. I think one of the things that we've done early from our earliest days is to sort of gauge what are the perceptions of the members of our communities what do they perceive as benefits or maybe not benefits of participating in a site like patients like me. The interesting thing about it is that the results are generally always positive. They're rarely negative and I think when they are negative some of it is partly because the community isn't well developed and so it may be a little bit lonely yet. There might not be enough people there yet for them to feel part of some of the same experiences that others have. But I think overall the perceptions of the benefits of participating on the site actually lead us to better understand how we can then engage them better. So many of the perceptions are the more connectedness people have they perceive their outcomes are better. So the more people they're connecting with on the site those patients actually report improved outcomes. So it's like okay well then let's start thinking about how do we then identify how to engage patients a little differently. And I'm gonna give you a little bit of data science as well as the design science that we use. So one of the things that we did a few years ago is do something called persona-based design that comes out of the design world.  Whether they're people who are there to lurk in the forums and not ever participate or they're there to track and monitor their disease or there's some combination of the two. So we actually understand that a little bit better. And then the other is we have our engagement team and our community team who are constantly looking at the level of engagement people are having from the moment they come on all the way through and who drops off and who stays on. Our data science team then takes that same information and starts to create predictive modeling. So we actually now over seven years of time can now begin to think look at a person who joins the site today, understand some of the characteristics about them and predict whether they will be on the site three months from now. And then better understand what are the things that keep them engaged. And some of it is just the connectedness. Some of it is helping them improve their outcomes. For some in epilepsy it has to do with helping them better understand the side effects of their treatment so that they get better understand when they make a decision to go to the ER or not. That a lot of different things that come into play but that starts to build a trust and it also builds the relationship. So I think studying that is actually really important and so we continually try to evolve our user survey to kind of get a better sense and we just haven't found a lot of negative as yet. Okay, not to press the issue too much but and you know I think the world of patients like me I actually use patients like me. I'm a patient like me. And but it's conceivable to me that someone could in a site like patients like me have the perception of having a positive outcome. Yet relative to people who aren't engaged in the site have a negative outcome. So I believe my outcome was great but actually my life expectancy was less than someone who wouldn't have joined the site. And so that's just an example but to say do we have an obligation to actually measure and study those things and report those risks to people to say maybe even to say we haven't studied this and you might actually not be better off measured in these ways. I actually don't think about it in terms of studying it I think in terms of evaluating and that throughout the entire process you're evaluating whether or not people are meaningfully engaged whether you're meeting all of your outcomes and engaging people in the evaluation process constantly having conversations with them and getting your direction you have to be adaptable and getting your direction from listening to people and having those conversations. So that evaluation has to be an integral part of anything that we do. So I think from the perspective of NIH these issues of the role of pharma in creating the infrastructure that allows for engagement and collects the data and makes the data usable is really interesting to reflect about what kinds of questions can't get answered if that's the funding model and I'm thinking in particular from my own perspective about environmental factors in illness and prevention. There isn't really a comparable industry like pharma that can fund prevention and there's like if you're preventing disease by avoiding environmental toxics there's not a profit generating opportunity in the same kind of way. So NIH needs to think about who can make the investment in the infrastructure to answer those questions.  At some point we've got the Apples, the Googles, the Facebooks who are in the position to build partnerships for broad citizen science engagement and there's no clear profit or model for that or obligation to them there but they are there and so I think even though pharma would be therapeutic based other partners are I think equally important and NIH would be great to have a policy that would extend to better understanding how we can develop those models and designs. I would also add, I think it's a great comment and I would also add though that in our experience over the last couple of years with pharma for those that don't have a specific therapeutic molecule that they're studying but they still want to get some patient insight into a variety of different things they're actually starting to ask questions that are actually quite interesting around quality of care and understanding what does empowerment mean and starting to really try to get a better sense of what is it that motivates patients to do anything in particular around their health which actually I think is a real shift in thinking. When we first started most of the pharma money that was being thrown our way was from the marketing department and we weren't all that interested in that. We wanted to talk with the epidemiologists and the scientists and really think about what's our data start to answer questions about that really matter. Not necessarily whether the packaging is right although we did do a study on packaging and we informed them a lot about it but really the marketing piece of it wasn't as interesting to us as the research piece of it. So I think there's actually ways of being able to push that envelope a little bit by answering certain questions that matter to most people around the environment and finding ways of pushing the buttons of big companies like Google's and Facebook's and things like that for which this matters frankly so. Thanks. I want to maybe talk at a little higher level for a minute. So the title of the session is building the relationship and I think what we've heard and probably what we knew coming in is that there are multiple relationships and the way that the structure of how we think about research was about people who participated in research and people who carried it out but it's quite different than that and so I wonder if to hear from you all we could hear from you all about how you think about taking Kelly's direction to heart about generative work. What we need to know about that we don't already in terms of the relationships that are different in citizen science between those who participate, those who carry out research, whether they're researchers, whether they're private companies, pharmaceutical or PLM or the institutions where people work. I work at Johns Hopkins which has had a kind of checkered history frankly of doing research involving communities. So how is this different than what we've been doing in the past and what do we need to know from the perspective of generative ideas that we don't already as we go forward, we being the royal we here, NIH, us in the room, society, everybody. Just off the cuff I think we need to know if it's worth it to maintain these, I mean we need to determine if this sort of engagement can actually truly generate outcomes and at least moderately generate outcomes for the investment in the engagement that we're trying to address. We've seen it in small scales but very few of the citizen science projects we saw earlier are really, I mean they generate research data but are they generating positive benefit or economic benefit I guess. So I don't know, what don't we know? Is it worth it yet? It might not be. I'm thinking though from an LC perspective, right? Since that's why we're sitting in the room today among the reasons. So the kinds of ethics, law, policy, social implications questions that ought to be thought about and pursued. And maybe you don't have a thought about that but I wanna try to keep us, to me that's where the action ought to be and a lot of this, the general conversation is very interesting too but I wanna sort of dig in and ask, is there anything different going on here in the context of this notion of relationship? So I'm gonna echo something that was discussed previously which is this idea of access. So we can, from our side, we can educate which is really required, that idea of research imagination but we are still, I'll speak for myself but I suspect that applies to many people, we are still trying to engage a small subset of the patient caregiver population within our net. We're trying to engage abroad, we are engaging a small subset. So we still have really overriding issues of access to this whole process and representatives in our engagement. And I think that those issues, while it's not a new issue, is a really compelling issue that is going to have a major impact on how successful this whole enterprise is. So I sort of half answered your question but I do think that this is something that is an area, so does it work? So does it generate new ideas that are novel and important to communities but when we say important communities, are we talking about the top 5% of the community or are we talking about the community? Okay, I just wanna add, we work with RAND and Lifelines and we meet with them every Monday around 10 a.m. for about two hours. And I can say that maybe the recommendations and the things that we're saying just seem obvious but they're not obviously being operationalized. That maybe some of the input that we have may seem like well anybody can figure that out but in fact people are not doing these things. These relationships are not working in a lot of spaces partly because of the conventional training that people have because there's old school thinking about community. And so it's my hope being here that people can get out of that space and think about these things differently. It takes a lot of work. If you're someone who has a particular discipline, your discipline is to study, I don't know, epigenetics or something and that's your thing, right? That's your passion. It's very difficult for you to get out of that space and start thinking okay I gotta build these relationships. This is really important to collect the data so that we can get these outcomes. It's not the way that a lot of folks in this room have been trained but it is a process that is going to contribute not only to a body of work but to changing the landscape in a way that has to be changed. And so while some of it may not be, I don't know, higher order thinking, it is the kind of stuff that results in collecting data, getting policies changed as a result of data collection on the ground with our elected officials to getting people in our community access to resources that they may not typically have if we hadn't been able to prove that there's a connection between these different things statistically. I just wanna add something about sort of what's new in the Elsie part of what Elizabeth and Holly said and I think it's just a modification maybe of the old justice issues and so we have been talking about who benefits from research and we've moved towards, oh, now we study women and now we have to study minorities and we have to report that and be accountable. I know, but that's how the NIH, that's how it's discussed. But now we have to move that discussion into now what is the just way of engaging communities at other levels, the other levels that we've been talking about and perhaps maybe not just patients because we have been sort of focused on patients but I think there's a lot of other stakeholders in communities that might be considered in this citizen science space. So Sally, and then I know Effie had her hand up for a long time and Jason and then. Just real quick, I think just to follow on that I think the key piece here that NIH needs to really think about and this is not news to them is that we need to modernize the clinical trial and the research paradigms such that we are actually starting to address and get a sense of why people would wanna participate in research today under the current governance and legal issues and ethical ways that they have to deal with all of the constraints of getting into a research agenda. I think we need to really push that envelope because people that want to participate it's just that the barriers to participation have been relatively high. And so I think modernizing it, understanding what do we need to do to get to this action point does involve creating relationships with people to better understand what's keeping them from participating. But when we ask would you participate if you were given the opportunity to do so, they say yes. They tell us why they haven't been participating and a lot of times it's process and structure issues. So let's start to think about what those look like and I think we'll start to break down some of the barriers. My question follows closely to Jeff's and one of the new things that we're hearing and I think Sandra mentioned that earlier today and I think sadly you mentioned the work democratization of research. And so I'm wondering if the way we're using democratization is clear to all of us or we have criteria that need to be met in order to say yes we did democratize science or we did democratize research. Do you have a sense of what those criteria would be? Is it simply just involving more people? Is it simply increasing numbers? Is democracy or democratization requires a different degree of engagement more meaningful or more advanced or augmented or I don't know what. So I'd be interested in hearing your thoughts on that because I think this is one of the new things we're hearing in comparison to our standard research that maybe wasn't democratized. So take a shot at it. I think one of the things that just information access has done obviously is given a democratization to everyone to have a better access to something that they might not have had access to before in terms of just knowledge about things. Now when I say that I think in the context of trials and research studies that are being done in a place like NIH I think what we are offering is a heterogeneity that you just have never seen before and that the homogenization of research actually has not really given us the kind of insights we need for people to live as well as possible with the illnesses or conditions or situations or environments that they live with every day because we're not heterogeneous enough. So I think if there's one thing I think that democratization could start to provide us is access to people who would not typically have participated in something. And then I do think however that there does need to be some structure and process around that. That it's not just a free for all though crowdsourcing and the other ways of doing cell slide, I love cell slide. I'll go on that, identify cancer cells all the time. But I think the idea of doing what we are trying to do is not necessarily just corraling a lot of people and then suggesting that's going to answer all the questions. I think there does need to be some sort of science behind what that looks like. And hopefully we're starting in some of the examples we've talked about to create some of that. But I think the environment is there, the milieu is there. Now we have to figure out how to apply some of the traditional and yet contemporize them approaches to thinking about how we get the best answers possible with a more heterogeneous population. So I think there's the need for democratization and then there's the now. Right now, for example, the CENA project has, it finally recognizes that there's a very broad spectrum of how anyone defines their expectations of privacy, their nuance, what they wish, what they may know, what they wish, some participate actively in everything, give away their 23 music sequences, others do not. And this moves, this moves based on the social media, this moves on based on family and influences, moves on a thousand things. So we're trying to, we've established a mechanism by which that can be defined, characterized, that hopefully will allow anyone who wants to at least take the step of defining that ability to define that. And they can change that. Right now though, we're only testing that in a limited number of actual, much more focused, less democratic, I guess, disease communities because that's where our funding is right now. But the goal is, again, to identify and evaluate that as a method for recognizing that we all have certain expectations for respect, for representation, for privacy, and we need to define that in some way, and they will change, and they will change, and we need the mechanism for that to change, and then of course test that. Hi, it's a great panel. I'm having a lot of fun, and I'm trying to, trying my hardest to get my thoughts together, but it really touches on circles around a little bit. Many of the comments have already been made. But part of it is sort of painting a scenario that we are in such early days of citizen science in that we should anticipate in many ways that there are sort of infinite combinations of the ways a research study and its governance, its communication style, its presentation, its features could be set up, and that we've talked a little bit about sort of research literacy too, and that a lot of people are coming to this with very little to know, no research literacy. What is a good scientific question? What are high priority research goals and how do we design appropriate research studies to achieve those goals? And as people, hopefully, if this movement is successful, we will go from sort of a rate of participation and research that is very low to one that is very high, and that participation and research is a cultural activity that's very much like people go to museums and concerts and all sorts of the zoo, all sorts of things, and the same thing people should consider sort of participating in research as one of those things. And as, so I guess the point that I'm struggling with, and I wonder how much each of you, or if whether or not you've thought about this, is you can sort of imagine a future where they're on the one hand, there are like a couple big platforms where they provide all of the possible combinations of features and governance and whatever so that they can be applicable to the most people possible. And then on the other hand, there are sort of small communities of practice who have sort of developed their own styles of research that totally transparent, you can call into the IRB meeting, data is shared with each research participant or not and all the different combinations. And so in that sense, since we're sort of so early on this, everybody should anticipate failures of relationships that whatever you are building for people may not be ultimately what they want, but they don't know what they want. And so how much are you planning in advance for failure and the ability of people to change their mind and take their data and donate it to some other research enterprise, which actually meets their new expectations? Maybe it's something that needs to get on the agenda. I think one of the things that I've learned, I'm a clinician first and foremost, and I think one of the things I've learned from my software engineer and technology experts that I work with is that seeking perfection is probably folly, that we need to get to a point where we're constantly iterating and learning and we're aligning ourselves against some measure that we're reaching for and then we're evaluating how we've done to get to that point. So I think while we've certainly had some failures, there's no doubt about that in terms of how we've developed the taxonomy or the infrastructure within the company. Each one of those has been an iterative learning experience to say, well, we're not gonna do it that way again. Or gosh, some part of that actually worked pretty well, but maybe we need to think about a different approach the next time we try to deploy something similar within the technology. So I'm not exactly sure how to answer that question other than to say we have a hard enough time sometimes just as a company celebrating our successes. You know, we're really hard on ourselves. So I think that we don't oftentimes sort of think about ahead to say, okay, well, this could be a failure so we better prepare for that. I actually think that we have a more optimistic view and we go forward thinking we can accommodate and adapt as we find things that actually haven't worked but not necessarily call it a failure but a learning opportunity. And I think that's actually sort of embedded in the continuous learning system. Sort of really thinking about what did we learn from that even though it didn't work as we expected it to work, there was something that we actually got out of that. So I don't know if that's the answer to your question but I think we don't necessarily say, gosh, what will we do if this doesn't work? We're sort of thinking, how can we iterate this to the point where it's good enough for us to feel like we've got something really, you know, satisfactory for what we were trying to accomplish. Elizabeth. Yeah, I think even our responses tell you how complicated it is because we come from very different places. So you say citizen science is early and I think that indigenous people and people of African ancestry have been doing it for a minute for hundreds of years and so we have our own ways of thinking about citizen science and I think what's new is us coming together to try to figure it out together and that's exciting. But I think about, I agree that you have to adapt and that you can't have a cookie cutter approach. That the approach is gonna be different depending on whether you're an Indian country or whether you're in Brooklyn, New York or whether you're, you know, for us, it's easy. We're very localized, we're in one particular community. We know that community. The organization's been there for 50 years so we have deep roots and so those relationships are much easier. But trying to come up with a framework that provides guidelines for the entire country is impossible. It has to be something that lives, that changes depending on who you're approaching, what their motivation is, what their interests are, what the leadership looks like on the ground and that flexibility has to be built in to the approach. The thing that should motivate you is this desire, I think, if I could say that, to address something that needs to be addressed in a way that brings all these great thinkers together. We meet, I had mentioned we met with Rand and Lifeline's, we learn an enormous amount from them and they've learned an enormous amount from us and they've changed their approach and how they communicate with our community and we've changed our approach in terms of how we gather information and how we document it. It's been learning across the table and I think it's been a really positive experience but it's unique to us and it's unique to our community. Okay, we're getting to the end of our time here so I know you had your hand up for a really, really long time and so you might have the last word.  Okay, Leah Shanley and I co-chair the Federal Community of Practice for crowdsourcing and citizen science. So when I was a young grad student, I was sent up to work with the Menominee Nation for three years on a project and every single community member I met asked me three questions. Who are you? What are you getting out of it and what are you giving back? And there was, we talked about conflict of interests, corporations, first of what we jumped to but they perceived that I was gonna be making a lot of money as an academic in publishing their knowledge. So we have to be very sensitive to that. Also time scales, I don't think, we've talked a little bit about it, 50 years. So I'm an academic, I'm thinking three years, wow, I got a three-year grant, that's a long time and the community's like, drop in the bucket.  What do we do after you leave? Also the pace, the academic pace, we expect, you know, you gotta take exams, you gotta get your papers out and the community needs to build trust and get to know you before they make decisions about what they wanna share. So I think there's a disconnect between the academic and local communities. So I wanted to ask you, Elizabeth, we talked a lot about IRBs and institutional IRBs but what about community decision-making about what research they invite in and what research they don't? So the tribe I worked with had a board of elders, leaders that made that decisions about who they wanted to invite and how did they make a sustainable relationship with these researchers? Our organization doesn't work on any issue that the community doesn't bring to our attention. So all of our campaigns, all of our initiatives grew out of a request from the community. So for example, when we built the Climate Justice and Community Resiliency Center, that was in response to the community saying that after Sandy, they wanted to know more than changing the light bulb. So we started a block-to-block organizing effort on adaptation and resilience. So it's the same thing for the research. We basically have people come in and say, you know, we're really concerned. People went out during Sandy. They were engaged in recovery. We're just wondering, we have all these brownfields. What does that mean? Did we breathe something? Did we touch something that was unhealthy? And what we do is we play this role of facilitator. We facilitate community, meaningful community engagement and we bring together partners that make it possible for them to get answers to their questions and use that information. I don't know if that answers your question, but that's how we do it. We train our young people to facilitate community meetings. We think that young people are not at risk that they're at potential and that they need to be engaged in meaningful way early on in science. We start as young as third, fourth and fifth grade where our young people are learning how to organize and they're learning about environmental justice. And so we have our elders working really closely with our young people, but a lot of times the request, the demand is coming from our elders. Okay, I'm gonna let you finish it out, Dave. Thanks, Miller. I appreciate it. First of all, great panel, great moderator. Thanks guys. And I won't hold us up on lunch too long. Just maybe one thing that you all could think about thinking about if it registers.  The presentations you gave show that you're all sort of moderators and mediators between communities and research interests, corporate interests, you very much serve this mediating role. And I mean, I guess I've sort of trying to get back to what Jeff was asking. I mean, I would ask you guys to think about whether, as in these mediating roles, are you helping to cut down or nip in the bud potential ethical, legal and social issues? And if so, is that something that are obviously one size does not fit all, but do we need to consider having in citizen science, mediators go between professional mediators of some sort. And the flip side of that coin is as mediators, I think you guys can sit in tight positions where issues come up. And you also sit in positions of power and influence on both sides. And so maybe something else to think about is, as mediators are not, are there new ethical, legal, social issues that you all have to address or that being mediators bring up? So lunch goes from now until 1.30, you've got 45 minutes. The only, and the food upstairs is not bad. There's some decent sushi all the way in the back of that place. And the only thing I would ask, as we go around and ask questions and speak, maybe we can just quickly say our name so that, I mean, I think a lot of us know one another, but by the end of the day, maybe we'll pretty much all know who each other is, would be good. I'm Dave Kaufman, NHGRI, sorry.