My current job title is Lead Interaction Designer. I say that because I don't actually program a lot in my everyday life. Well, actually not at all currently. I have, however, been working very close with C and C++ developers and various flavors of web developers for almost 10 years now. The reason I'm telling you this is because I have actually proof that you can learn stuff by just listening to people talk about what they're doing and googling. I'm really good at googling. I have googled, I don't know, deep pointers, memory leaks, all sorts of things. And I'm going to conferences like this one. So I'm here today because I, as a designer, have been told for a very long time that I need to know how to program. I need to know the technology so I understand the medium I'm working in. And I think this is correct because it's really hard to do my job without knowing what actually is going on below what I'm drawing up in my wire frames. We do understand that. We as designers, and I think there's a lot of agreement also in this room that this is how it works or should work. And I know that you, like you, the developers, are excellent teammates and partners in crime when it comes to discussing functionality and my designs. So my wish, personally, is that you just knew a bit more about design. And we heard this morning, like borrowing UX principles for API design actually helps your work. And I would love everybody to come out from this presentation with a feeling that knowing the discipline of design, which is very much what you're working with, maybe side work with, would make our discussions that we should be having much more valuable and it would lead to much better outcomes. So I would like to tell you a story. Meet Marcek, or this is like a blurry version of Marcek. He's sitting on a fish in the middle of Oslo. We work together at CF Engine, which is a Norwegian startup building configuration management software. Very complicated, very complex beast. And Marcek was our database engineer. He was working in C and at the time we were working together, he was tasked with rewriting the whole database. This is a sketch, one of the many, many, many sketches I made. Illustrating one of our shared problems. So we were trying to make filtering on tens of thousands of hosts work for various people in various use cases. I was working on the UI. I don't need to go into the details here. But the database had to be working, or it had to serve multiple purposes for multiple users. So we couldn't just say like, oh, you make the database, so it works for me. It had to work for APIs. It had to work all over the place. So we had a lot of discussions. We had a lot of discussions on whiteboards. We had a lot of discussions over paper. We were banging our heads against the wall, solving really, really difficult problems. And one day Marcek came back to me and said, you know what? I found it. So he had been digging for, I don't know how long, and finally found a way that made it possible for me to design a sensible solution to this very difficult filtering problem. So effectively, he had been co-designing parts of the user interface. So how did that happen? Mainly, of course, because we talked to each other, why we were both working on it. But also because I explained what the problems were that I was trying to solve. So I took design a bit down to Earth by discussing my thoughts, asking for opinions, trying to make him understand what I was trying to do, showing very, very early sketches, like even earlier than this one, because this one is already pretty clean. I've been drawing boxes and arrows and things all over the place. And I demonstrated to the whole team of C developers that I was solving concrete problems and I was working at solving concrete problems. So discussion by discussion, everyone picked up bits and pieces of design principles and constraints and learned more about how and why I made those decisions. As a side effect, I was also exposed to a lot less bike shedding. So I wish I had that planned. I hadn't. I would like to start now with some myth-busting because I see a lot of misconceptions about what the designer is, what the designer does, what the designer, how a designer works. Because that is important to understand in a way because we're not in this box. I feel this kind of a mysterious animal to some people. So myth number one, one is born a designer. So the genius designer doesn't exist. I am totally not a genius designer. I suspect that that genius designer person exists just doesn't exist anymore than the genius developer. There are no unicorns. And all of the things I know, I have learned. I didn't even study design. I just studied opera singing. So it's perfectly possible to build knowledge just by sitting down and practicing and reading. So I spent many, many hours on learning, practicing, improving my skills, reading, asking other people about what they were doing, how they were doing it, looking at solutions that other people came up with. So design can totally be learned. There's plenty of resources available. I put some up. There's books, there's online resources. So there's actually no excuse to not look at them unless you think it's important, not important.  So the basics will do, just like the basics will do for me. I don't want to become a programmer. Another misconception is design is this creative chaos. We disappear into a magic bubble and reappear with a solution or something visionary.  As if no one could expect any kind of structured thoughts or works from me, it would just appear. So first, if you're not able to work in some sort of structured way, you will not get anything done. And design is work, just like any other kind of work. And I would also say that programming is equally creative because problem solving is a very creative process. You start looking at the problem, you explore options, and eventually you land on a solution. So this is my process. I have talked to a lot of people and they all have the same process. So the kind of design I do is probably closer to what your work is than any kind of digital or graphical design. So what I do is I drill into the details, I kill some darlings, I explore different options, I Google a lot still, I fledge out all the tiny details, I change some, I streamline the designs. I usually start from a very high level, I work my way down into the details and I rinse and repeat. From what I know about working, what developers or what you guys, the kind of work you do, this is very similar. And we iterate over it all the time. So if you're refactoring your classes, I'm probably refactoring my sketches. And for all I know, this is probably very similar for any kind of person who is solving problems. Another one, this is just plain wrong. Design is full of rules and principles and theories. And there are smart people who have written books about this, like Alan Cooper, the about phase that I showed before. He's like the father of interaction design, who's one of the, who wrote this about Facebook. And I don't think you should read that because they stick and it covers everything, but I did. But there's stuff out there that is more appropriate maybe for developers. So this is a very quick and easy read, it's very entertaining and it's very practical and it's just enough so that you understand the principles and that you can put to work right where you are right now. He also has a fabulous cheat sheet that is called the five fonts you'll ever need. If you stick to those, you'll be safe. Or you could take a look at the material design spec that Google has published. It's excellent, even if you don't do anything that is related to material design or Android applications or whatever. The way they explain the decisions they have made and how they came to this design spec is really, really useful. So you'd learn how animations lead a user through a process, why you can't make stuff fly in from the right after the user has clicked some place on the left, which should be sort of obvious. But the moment you start thinking about this, you see the world in a slightly different way. So why am I talking about all this? It's because I want us to be better at collaborating with each other. I believe we can all agree that designers should know more about technology, as I said before, to understand the medium. And there's a lot that still has to happen, but that's a different talk. But to be truly able to work together, all of us have to expand our skills on both sides of the spectrum so we can actually really meet in the middle and create an environment of mutual respect and understanding of our specific skills and trades. The first reason is obviously shared understanding. So if you think back to the sketch I showed you earlier, the sole purpose of this was to discuss a potential solution that I was working towards to solve our filter problem. And I didn't expect that to be implemented that way. I wanted Marcek to understand what I was trying to do. So he could actually solve parts of my problem or of the actual problem we were solving in the application because he understood what I was trying to do and how I was thinking the UI could behave. So that wouldn't have been possible if Marcek had never been actively involved in this whole process because then he would have been done with his database, had kind of handed it over to the rest of the team and then I would have worked my way. I mean, in reality, I look at the database and then I have to work in these constraints and sometimes it's really hard to solve the problems that a user has that is not looking directly at the database but who wants to interact with the data. So, with us working in parallel, we could inform our decisions based on what the other person was doing. Katie Kowalschen, I hope I pronounced that correctly, has had published a very interesting, a very thoughtful article on a list of parts where she states that involving engineers in all of their design reviews improve the team's overall understanding of the system, which obviously then led to much better results. So, she writes, the developers began to understand the thought process behind our design decisions and everyone could holistically understand the systems we were all building together. So, it's this kind of shared understanding that really puts value in our discussions and then leads to much, much better results because we know what we're thinking on both sides. Another one is shared vocabulary. While we're building this shared understanding of the system we're working in, we also build the shared vocabulary. So, an example from a recent conversation I had just some weeks ago, there was a developer saying to me, oh, but this doesn't work because I need a click event to run the query and update the UI. I suspect everybody in this room understands what that means. If I was standing at a design conference, I wouldn't be as certain. But how about we need to improve the glanceability of our navigation to increase the conversion rates for this particular scenario? Who understood what that was supposed to mean? I saw one hand, you win. So, just take a moment to imagine what those discussions and the decisions we're making could look like if you had some working knowledge of design while I have a working knowledge of engineering. Another reason is that we come to solutions much faster. So, we will achieve much better quality because we understand what we're working on in significantly less time because we work in a much smarter way. And that's Jeff Gotthoff. He is the author of Lean UX, which is a really interesting book. He argues for cutting deliverables and focusing on quick iterations. That's what he calls get out of the deliverables business. Let me just give a quick definition of what deliverables means here. It's like me working on wireframes, for example, that in addition to me sitting there and solving the problem, I have to document meticulously what are the different elements on the page, how are they supposed to behave, what are the ways from one screen to the next, how do I envision the functionality to come together so that the interface works as expected. This takes a lot of time. And it still is no guarantee for no misunderstandings or wrong conceptions or just different mental models on both sides. So, but to understand what this actually means, we need to look at how many of us, how many of us work today. So, we need to talk about process. Show of hands. Who is using some kind of agile process at work? A lot. Who of you who had your hands up in the air right now are working on large enterprise projects? Some. Keep your hands up. How many of you are working with designers as part of that team? Yes, hands go down. So, voilà, this still exists. This is a classic model which a lot of big companies still use today, especially if design agencies are involved. So it looks something like this. Analysis happens on the business side. Someone figures out what needs to be built for whom. Then the concept and design phase is taken by the design team. Or an agency. In a worst case scenario, the concept comes from an agency and then the internal design team has to make it work. And then there's the implementation phase which is obviously the engineering team's responsibility. We have handovers at every single side of this. Everybody focuses on the work of deliverables and the documentation which as I said before takes a lot of time to produce, a lot of time to read and digest. So, luckily a lot of people understand that this is not the ideal way of working. So more and more designers get better at understanding programming, engineering, and weasel their way towards the implementation phase. Some create functional prototypes or static websites as their deliverables rather than Photoshop files, which is already a good step forward. And we do that because we have understood that the classic model of handovers don't really cut it. There's a lot of misunderstandings and we're not happy of how it worked out because we care about what the end result is going to be just as much as I believe you care about your work. So, how about you weasel your way into the design phase? And I call that weaseling because you can't just say from today on I will be designing. You have to learn and then you have to make a change in the way, in the process you're currently working with. And I believe the weaseling way will make your life easier. So, I hear you.  You have a lot of things you have to do. Technology is constantly changing. It's a never-ending stream of things you could learn and improve them. Take new technology showing up that you should probably learn. So, do you really need to add yet another thing to the list? Well, you already are designing.  You define button states, transitions between streams, database schemas, APIs, JSON schemas, graceful degradation, all these kind of things that falls into the engineering bucket of the work have direct impact on how the functionality is and therefore is design work in a sense. And this is what Paul Boagra writes in an article on Smashing Magazine. The truth is that as a developer delves into a building a project, they will have to make decisions that affect and refine the design. Designers rarely have the time to consider all nuances of a website. The rest falls to the developer. So, the more you know about what we are trying to achieve, why we made certain design decisions in even the deliverables that you get, and more you know about the design process in general and the tools and constraints we are using to get to our results, the better all those design decisions are going to be. And this is not new, it's your reality. You are making all these decisions today. So, what about you start thinking about these details that you are deciding every day while you're implementing what someone else potentially has come up with? So, I would just like to leave you with this. Go forth, learn about design. You can learn on the job because you could ask to be included in design reviews and usability tests. You can read online magazines, such as A List Apart and Smashing Magazine. There's tons of books out there. I really recommend, for example, Steve Krug's Don't Make Me Think. He's like the usability guru. It's really nice to read and it's really entertaining. And there's so much in there that you probably do from a gut feeling that is now set on some sort of principle or framework so that you start understanding and seeing how these things work together. You could also start thinking about those decisions and be aware of them that you're already making and the effects they have. Talk to designers that you have on the team and ask them about this. And if you feel like really diving into the subject and get some more comfortable with how other people's work, take a close look at interfaces you like and figure out why they work and how someone potentially made them to work. A good exercise is to copy and interface and just try to find out how the elements fit together. And I'm saying copying not in the sense of make it look the same but make it work the same. So I hope we will know all leave this conference and start breaking down the silos. Thank you.


Good morning everyone, we're coming to you from sunny Cyprus and today we're talking to the partnership manager of Tick Milk, Mr. Alexi. So, today we're going to go through some information about becoming an IB and we're going to discuss some topics that may interest you and help you succeed. So, good morning Alexi, how are you? I'm doing very good, thank you and a good morning. Yeah, nice to have you here. Thank you. So, what I wanted to start off with is could you tell us a little bit about the ideal kinds of people that are going to become an IB? Ah, sure. What I can say is that from my experience any person can become an IB. There isn't a specific criteria that would justify a person's ideal or not. It's all about dedication, it's all about their effort and it's all about passion in what you do. And it's only a matter of how much effort you want to put into it to become a successor. Ah, okay, so it's not more about the kind of person, it's about the effort you put in. It's all about effort and it's all about your will to grow yourself in this business and there is a person trying to deliver really good product to their customers and it's all about doing something, taking an extra mile. Right, okay. So, we were talking about individuals, but what about businesses? So, what kind of businesses could be good as an introducing broker? Well, there is a belief that there are certain businesses that you'd better IB business. I would say that the businesses which are connected to the financial industry, the businesses where people interact with others, businesses where you come across others willing to invest, yes, they give you certain potential to grow your business faster. But overall, I can say that any business can be good enough to start an IB. When you were mentioning about finance businesses, what kind of businesses would those be just to give our viewers an idea? I've seen many people who would become successful IB who acted as, let's say financial advisors, they worked in banking business, they had some connections with investors and so on. However, I saw others who were much involved in all kinds of marketing and I believe this gives them even a better start as opposed to people who are just involved in, let's say, financial industry because our IB business is mostly about marketing. So, if you have these skills, if you have this knowledge, in my opinion you are more likely to succeed. Wow, okay. So, it's taking a lot of skills from a lot of different places? Yes, it did. Okay, cool. So, what characteristics does this successful IB require? In my opinion, if you really want to become successful IB, there are just a few really critical points and really critical factors that you have to consider. One of them is that you have to actually deliver a value to a person. We live in the 21st century and a lot of things happen faster and this industry is very competitive. Therefore, it's not enough just to invite a person or try to convince to follow you or join you. What's more important is to deliver something that would have a value in the eyes of the person you are talking to. Another one that I would like to mention is that you have to be delivering a really good product. We live in the times where audience became very mature and now you really want to have something worthy from you. Competition in this industry is really tough. That's why if you are delivering a product which fits and suits the standard and all expectations of the people, they will be very much delighted and they will stick to you, they will be lower out to you. At the same time, apart from having a good product, you have to really deliver good service. Because it's all about talking to people and people expect you to deliver good service which means that you have to be very responsive, you have to be a pro in what you do, your information has to be very useful and you have to speak the same language, etc. So it should be quite easy for them to communicate with you and to use your product. Another thing that I would like to mention is that you really need to speak to a good broker because as an IB you operate as a middleman between the broker and the audience. That's why you are very much exposed to the people you are referring to the company. And it's very critical to work with a good company because if something goes wrong, your clients will really blame you for referring them to the company which is not good. Just so we can clarify, the successful characteristics you are talking about are first of all providing a value-added service. Value-added service, yes. So there's something else that traders can gain from working with you. There's the extra content so they can actually read up about certain topics, improve themselves, education wise. And then also you mentioned about the broker itself. So obviously it has to be a good broker that is getting the right trading conditions. What do you feel an IB could do aside from those factors, or maybe a development of those factors that could increase the quality and quantity of the people that they're referring? So getting more people to agree to that. My personal research would say that you can succeed only in the event you aim to reach as many people as you can. Therefore, there are two really critical factors that play an important role here. The first one is the quality of the leads, the quality of the people you're approaching, and the second one is the quantity. Since we live in the times of social media, internet, the really fast communication, regardless of how many personal connections you have, in the event you try to reach over to people through internet, you can really expand your network much more other than when dealing with the people that you personally know. How many people can an average person know 100, 200, maybe a thousand? When we talk about our Facebook with over 2 billion visitors and 2 billion users, definitely you have much more potential. You have to have a network of approximately 2,000 people. This is the number when you feel yourself as confident that what you're doing is being paid off. Basically, it's your network because out of this active referral there will be people that would be actually doing what you're asking them to do. The others would be just basically following you but also it's a matter of word of mouth. They will be referring you to others and others will be referring you to others, etc. You have to be building kind of like a big fan club. So it's like exponential, so it keeps rolling like a snowball. Absolutely. That's point number one. Point number two is the quality. You may have really lots of people following you but only small fraction of them would become your clients but this fraction would be delivering you the profits. And having a skill to identify these people, having a skill to convert these people from being just followers to being the traders is really what matters a lot and segregate these people from the others, dedicate more time, deliver a better product to them and it will pay off since these people will actually be the ones generating eventual commissions to you, isn't it? You were talking about commissions, so leading on from there, what kind of commissions or structures are available to IBs now? There are many different options available. Let's talk about the ones that are more common on the market. The one which is used by majority of brokers and which is very well known to most of the IBs is a volume-based commission. This is the commission where the IB gets the benefits from the volume traded by their clients. There are maybe certain amounts of money paid for each lot or certain number of people paid for each lot and the concept here is that the higher the volume generated by clients, the more the IB is going to receive. There is another one where an IB gets compensated for the clients that became active with this broker. It's called CPA cost per activation or cost per acquisition. That's an option where an IB gets commissioned for referring a qualified person to the company, a qualified meaning that this person has to fulfill certain requirements. That's mostly used by the people who are capable to generate significant traffic from their website, for instance, or from their web page to the broker and the hope that at least a certain portion of this traffic will actually come in. Can I just ask those kinds of clients? Would that tend to be more people that are following the IB and maybe their trading strategy in the way that they work rather than individuals that are kind of autonomous so they have their own trading strategy? Maybe the different commission schemes are working together. Basically, it can be both. The whole constant between the two is that in the first case scenario, their IB is more likely to actually interact with their clients, support them, provide their feedback, etc. In the second case scenario, it exists mostly for those IBs who don't really talk to their clients at all. They only hope to generate loads of traffic and to have at least some of this traffic converted. That's the major difference between the two. What are the kind of mistakes that a lot of IBs are making? What are the ways that you would say that they could overcome them? That's a fair question because I tend to see there are quite many people who just started this business trying to find their way and build their business in an IB, but they all make the same mistake. The clients became more experienced and the expectations are quite high. Therefore, if you decide to proceed with being an IB, you have to stick to the idea that you should not be doing any direct hard sales. People would like to learn what kind of added value you are going to deliver to them. They will evaluate it. If they're happy with it, they will follow you. So having these hard sales, direct sales, would really scare them away and play the opposite role. At the same time, which comes along with what I've just mentioned, aggressive sales don't really work in this business any longer. You may have certain results, but if you think of this business as a long-term project, it will not work to what you benefit. Do be professional, do be delivering the value, and then the success is going to come. If you're aggressive, people will not really like it much. You briefly touched on it before when you were talking about the characteristics or the things that a successful IB would need, but could we go into a little bit more detail about what an IB should be looking for from their broker? I believe that nowadays the IB should be looking for a well-regulated broker. So it's not a guarantee that this broker will be the best broker in the world, but it's kind of an extra layer of security because nowadays it's very difficult to receive a license from such places as the UK, Australia, Singapore, or Europe in general, and this license has a certain... implies certain requirements toward the broker in the sense of having this broker follow quite strict rules and protect the interests of the investors, etc. It's good to stick to that broker. Another point is the reputation. Even though the reputation is not something material, you need to really collect as much information about your broker as possible, but have some healthy skepticism when reading about this or that company. Both too good or too bad comments about the broker should actually alert you because nothing in this world is perfect. Okay. So what do you think are key factors that are really, really important to building an online presence? You were talking a little bit earlier about when you were talking about the traffic and the way that people are faring through. In my opinion, if you decided to go into this IB business, you have to consider it as a long-term project. It's not the place where you can actually make your money fast. Our research shows that first three months are the most critical, the most difficult, the most stressful because this is the time when you're trying to position yourself on the market and this is the time when people just get to know who you are. That's when you cannot really expect much of returns in form of commission, but that's the time when you're actually building solid base for your future. So do consider it as something that's going to exist for a long time. And last point is marketing. Nowadays, marketing plays a very important role.  It will not work to your benefit. So the last thing I want to talk about is the models that IB is using. So we were talking about the products they were offering, the methods that they were capturing, but are there any specific models, so a fully comprehensive solution that IB could or would use? That's a very good question because some people think that in order to succeed in this business, you have to be offering something unique. In my opinion, it's the opposite. There are maybe 10 products which I use most often by IBs in order to build their business. We're talking about such things as signals. A lot of people are trying to find their way how to trade good. And they use the signals as a reference to either support their trading ideas or maybe find an alternative to their own thinking. Teaching and tutoring is also very much demanded because it's an enormous market and a lot of people join for its business every single day. A lot of them do really need a certain amount of training, certain amount of information, and they seek for professional advice. Training nowadays is very often automated. And in this industry, expert advices or all sorts of automated training solutions are very much demanded because, again, not everybody has time to learn or has time to trade, but there are plenty of people willing to have access to their softwares which allow them to profit from the currencies. Not all of them know how to code, not all of them know how about programming. Therefore, they're looking for their service providers capable to deliver them these pieces of software. From my observations, I would say that you should not be looking into something unique. You need to pay more attention to what you deliver, the quality of your product, even though it's something which is already offered by others and how you market it to your audience. Wow. Okay. So I think we've covered quite a lot today. Just a few takeaway points. So, Alexi, you feel that the best way to kind of progressives an IB is by adding value added services to work on the marketing. Correct. But you also mentioned that it doesn't have to be specific to one way and it also doesn't have to be unique as long as you're adding the proper value and you're paying real attention to the level of service that you're providing. Yes, that's correct. People became very much demanding. This industry is very competitive. Therefore, your clients expect you to deliver a certain level of service to them. You have to be supportive. You have to be quick in the sense of responding to them. And you have to be quite professional in knowing what you're doing. So if you're a coder and you know how to build automated trading solutions, you have to be quite confident in how to adjust it to changing the situation on the market. If you're a tutor, you need to be knowing certain things and be able to deliver the knowledge to the people. Yes. Thank you everyone. So this is all we've got time for today but we'll be back soon with a lot more about Tickwell. Thank you very much for joining us and have a great day.


Hello and welcome to the OpenStack Telemetry project overview and update for the Wallaby cycle. My name is Matthias Runge. If you don't know me, I'm a principal software engineer at Red Hat and I have been around with OpenStack since the Grizzly cycle. I've been recently elected as PTL for telemetry and I am the successor of Rong who is currently serving as PTL for the project Murano and for Solom. So what does telemetry do? Telemetry is used for gathering metrics and events. It is listening on the message bus and captures events like a VM was spawned, a network was created or a volume has been deleted. It is also using the OpenStack API to pull data out of the services to collect information about usage, memory consumption on a tenant basis, combined with the service AODH or A, and together with HEAT it provides an autoscaling service, for example to scale up or scale down resources. OpenStack Telemetry was probably founded during the Grizzly cycle and it started with only CELOMETER, which was later split into separate components during the Metacar cycle. Actually, I couldn't find accurate data about this and please correct me if I'm wrong here. Over the past cycle we had about 20 different contributors. While this number sounds pretty great and sane, I would love to encourage you all to contribute more because telemetry is an important part of OpenStack as we see from the latest survey numbers. Telemetry is used in 45% of production environments, is being tested in 8% of deployments and another 14% are considering to use it. Since there are a lot of different names around and under the telemetry umbrella, I have an architecture overview here where you can see CELOMETER is collecting data. Metrics are being sent to PANCO, metrics to NOKI and A is the alarming component. It is repeatedly pulling metrics from the NOKI API, compares the results with its rules and would issue a call to HEAT if an alarm is triggered. The most notable feature over the past cycle or past few cycles is the dynamic pulse subsystem. The idea here is to create or update pulse tests on the fly, which wasn't possible before. Pulse tests are being used to pull metrics out via the OpenStack API. For the future, we have two major changes in planning. You may have seen discussions around NOKI and NOKI being supported or not. NOKI being scalable or not, etc., etc. Personally, I would like to solve this rather sooner than later. There has been discussion around getting NOKI back under the OpenStack umbrella. To be honest, I'm not sure what that would solve compared to NOKI being independent. At least the discussion about that was uncontroversial without a clear outcome. Unfortunately, HEAT also stopped testing against CELOMETER and NOKI because it caused them to many issues. For the future, I would like to encourage you to contribute to this Leverie project. With that, I'm turning to my last slide. And I would like to hear your feedback, hear your pain points or see use cases. And finally, if you didn't participate in the survey yet, please consider to do so. Thank you.


So, to continue to expand on this idea, we're going to just continue to make other things. Well, any invoice actually kind of has a reference to any visit, if we remember that was kind of something I had said earlier. So I actually want my visit ID, my visit ID from my visit table, that guy right there. I want that to also appear inside of my invoice because, again, I need to reference it in some different way. We're going to keep it as a short text this time, but instead of this being considered my primary key, when we start to build our relationships of our database, this is actually considered something called a foreign key. And that foreign key is just saying that this is a unique identifier to another table. And I typically like to at least specify what my table is that I'm working with. I want to go down here to my properties section. You notice how for both of my invoice num in my visit ID, as I look at them, they both have pretty much the same properties. Well, let's say, for example, for invoice ID, I want to click on that. You see that it says down here field size 255. Well, what does that mean? Well, what that means is how many alpha numeric characters, letters, numbers, symbols I can have appear in this field, 255. It's a little big for what we need. If you think about our student ID numbers, those are only seven digits long. We seem to be doing just fine. Since we're dealing with an urgent care place, I don't expect them to be booming with business. There's tons of them. We're going to actually change this to just five. Now again, what this does is it says, it limits it to say only five alpha numeric characters can come in. We're going to keep on going and I'm going to actually say we also want to change my caption. Again, if we look at my field name, it's invoice num and there's no spacing and it's all one word. Well, again, that's good for the IT guy. It's good for the IT guy because once upon a time, databases couldn't handle spaces. It's just kind of a thing that we still do today just as a convention, but I'm not IT guy. I'm normal person who has to do this for homework assignment. What we can do is we can come in here and say invoice, invoice space number. That's perfectly fine. What will happen is this will actually be what we see in just a little bit. Just to continue expanding on this, now, visit ID. Well, visit ID, if we look at the visit table again, there are only four characters. Every one of my visits is only four characters long, so I don't need to make this five, but I do want it to be four. The same idea comes in here is that if we think about it, I don't want to give a visit ID. I don't want to type in six numbers because there is not going to be a visit ID of six numbers. There's only going to be a visit ID of four numbers. Keep on going. We're going to at least also change the caption of this to visit space ID, and I think we're good for those two. Now we're going to continue to expand on our fields. We're going to go ahead and create, again, we're dealing with billing, so I'm going to create something called invoice AMT. What's A to MT stand for? We probably have thought in billing, process of elimination, probably dealing with amount. We're going to be dealing with the amount of the invoice. Again, this time, instead of my data type being a short text, in that drop down menu, you see I get a slew of different options. One of those because we're dealing with money is money. You can see that automatically puts that into place. My description, I don't actually need to put anything in here because, well, it's amount, it's currency, but it doesn't have any kind of relationship to any other table. It's a unique element to this table. But I am going to change one thing. Instead of my decimal places, notice my properties have changed here. Here for short text data types, I get all this stuff. For currency, because it's a numerical value, it's a lot more strict. The only things I can really change are things like my decimal places. And instead of auto, I'm going to change this to just two. Two decimal places because sense. I will change my caption from just invoice AMT because I don't know what that is. I'm going to change it to invoice. Amount. That allows me a little bit more understanding of what this one is dealing with. We'll keep on going. Next one is going to be invoice date. All one word again because invoice date now says, well, when did we make the invoice? We again are going to go in and this time we're just going to select the date time data type. And again, you see we get a few different options for our caption. We're just going to say invoice date. And let's take a look at that format option. Right now, format doesn't do much. It's empty. But if we click on the drop down menu for it, you see we get a number of different options of how we want to present this data. Again, just like we learned in Microsoft Excel, dates are this weird hodge podge of numbers and symbols that aren't really alphanumeric short texts. They're their own kind of thing. So I'm just going to go ahead and say short date, month, day, year for the US kind of timestamp. We're going to continue on. And we're going to go ahead and add in one more option, invoice paid. Has this invoice been paid or not? Well, that sounds like a yes or no question, which sounds like another one of our data types, yes or no. And you can see, again, we have even smaller properties because we're dealing with something very strict. It can only be yes or no. So only things I can really change are maybe what it means to be yes or no, true or false, on and off, and my caption invoice paid. And once we've built this out, we do need to save this. We need to go ahead and click our little floppy diskette. We're going to go ahead and call this one. We'll go ahead and call it invoice table since we got a lot of invoice words here, invoice, invoice, invoice, invoice. And to change the view, to change how this looks, notice over here on our left, we have that table, that view, that option. If I just simply click on this, you see that it switches over from our table design view. Again, you can actually see the symbol changes as well. And I can click on it again, and it goes back to it. I can go back and forth from these. And so now I'm able to adjust these. Now, they're a little cut off, so I can resize these a bit. And what you'll see is all those captions we wrote, they now appear as well.


Okay, well, let's start. So let me welcome everyone welcome to this which is the first meeting of what we're referring to right now as the our submissions working group with the art consortium. And I'd like to start out by just a couple of concept comments on the various our consortium working group so as many of you know, we have had a working group for quite some time devoted to our validation the our validation hub. And we also have a working group with focused on making tables in our calling the RT RS working group. And now we have this working group, which the intention is to focus on infrastructure issues, it issues and the like that have to do with moving towards all our submissions. So some of you may wonder why so many working groups. Well loosely, what I have in my head is the model that the IEEE used in the early days of computer networking. So for instance, the Ethernet project was divided by where you were on the protocol stack, you know, the physical layer. The MAC layer, the media access control layer, the networking layer, etc, all the way up to the presentation layer. So loosely speaking, I think that's what we have here is a project where there are overlapping and not independent projects. But I'm hoping it makes it easier for people to not become overwhelmed and to get focused within the working groups by having a kind of like a layer on the stack where we're focused. So, so this is the infrastructure one. According to the, you know, analogy, we'd be down there at the bottom physical layer, the media access control, right. And for other things like the tables would be up at the presentation layer, how to, what happens at the certain levels of interface. So if this works, it'll be great. If it doesn't work, we'll try something else. Now, what I'd like to do, Paul, would you like to make a few comments on, you know, what the, what the environments like. Do you want me to show the slides that I got prepared? Sure. Okay, so let's see if I haven't done this with zoom. So this is the first for me. Okay. And can everyone confirm they can see the slides. Yes, we can. That's good. Okay, let me go to the slide show format. So I thought I'd give sort of a brief overview of some things. You may be familiar with this already. And if so, I apologize for going over other ground. So one part, if you're folks are doing a submission to FDA is there's what's called the electronic common technical document. And this sets out the overall framework that a submission would need to take. And it's fairly complicated. For those who have to look at it the first time, but there's, let me point out where some things are available. We are through the ECTD technical conformance guide, which is getting into some of the nitty gritty and then there's actually an FDA guidance on the topic. So this part in and of itself is actually handled by the Office of Business Informatics under the Office of Strategic Programs. And I think that there's something on the order of over 100 staff who are working with this type of aspect. You may be familiar with this, but let me go ahead and just mention that there's an overall aggregate file structure, the M1 through M5 format. So folders and naming conventions are supposed to be set up in a standard way. Actually all lowercase letters, which this doesn't quite follow, but administrative information prescribing is M1 summaries M2 quality is M3 non clinical M4. And what I usually work with is M5. So here's the study data resources. So the ECTD sets out sort of overall framework, then the study data actually sets out how actual clinical trial data and other sorts of studies should be submitted. There is a entire website for study resources. And we have the study data technical conformance guide. And this is the part that is worth looking at on a regular basis. It's usually updated twice a year. So their ongoing requirements or changes tend to be reflected in the technical conformance guide. And let's just see if we can click on that. Let's see. I'm sharing the application so you're probably not seeing this. So let me just skip that. That was probably, I don't think you guys can see that, can you. No, no, okay. Let me not confuse the issue then. Sorry about that. So we can make these. All public links in this particular one so I can share the slides with you later. CDISC is probably something that everyone is familiar with. It stands for the clinical data interchange standards consortium. And it comes in a variety of different forms. We have the study data tabulation model or SDTM. The idea of an analysis data model or Adam, and then there's the standard for exchange of non clinical data or send. So, typically the last bullet will be things for like animal data. Most of the statisticians and data scientists will be looking at SDTM and Adam. And most statisticians mostly deal with Adam directly in theory. The idea was that SDTM would be close to the raw data. Adam would be derived directly from SDTM. And at one stage, the idea was that it was one procedure or our functional way from the actual results. Although I don't think that particular model has worked. So this is the basic setup. See if I included any of the other technical aspects. SDTM has some different types of variables listed. There are the required things like the study ID domain use subject ID. Those that are expected, they should be present, but they could be missing. So for example, a death flag should be present. But if no one dies in the study, then there would be no point in including a death flag. And then there are things that go by the term permissible. We want the data if it's collected, but it may not be. So some of this would be things like ethnicity investigator name and so forth. Let's see timing variables. This may be something you're familiar with, but SDTM timing uses ISO 8601. And essentially, there's actually it's discussed in on Wikipedia. The, what it does is it's sort of a reverse. It starts with the highest level year, month, date, hour, minute, second, and then even I think it's hundreds of a second or it can go to thousands. Usually expressed in local time, but it can be in Greenwich meantime. The part about this that can be somewhat challenging is that essentially this is an interval. It's a point estimate. If you only know the day on which something occurred, you'll lose the last part the hours, minutes and seconds. So it could be anytime during the day. And most of our software programs treat that as a point estimate and give it a start time say at the beginning of the day. What that can do is create a problem is that an adverse event can be registered as ending before it actually started, for example. So that's something that can occur. So without going into the details, I thought it would be appropriate just to point out there's all of these references and resources that specify how things are to be submitted. Most people are familiar with the fact that XPT data sets are used. What may not be quite as familiar, for example, that's embedded in this study data technical conformance guide is the idea that data sets are supposed to be limited currently to five gig per data set. So there may need to be splitting of data sets, say for labs and questionnaire data. So those are some of the trickier parts of this. So there are are literally hundreds of pages of documentation that I've very glibly just put up links to. On that note, let me do a stop sharing. And I guess the next thing is to turn it back to you Joe and ask if there are questions and comments so far and details that you'd like to see further. Thank you. So, questions. All right, so I guess I would have one question. It took me a while to unmute but thank you for the summary answer that part of the infrastructure is not what do you expect to receive from the health authority site. How flexible do you think this is for negotiation or for improvement as we, if we were to move towards another mission. Because I particularly would like to see that what we submit is runnable code so we give you essentially a collection of packages that have all the data that the analysis code the data derivation codes. And for that our packages are great format. That's certainly something that can be negotiated. The one thing that's embedded in all of that are some of the naming conventions. So, for example, one downside right now is that the systems are set up that file names are supposed to be including the dot suffixes are all supposed to be lower case. Okay. So, the standard convention for our code is dot are with a uppercase are, whereas this would require a lower case are. And I think that also would make it impossible to submit in our package that gives you the documentation because they have like namespace has to be uppercase and description. So that might be something that we would have to negotiate. Well, if I could comment Paul and thanks for a great summary and hi everybody. It's great to see all of you again. How maybe take this a step further. To me, I think the real nugget that would be this is really pine the sky kind of thought is if we can pass not just a code and not just say the packages that we use to do the analysis and the submission, the actual analysis execution environment itself, then you have everything baked in and it's literally just running in like a container or some kind of, you know, our service, choose your adventure and cloud computing or whatever. But then there's like no ambiguity on like what's being transferred it's like the exact same run times or packages are versions as what we in the in the companies are using. We know that we've been pushing the envelope over and some initiatives, especially with CID. I wonder if that's something that we can take into this at some point. I know it'll take a lot to get there though. So the, that's been a mood in some other venues such as fuse is can we can say a sponsor just say, say they have an instance up on a in the cloud, say AWS or Azure and could they just give us access to that environment. I don't think that will quite meet with regulatory requirements because what happens if well there needs to be the ability to go back and look at the data sets and the environments, you know, basically work with the actual data at a future point in time in some So if there's basically a say serious adverse event that occurs in the future for a drug, there may be some desire to go back and look at the totality of the data. So I think in that sense, having a an environment that's just say there and then is say closed when the sponsor obtains per approval, definitely will not meet long term FDA requirements. Yeah, that makes a lot of sense. I was thinking it'd be something where and we even talk about this in the project you're not cooperating on the past. If we give you the instructions for the point that you all do that at your infrastructures women that's obviously another kind of runs of having your own infrastructure to support it if we give you like the recipes for it. Yeah, and if it's done the right way which, by the way you did. Anyway, we try. Well, we got something up and running. It may not have been optimal. Eric and I worked on a CID project. Eric did most of it. I just did a little bit of facilitation. Don't sell yourself short it was very helpful. Well, yeah, we're, there were still issues. So the we ended up having to kind of make up our own rule book and go around some rules which may have worked okay for a CID submission where we're a little bit more free leeway. Right, but probably would not work for a standard regulatory submission. Yeah, I realize it's a different mindset and it's a different set of rules mean not to sound harsh but it's definitely a more higher set of rules I would think in terms of what's been established already. And trying to get a docker environment and everything set up and passed into FDA through the submission gateway has been a challenge to put it mildly. Basically the containers I can stand up, for example, aren't allowed access to the local hard drive. So there's various sorts of other rules. In principle, I think docker containers and something along those lines make a lot of sense. I think that would be the way to go. It's trying to get things to work. A lot of things that are set up for example, to take advantage of Linux, tend not to work as well with our available machines because our IT folks, with the exception of some servers are almost exclusively windows based. Interesting. Okay. So trying to get those environments and different operating systems synced up has proven to be. Let's just say challenging is perhaps an understatement. Well, a quick follow up on that Paul. So is every reviewer's desktop environment potentially different in terms of our Yes. That's something that maybe we're some consideration in the future. That's a challenge. Okay, so it turns out I wear many hats. You just don't see them all. One of the hats I wear is, and I probably need to get out of it is actually manage a cluster or a group of workstations. So those I can control, and I can control the environment and the packages and how that set up. And actually what we did with and working with Eric is when it was specified we knew this are with this and actually used I think you use pack rat to get all the packages. So in that case, since I could control that environment I could set it up. We have had discussions of setting up what we are calling a statisticians desktop. But that's more of a concept that we're, you know, still in progress. So, in that case one could standardize it. If we have sort of like a VM everyone logged into and it had a standard suite of tools. But that's going to be a while away, just because just negotiating the licenses to be able to do that with our standard software products is going to be problematic. So we can do that. What that does suggest though is use of appropriate package managers, whether it be pack rat or RENV will be one of the major steps we need to consider as part of the submission process. Yeah, I agree with some of the things that we can control better first and then you know obviously the bigger items, I think we'll hopefully fall into place later on but package management. By far is one of the biggest things I try to champion around the projects I work with. So, so Paul, this is Sean with our studio I guess one of the questions I have. Maybe you could help shed some light on this. It seems like from our perspective as a software provider. We're talking about problems that have been widely solved outside of the realm of FDA submissions. So could you help me understand more like it sounds like you're sort of on an island with very little support from an organization that claims to like want to innovate and we'd love to help with that. So how do we go about like this group at the Arkansas shum, like approaching the FDA internal like it structures or whatever it might be that at some point are going to have to change, because like this conversation about dot lower case are it's almost absurd. So like I appreciate what you're saying that we have this, you know, that you've gone out and done these things with Eric. Well, what can you expand beyond kind of the island of influence that that you have access to. That's a good question. I wish I had the solution. Essentially, we have. I won't say we've set up a second it shadow network. But we almost have. We're calling it scientific computing. So one of the problems that as a government organization we encounter is there's sort of the business government type of side of things. And then there's this need for a computing infrastructure that will support all of the science that's there. And so that's why we've actually developed something called scientific computing. We have a scientific computing board. And that tends to be where we do a fair amount of work. Our high performance computing centers are actually outside of our office of information technology and management. And actually information management technology. So, you there, yeah, you're, you, you might say island or sort of parallel system that has been developed over time to actually facilitate and enable the science to be done. So what we can do is try to engage with some of the key partners. Say, with the submissions process itself to see if we can do some. We can overturn everything but I think having a pilot that would be an actual our submission and see if we can obtain some waivers without having to break the entire system. Yeah, would be a next step. Within work, we also try to build a pilot study that have part of analysis using R to submit to FDA in next year. And we summarize some of our understanding in a slide deck. Would you mind I go through that to share with the team here to see if that's a common understanding based on the current infrastructure we had. That's good to me. Let me share my screen. So, basically, what we're trying to do probably as a pilot in next year is to submit submit one analysis using our based on the SDTM and Adam data that already being created by sauce. So we need to focus more focus on the analysis side to generate table listing and figure. And always go with more focus on the primary or secondary ethics analysis, because as a typical leader part that require a sponsor team to submit to the code to FDA. This is in line with I think the document just mentioned for the FDA study data technical conformance guide. And those part highlighted is more of related to analysis side, because from the guidance I think the minimal requirements for primary and secondary ethics analysis. And we also need to mention what software to be using over a DRG. And we also understand that the code itself based on current FDA guidance is not to say it's actually executable from FDA reviewer. It's more like a source to help reviewer to understand what's a key variable and algorithm we use to perform the analysis. Actually as part of the committee that wrote this. So, yes, the idea was that .exe for example is will be rejected outright. So somebody writes a specific executable code. It will not be allowed to be submitted. And part of that's just essentially the guard against viruses worms Trojan horses. The. So, other parts. Yeah, so essentially it's supposed to be an ASCII text which is what the typical our data set. Excuse me, not our data set but our program is submitted. So we should restrict the code return all in the ASCII text. So those special correct should be avoid. Correct. And that's partially where things are with their. There is a before we even get the programs and data into the past the gateway and into electronic document room there is a staging area that actually will scan everything and ensure that we're not being hacked or corrupted data files are being submitted. I see. And so that's how we've tried to build up over tentative solution for the pilot study. And this is actually the action item we trying to proceed. So internally we perform over analysis in our markdown document. So what we're trying to do is to extract the code from our markdown and put into a TXT file. And we also do certain components check to ensure the code only content as a factor. And also we will have a read me file to summarize the running environment we had. ADRG or ARM. So we will provide step by step instruction to select what's our version and K package and it's our version that to be used to rerun the analysis. And another part to ensure if a reviewer really need to run the analysis, we try to give a public available our studio package manager pass that have the same snapshot as we had to run the analysis. So, if a user for them being a Windows computer they want to run the same environment they can install from this repo to get the same version they need. So that's one way we think that's probably lower the environment running environment to reproduce the results. I'm not sure that we have the commercial version of the package manager widely available yet. So this way, we don't need a commercial one install. So this is the community addition. Okay. Right. This is a public available link, as long as, yeah, so you can link outside of the internet, they should be able to download the package. Okay. That should work. And we're actually attempting to do something along those lines internally. Okay. Well, there a way that you're aware of that we would be able to deliver the exact set of source code of the packages that we use along with the submission, instead of relying on a public package manager. The short answer is I don't know. We could that might be something we can explore. So you're basically saying you would have all of the Are you talking about using like a say a package manager to have a all the packages. Compile together and then What exactly did you have in mind open for discussion because in any validated instance that you Are putting together you're going to need to have snapshots of your packages available on your your local system somewhere and like our studios package manager project or product is one solution to that. But the gist is that at any point we're able to recreate our environment ourselves with whatever qualified environment that we're using for those submission has been set aside. So it sounds like a big piece of this would be understanding the best way that we can give you those components so that you are able to reconstruct it yourself. And I'm hearing the limitations that you have there. I'm kind of having a public access point for this is one one piece, but I mean, with organizations are going to have their own individual packages that they're going to need to hand to some funds well so there's different pieces of that puzzle. If they're, if it's a internally developed package that say not on cram, or say widely available via GitHub. I think there would be some concern about validating that package. So let's just assume for a second that every package that we're using has undergone that validation process, and let's say that we can provide documentation or supporting pieces for that. One potentially could submit all of this as part of a submission. I'm not quite sure how. What type of validation will be considered sufficient. I think that's still something one would have to consider and negotiate. So, I guess when one area that we have done, for example, and maybe even can comment on this is, I know GS design has been externally validated. And actually is part of the Moderna SAP back submission that is actually going up for an EUA. Yeah, we are one of the Merck alumnus, I guess is statistician at Moderna. And I was very pleased to see that that was there. We've also been going through some of the testing and we should have by the end of the year so that code coverage for the package is over 80%. You know, for any new package we do, we're doing it according to our system development life cycle which includes specification and validation. So we're prepared for an audit of any package. I think that we would submit for any critical analysis. If that's helpful. And also, you know, do you want to mention the potential way of submitting a package, according to the current. Yes. So, while we discussed the question Mike mentioned to submit internal package we developed to FDA. And we're trying to say select to zip the package into a TXT file. So the general idea is that we leverage the key component of source coding our package. And then use a special correct to separate each file and put into one TXT file. And then we can unzip that into the original our package if statistician from FDA run a code from a function. So you can see one example here. So this MK serve is one of the package. We try to develop internally to for some survival analysis method. And we put all source code into the TXT file and using another function to unzip into original package for the structure. So, instead of to having a FDA statistician to go through a lot of files in our package. So that can be combined or in a one TXT file structure. So as long as the idea to put that conversion function into a package that could be available on cram. Right, so we have a colleague currently developing this kind of feature and trying to public that package. Definitely in a validated way, and then FDA statistician once install that package we can provide instruction to on the former TXT to a package that can be loaded into the the Windows machine from FDA side. So, well that sounds interesting. I think it is a, yeah, it is a complicated. Well, I was thinking that Sean would probably say that's kind of a complicated work around to get through this. The simple part is you don't have to go through the FDA bureaucracy. So, sometimes the technical solution that avoids having to tangle with the bureaucracy is, it looks like it's the long way around but it's actually the shortest path. Yeah, I can definitely see that. I can see that but at the same time as like a enthusiast of all these enabling technologies that I get to use every day at my job with obviously containers and HPC and cloud deployments. The technical side of me just admittedly feels kind of sad to hear all that you have to jump to just to get there. There's more term solutions and long term solutions. Right, right. Progress on the interim, you know, towards more sophisticated methods in the future. Also, our packages and the structure of our packages historically have been proven to be quite beneficial. I mean, the FDA probably is also interested in like having, you know, all the features that come with it, having the help files and the other things. So, this is true but the, we have our own, well, external folks have issues with our bureaucracy, we have our own internal issues with our bureaucracy. So the gateway, the office that manages the gateway is actually separated from the actual end users, and that sometimes can lead to issues and problems. I mean, we're still using XP T files, which is a 20 year old standard. So, and I've seen two attempts to replace those with XML, but it still hasn't taken place. So kudos to our colleagues for their ingenuity with some of their workarounds but I would agree that it would be good if we can revisit some things and see if we can work with folks but it's probably going to just from what I know of how their gateway systems works. It's probably going to take a while to a build up a justification for why it needs to change and then actually be implementing an actual change. So, if I think that I realized from the outside that that looks kind of idiotic at times, but yeah, Paul, another thought is that it takes a lot of change. If you need any encouragement from, you know, say pharma, they are active in this area with with transcelerate. So, if you need advocacy from the group like that, in addition to say this, you know, we can certainly work on that. That would be helpful. I know that the current OIMT head Amy Abernathy has worked with actual clinical trial data and understand some of the issues that are involved with her experience with flat iron. So I think trying to say, okay, let's actually sit down and figure out how to do a better process. One of the issues is that nobody would do things the way we do them today. If we had to design it today. A lot of it is left over from various legacy systems, starting from well over 30 years ago, and then basically things, a creed knowledge, and other things rather than basically being redesigned from scratch. So at some stage it would be probably helpful to consider how to update and modernize the submission gateway. But you know that that's easy for me to say because I'm not a part of the office that handles that. So, it's kind of like saying to your neighbor she would be nice if you could paint your house. So Paul is that office that owns the gateway is it really one office or is it really like another some group that starts with security that they defer to and another group that you know has various other responsibilities. The gateway is handled by the Office of Business Informatics, which is actually under the Office of Strategic Programs. And the group that actually does the gateway is mostly contractors. So the experience and working with contractors in a federal setting. They're not necessarily interested in improving the process. For say they're interested in fulfilling the terms of their contracts so they get paid. And I say it's the easiest step to make a small change like how do we get to uppercase fine names. I think that's the first. Very, very, very nice suggestion, maybe. But I'm sure that the our consultant has some routines in place to make sure that the submission that come to the current servers are legit, and not some malware or something. Maybe there's some possibility to benefit from the procedures. I mean, I can see that, you know, you need some checks to not being getting stuck and stuff. And like trying to renew things is very difficult. I hate to say it, but the simplest way. Well, okay, so part of it is I don't. Well, it sounds good from outside if a government organization could borrow from the foundation, such as they are foundation. The likelihood of that being carried out is close to probability zero. There would have to be a contract and have to be led there's a specifications. It's a government bureaucratic nightmare. I can think of one way. And you, you're not going to like this, but it's the work around system is to submit everything in lower case, and then you write a script that then takes all those lower case things and attaches uppercase and then it lets you work with your project. So then you're back to a long solution. So that is, believe it or not, that's probably the easiest step. I had a kind of another question for you, which goes all the way back to some of what Eric was saying at the beginning, which is that in some of these pilots. The FDA reviewer has kind of come to the pharma environment, as opposed to the farm was environment coming to the FDA. And you express some concern there that you know, a sponsor may need to keep an environment around you know indefinitely. I understand I mean that that actually seems very feasible compared to some of these other things we're talking about a shelving a Docker container or an AMI or whatever. It's certainly easier than writing a script that messes with the case of my file names. Is there any like appetite there. I guess, in the FDA for that type of model where a sponsors environment becomes the source of truth or are we really in a world where it has to kind of be thrown over a wall into the FDA. An interesting question. My own concern is that FDA should always be able to look at the data in its environment. And part of it is, I may trust a company's statisticians, I may trust the IT folks. I'll tell you what I don't really typically trust the marketing folks. So, I could see where at some point in the future, marketing folks might get control over some of the environments at least partially and say slow down things as a way of thwarting investigation.  Right. And then, you know, I could also see bad players within a company, even saying, Okay, we can, we can basically track. If we control the environment we can track everything that FDA is looking at in this. And we can anticipate. Essentially, you can see the cards that are being dealt to the FDA. And we can anticipate how to game that. I'm not saying that your companies are necessarily ones that would do so. But if that power exists within a company, I am not sure I trust everybody in all commercial enterprises to behave in an ethical manner. So can I ask you, I've been, I've been told by a senior person or senior in age that the pharma companies used to give actual PCs like computing physical boxes to the health authorities and they at some point said, you have to stop it. You have to be dealing halls with your computer. But it seems like almost the easiest ways to give you a PC without the network card or laptop nuts. Essentially on your own term. Yeah, our environment at one time because Steve Wilson probably has reminded me that back in the 80s. This was done that and or government procurement and it was so lagging that an environment was physically given to enable this but at the same time there was still the paper submission. Which go back to that too. Yeah, if you've seen the paper submissions. There's like truckloads of them I've seen pictures. And I can tell you I've seen, I've dealt with a submission that I looked at. I may have actually once hit print accidentally and I realized that there were 10,000 pages in this submission. And then I had to hurriedly go cancel the print job. So, I think part of it is, let's identify what needs to change. It might be something that can even be advanced through various channels. Transcelerate is one even Padufa is a possibility. If company say when there's time at the Padufa negotiations that we would like FDA to modernize its gateway. I know what FDA negotiators will say is that, okay, we'll FD. Let's negotiate what that will cost. And how that will proceed. Most likely won't be prohibitive. True. Yes, I would guess so too, but those are some of the aspects that occur. So industry has its has some leverage as well as FDA. Well, I mean, especially if you don't follow the accretion model you mentioned Paul. But I don't know. Maybe I'm wrong. So we're running out of time here. This, this has been, I think, a fascinating and help productive discussion. Some administrative things. I will work. I'm a little bit behind, but I will work to get a GitHub repo setup. I think that will include the video recording and minutes from the meeting. And please if you long and Paul, if you could send me your slides, I will make them available to in the repo. And then we should decide on the next meeting. And maybe the cadence of meetings. You would like. If this time slot or perhaps a little bit earlier would be, would be better. So how about some time in January is the week of maybe the second week in January is that a good time to meet next to the week of the Monday the 11th. Friday the 15th, nine or 10 a.m. Pacific time. Anybody have any. Looks fine for me. Works for me. Yep. Yeah, I'm fine. 915 9am Pacific. And I'll make sure everybody gets an invitation. And we can, once the GitHub setup, we can use the issues and, you know, and whatever to continue the discussion in between time. So, thank you, everyone. And let me know if I can do anything further to, you know, facilitate communication. Thanks for setting this up. Thank you. Thank you. Thank you, John. Take care. Goodbye. Bye.


Hi, I'm Jackie Arnold, I'm the Strategic Relationships Manager at Sailor Academy. I'm here to talk about skill gaps and why we're focusing on them at the summit. A skill gap is a gap between the skills employers need and the skills that are available in the marketplace. Skills gaps can manifest themselves in many different ways and have many different causes. For example, non-native speakers may find English to be a barrier between them and living wage jobs. Hard adult learners that take a break from college may find it difficult to achieve career advancement and personal fulfillment because they don't have a degree. And changes in industry can cause the state to bring private sector as well as higher education and workforce development in the same room to bring hard workers and jobs together. We're proud to show you how three organizations developed models to close these skill gaps and bring opportunities to the communities they serve. If you want to learn more, click the link here and register for the summit on the credentialing economy June 7th and 8th in Washington, B.C. See you there.


Suspense! In America's most discriminating homes and clubs, where dining and entertaining is the last word in luxury, the first word in wines is C-R-E-S-T-A, B-L-A-N-C-A, Cresta, Blanca, Cresta Blanca. Yes, from the finest of the vines come Cresta Blanca wines, created to please the knowing tongue. Enrich your dining, add luster to your entertaining with Cresta Blanca. There's a magnificent Cresta Blanca wine for every occasion, for every taste. Schenley's Cresta Blanca wine company, Livermore, California. And now, Schenley brings you Radio's Outstanding Theatre of Thrills. Suspense! Presented by Roma Wines. That's R-O-M-A. Roma Wines for your everyday enjoyment. Tonight, Roma Wines of Fresno, California, present Mr. Lloyd Nolan in Double Ugly, a suspense play produced, edited and directed for Schenley by William Spear. His name's Eberman. Matthew Eberman, his name was... He had his name in the papers quite a lot, Matthew did. There's kind of a funny story concerning Matthew, too. Had to do with murder.  Ugly is a mud-fence, and he hated that.  Or ever since he was a kid. I won't play if I have to be Matthew's partner.  Well, I won't play, and I'll tell him so to his face. Matthew Eberman, you're the ugliest, most horrible-looking boy I've ever seen. You're Double Ugly. That's what you are. Look at you. Look at you. Sometimes, Matthew would cry. Hit your home and beat his big fists into his pillow and nearly choke, getting the sobs out of them. Don't cry. What do you care what they say? Friends don't matter unless they love you for yourself.  Mother thinks you're a wonderful boy. Well, then, how don't you cry? And then Matthew wouldn't cry. It was a funny thing, too. He came from a good family, enough money, education. And if he hadn't been so ugly, if he hadn't been, his voice would have been kind of pleasant to listen to. One time, I remember, he got alone OK with just his voice. It was at a masquerade. After he got the costume and the mask all on, his mother said, why, I hardly know my old boy. All dressed up like that. Why, ever when, I don't know. Wanted to be with people, I guess, anyway. He did go to the party and he danced with all the girls in the place. And one girl was so beautiful. Some lovely duchess, maybe from a hundred or two hundred years ago. Like a soft cloud of ruffles and lace and the scent of magic clung all about her. Who are you? Someone who's in love with you. A masquerade is so romantic. It's perfectly all right to fall in love with a stranger. Someone you haven't the slightest idea who it is. What do you mind if I fall in love with you? It's a masquerade. I've never said it before. I love you. Who are you? I love you. Who are you? I'll tell you who I am if you'll tell me who you are. Here, I'll take mine off first. No. Well, I love you. No. Oh, you silly. I'll take it off of you myself. No, no, don't. Please, don't. I will. I will. Matthew Everman. I'm a lovely duchess. A soft cloud of lace and the scent of magic all about her. Matthew's mother died. Matthew went away from town. It was after that that he had to do with murder. For Suspense, Roma Wines are bringing you Mr. Lloyd Nolan in Double Ugly. Roma Wines' presentation tonight in Radio's Outstanding Theatre of Thrills, Suspense. Suspense, Radio's Outstanding Theatre of Thrills, presented for your pleasure by Roma. That's R-O-M-A. Roma wine selected for better taste from the world's greatest reserves of fine wines. If you're like most Americans, you're probably planning to do something special over the Labor Day weekend. Maybe a picnic for having friends and for dinner. So, here's a suggestion for making weekend hospitality really festive. Whether you're dining indoors or out, enjoy better tasting Roma California wines. Serve glorious golden amber Roma sherry, rich in nut-like mellow taste goodness. Yes, you and your friends will enjoy Roma sherry served chilled, a friendly wine, the perfect first call to dinner. And to top off a good meal, or later in the evening, serve red Roma port or golden Roma muscatel. So, this weekend for gracious hospitality, for friendly entertaining at low cost, delight your guests with better tasting Roma wines. That's R-O-M-A. Roma Wines, America's largest selling wines. And now Roma Wines bring back to our Hollywood soundstage Mr. Lloyd Nolan in Double Ugly, a play well calculated to keep you in suspense. I just told you about him as a boy, so maybe you could understand him later on. You see, he never knew a soul to talk to except some of the men who worked in the same plant where he did. He got a job there in this plant throwing sacks into a truck. At lunchtime, Matthew would listen to the other men talking, his big hands opening and closing regularly. You know what I think? I think it's better to be married no matter what. That's right, Hanson. Still, lots of men don't get married. That's right.  No. Oh? How come, everman? I don't know. I thought I'd never get married. I thought women married only good-looking men. If a woman had married me, one ought to marry you. I never know when he went to marry. Well, my boy and Shane will fix that. I tell you what, come up to the house Saturday night. Everman's the one I was telling you about that don't know no one to marry, Hazel. Well, I think the woman's touch and vice versa. Yeah, that's the truth, everman. Sure. Now look at Hanson here, Mr. Everman. He wasn't no use to himself at all, all alone. And now we got a really good life worked out. Oh, now Hazel, honey, don't force. Oh, that's all right. Let her talk. The fat wife talked and Matthew listened. And he began to wonder if maybe there wasn't something in what she said. It was, Lonson. He could quit the plan if he wanted. He had all that money his mother left him when he died. He only worked because it was so, Lonson. And then Hazel said, Now what do you go for? A big, easy-going blonde. Like a soft cloud of ruffles and lace in the scent of mad. No, no, that's getting it all mixed up. Anyway, Hazel arranged for them to meet at a party at her house. She had all the men and their wives from the plant and a guy named Lefty played the piano. Played piano like old-time movies. Everybody had a swell time except Matthew. He sat alone over in one corner of the porch. He heard the screen door open and close and a big bunch of white that was sally moved over to her. Oh, there you are. You mean by running away from the party? I'm your date tonight, aren't I? Sure, but I thought maybe you'd want to be with some of those fellas that talk more. You know, I'm pretty quiet. Oh, I don't mind. I think that's the mark of a gentleman. Good night, huh? I think nature's just about the grandest thing in the world, don't you? Trees and flowers and all? Ain't so many flowers around here, though. I like sunshine and clean scenery all around, don't you? Lonesome here, lonesome for a girl. That is the courses if you aren't married. I get lonesome, too. You do? I never thought men got lonesome. I thought it was only girls. Men can get out. I get lonesome. Oh, sure. Sure, I remember now. Hazel told me.  What are you laughing at? I do get lonesome and that's the reason only you're not to laugh. Oh, my goodness, I wasn't laughing. Except it's funny to think of a man then, so say I didn't mean to hurt your feelings. I'm lonesome, too, ain't I? You got the biggest hands I ever seen. Do you think I'm pretty, do you? Oh, I get so lonesome. Matthew couldn't get it out of his head about this funny girl. Do you think I'm pretty? He didn't even know. He couldn't even remember what she looked like when he went to work the next day. And when he started to get his lunch, there was something over by the gate that he'd never noticed before. Something pink and fresh. His big hands opened and closed. Hi, Mr. Everman. It was Sally, all in pink organry, like a pink balloon at a children's party. I guess you're surprised to see me, aren't you, Mr. Everman? Yeah, I guess I am. Maybe you might like a little home-fixed lunch. Doesn't it look good, huh? Well, we can eat over here in the shade if you don't mind having a lady as company. I was all alone today. That's why I come. Looky here. I even got some chocolate creams for after. You like them? I do. I do myself. Ain't this a cute bracelet? Little tiny bells. Must be hard work to make a thing like that. Now, don't you think? Oh, say, have one of those sandwiches, Mr. Everman. They're gorgeous! Sally chatted away all in pink and self-consciously turned to wrist to make the little bells tingle. Matthew ate steadily, silently, big, thick sandwiches. This is like a party, ain't it? After a while, it was time to go back to work. Matthew stood up and didn't know what to say. Just looked. His face was kind of shiny from the heat and the pink had gone limp and bunched. But a breeze picked up the loose strands of her hair. The foot twisted like a little girl. You know, you never once called me by name. It's Sally. I know. Aw, call me by name. I like being called by name. I will. Can't you do it now? Aw, it won't hurt now. I gotta go back to work. Okay. So long. So long. You look pretty in pink. Why, say I'm awful glad you like it. It's one of my favorite colors. Why did the client watch the romance develop? It's better to get married no matter what. No matter what. Matthew never asked Sally to marry him until the day he saw her out by the fence where she lived talking to Lefty, the man who played the piano at Handsome's party the first night they met. He saw Lefty pinch her cheek and he heard Sally laugh.  Look what you've done to Lefty. Why, you knocked him out cold.  I don't like the woman I'm going to marry fooling around with guys like that. Marry? Why, that's the one thing in the whole wide world I always wanted to be. Matthew took his bride to Niagara Falls. There was only one thing, the bracelet. Matthew wondered what he could do about that. The little house was a white cottage located pretty much by itself quite a distance from the plant. Sally had to walk a good way to see any of the people she used to know. How's that? The grandest thing, you should come by and see it, pay us a social call.  Yeah, flowers and a tree. You simply got to come and see it. And the men and their wives at the plant did come and see it. They sat around the house and put their feet on things and spilled cigar ashes on the floor and talked loud and common. Matthew didn't say a word. He just sat, listened, looked at the mess they made. They stayed a long time. Matthew thought they never would go.  Don't worry. Yeah, we'll be back. We're all the now that we know the way. Good night. Good night. Good night. Matthew looked at the rugs and the foot marks on the chairs. The place looked ugly. Imagine entertaining folks in your very own home. They won't come again. Why not? Look at the rug. Oh, that ain't nothing. I just run the sweep. We're over it. They're not coming again. But a house ain't nothing without folks coming in. We got each other. Oh, yes, but... They never had another party. Maybe Matthew saw as a man Maybe Matthew saw his own face and the ashes on the floor. The house grew quiet and Sally got nervous. Oh, it's just that we sit here and sit here and don't say a word, Matthew. I get lonesome. Can't we have a party? Huh? Oh, can't we go to other people's parties then? Oh, can't we do that? And the yard grew up in weeds and the flowers didn't do so well and the tree didn't grow. Even the smoke from the plant seemed to find its way over to the house. The white got gray and it got black in splotches and the paint peeled. Matthew lived. Sally put on weight. I got to match my hair. Lady at the dime store said blond hair got awful dingy looking so I got this bottle of stuff. Only a dime. Look how brightens up the color of my hair. Oh, have a chocolate cream, Matthew. These creams come from the dime store too. I can't even notice the difference from the ones you used to buy me. He walked away. He always walked away. He wanted to tell Sally how much he wanted to be close to her, but he couldn't. She filled the house with frilly things and all the fussy doodads that go with it. Well, you just won't believe what I got to show you, Matthew. It's just like Joanne Crawford had in the magazine. Look, it's a genuine toy poodle.  Oh, love beautiful.  My baby, would you like a little chocolate cream like the mummy's? Oh, Matthew.  Cringes.  Would you like them? Don't you hear? No. Have a little. I didn't taste the seeds. Whoa. No, no. No, no, no. No, no, no. No, no. No, no, no, no. No, no, no. No, no, no, no, no.  I just don't understand those birds, Matthew. They don't sing a song like canaries is supposed to. No, no, no, no, no, no, no, no! Now you not be bad boy, scaring your guilty sisters like that. I just won't talk to you again. Don't do that funny face in the muddy. No mommy didn't mean to hurt the feelings. Oh, Matthew wouldn't it a scream. Here little angel, here little free angel. Let's divide it, a chocolate cream. Oh, no, no, no, no, no, no. You must be all up for it. Oh, look at that, Matthew. All down my neck, you see. Here now.   What are you crying for, Sally? What is your name? Oh, look at him.   He's just like a kid. No. Matthew got up, went and stood by the window and looked out. Lonesome in your own home. Why go to those awful parties with ashes and foot marks on the chairs and the men with their arms around Sally? She shouldn't have allowed that. She belonged to him. She was the only person who ever did belong to him. Hadn't she come to him in the plant yard that day? Sure she had. She liked him. She shouldn't be singing that song that left the place. That left the pinch in her cheek and laughing at him. She shouldn't have allowed that. Oh, mom, did you get the angel hat? You wanted her to leave that dog alone. Why couldn't she treat him like that? Matthew looked out the window. It's too bad about the paint and the smoke that came over from so far from the plant and the weeds all around. He was surprised to see the tree look bigger. Maybe he hadn't noticed the tree for a long time. What are you doing, Matthew? Thinking. Oh. He looked at her. Where was that pink and fresh thing he'd seen before at the plant? The face shining from the heat and the loose strands of hair caught in the breeze.  These creams come from the dying store. You can't even tell the difference. Where was that pink and fresh thing by the gate? Sally was standing by the canary cage and wiggling her wrist again. She hadn't done that for a long time since Niagara Falls. Oh, Matthew, I found that bracelet with the bells on it. I thought it might make the canaries sing. Come on, birdie do. Do you think they're sick, Matthew? I've never seen them so quiet before. Listen to the bells, gritty babies. Here, here. Matthew's hands were opening and closing at his sides. Like they were grasping for something to hold and never finding it. Just don't understand those birds at all. You've gotten fat, Sally. Ah, don't matter. I don't care. Nobody cares if a married woman gets fat. That's what Hazel says. I like Hazel. She's a good company. I don't get down her way much now, though. I ankle swell up from these French heels. Cute shoes, though, ain't they, Matthew? It makes the foot look small. I ankle swell up, but it does make the foot look small. Her wrist swell up, too. I don't know what's wrong with them either. Oh, I wish I could get these birds to sing. Aren't you happy with me, Sally? Yeah, but we never do anything.  I just get so I can't stay still another minute or I'll scream. If we had some kids, then maybe we'd mean more to each other, have something to talk to each other about, and kids is awful, good company, and then conversation about kids is good company, good talk between a man and his wife. If you want to run, you want to kick the walls, tear the house down, anything. La, la, la, la, la, la, la, la, la, la, la, la, la. Mommy's little baby now. Where was that terrace? But it's seen once upon a time. Some lovely duchess, maybe from a hundred or two hundred years ago. Maybe if he grabbed Sally, held her so close, that would make a difference. Why did it have to turn out this way? They said that a man ought to be married no matter what. He'd only stop all that noise if he'd only just get a minute to think without all that noise. Then he could get his alley with so fat, Matthew stuffed his mouth full of dime-store chocolates, one after the other, one after the other, one after the other. The whole room began to turn. No trees, no flowers. The canaries and the dog were quiet, and was still as could be. Except the wrists. Her face hated him. She said he was ugly. Ugly! Can't you even say that you're sorry? Stopping up boys, sure know what to do with their hammers, don't they? Listen, I guess they're getting ready for end of the morning, cutting down a tree to build a thing like that. Well, that's the story of Matthew Ebberman, I guess. It's just one thing that I'd like to ask you, please. Sure, sure. Anything you want. What can we do for you? Well, my name, it's all right to print that, but... Sure, sure. I guess maybe you wonder how I could tell you all this, like I was talking about somebody else. That was because Matthew Ebberman wasn't ever a person I really wanted to be. He was somebody I knew and I didn't like. I hated him. Just like everybody else did. Because he was ugly. There's no excuse for what he did. Oh, Matthew. Hating makes so many lonely people. And it makes ugly people, too. Ugly and mean on the inside. I guess the best thing would be if everybody would... Wouldn't it be a nice world if people would... My name, see, that's all right, but no pictures, please. No pictures, no papers, huh? You know, mean an awful lot to me. You understand, don't you? I don't see any pictures in the papers, will I? Just promise me that and I'll be so great. No pictures. Presented by Roma Wines, that's R-O-M-A. Roma Wines, those better tasting wines enjoyed by more Americans than any other wines. Yes, Roma Wines taste better because Roma starts with natural juices gently pressed from luscious grapes gathered in California's choicest vineyards and the world's greatest winemaking resources. Roma Master Venter's guide this great treasure unhurriedly to tempting taste perfection. Then these better tasting wines are placed with Mellow Roma Wines of years before and from these, the world's greatest reserves of fine wines. Roma later selects for your pleasure. With a holiday weekend coming up, be sure you offer your guests a choice of glorious Roma California sherry, rich Roma port and golden Roma muscatel. Served cool. Or if they prefer a tall, cool drink, make it a Roma wine and soda. Remember, Roma Wines taste better. That's R-O-M-A. Roma Wines, America's largest selling wines. Lloyd Nolan appeared to the courtesy of 20th Century Fox producers of Kiss of Death. Tonight's suspense play was by George Lloyd. Next Thursday, same time you will hear Edmund O'Brien as Star of Suspense. Produced and directed by William Speer for the Roma Wine Company of Fresno, California. This is CBS to Columbia Broadcasting System.


You are working in the emergency department today. Your next patient has been brought in by his family member because of fever and cough for three days. His triage vital signs includes the following. Fever, tachycardia, hypotension, tachypnea, and hypoxia. He's quickly put into a monitored bed. You hook the patient up to a cardiac monitor and give him oxygen by nasal prongs. His oxygen saturation goes up to 93%. You ask the nurses to start too large a variety and start him with a saline bolus of a liter. Your nurse draws some routine labs including CBC, chemistry, and renal function. You also ask for a 12-lead EKG and a portable chest x-ray. When you talk to the patient, he seems confused. His family said that he has not been feeling well for the past three days. He was complaining of fever, shortness of breath, and chills. He started having rusty color sputum this morning. According to the family, he has not had any sick contacts or travel history. The patient has not had any history of surgery or cancer. He has had a remote stroke. The only medication he is on is aspirin. On examination of the patient, his vital signs is now the following. He is still febrile, hypotensive, tachycardic, and tachypneic. His oxygen saturation is normal with oxygen. He still seems confused. He has no focal neurological deficits. On examination of the lungs, you notice a tracheotug and tachypnea. You also notice there is decreased air entry in the left lung base. His cardiac examination and the rest of examination is normal. You decide to give him a second leader of saline. His chest x-ray shows a left lower lobe infiltrate with a pleurofusion. His blood work shows a normal white cell count, hyponitremia, and a high creatinine and urea. You start on the patient on empiric antibiotics and continue to monitor his vital signs. You calculated this patient's risk score based on his conical picture and lab results. Based on your calculation, the patient needs to be admitted. You then speak to the inpatient consultants who then takes over his care. In the next few videos, we will go through the classification and the approach to the patient with pneumonia in the emergency department.