Welcome everyone to this talk about cutting-edge technologies in automated testing by Anton Antelov. We are glad they can join us today. Without further delay, over to you Anton. Hi guys, welcome from Bulgaria. I'm really glad to share my knowledge with you. Today I'm going to present a few tools or libraries, whatever word you pick, that can help you in mobile testing and also actually no matter of the automated testing context, as you will see. First, who am I? Maybe you know me from my blog.  Also for a few years now, I'm CTO of a company that we're providing drink and professional services. Or you read one of my books about design patterns in C-Sharp or Java for automated tests. Since we have only 20 minutes, I will directly start. Actually, I deleted one of the slides. But anyway, the first thing that I wanted to talk about is how we can verify PDFs. This is not only PDFs, but there are certain documents and layouts in our tests that are hard to automate. For example, an invoice like this. Or sometimes it's like a preview of a print preview of your browser, right? Or on mobile, in any way. But sometimes there are certain screens and pages that we cannot interact directly with APM, Selenium, whatever. And I was thinking in the past, a way how we handle this is we were using some kind of an image comparison. For example, Sikuli through code or with another IDE. However, we hit the problem that the data is changing. And if you in some way you have different data, it's a little bit hard to automate this with this image comparison. Of course, there are some tools that can check on the parts of the screen and you can change the percentage of similarity and so on. However, I wasn't completely okay with it. So I was thinking how we can improve this. And in recent years, I was working many of them with cloud solution in Google Cloud, in Microsoft Cloud, in Amazon Cloud. Primarily 50% of my job I'm using Microsoft sticks. So some of the examples I'm going to give are with C sharp and in the Asia Microsoft Cloud. However, I will mention the alternatives with Google and Amazon. And also all of their so-called AI cognitive cloud services are actually available in all of the languages and mostly are exposed as web services. So in order to handle this problem with PDFs and similar stuff, what are the similar stuff? Well, for example, you may need to verify charts like this. I had this problem a few months ago when we were working for a medical enterprise and we had to extract information for such graphs, right? Which of course, it might be really hard, no matter the device. So I was researching the different cloud platforms and I found this computer vision AI service from Asia. And actually, it has many capabilities, but I'm going to talk about only one of them. It's called text extraction or CR, maybe you have heard of the term. Basically, the thing that this is doing is that it can you send through this web service your PDF or the file or the image whatever. And it's going to return parse it and return all the text information inside the PDF. And also, this works for nodes like this or like this, right? And the thing that I did is that basically I created a wrapper to the API for C sharp. You need just to install to get packages Microsoft Asia cognitive services vision computer vision. And then we just need to provide an endpoint in a key that we create in the Asia cloud. Keep in mind that most of these services are paid no matter the platform. However, for example, if you I believe that up to 1000 requests, they're free, then you pay a request, I think something like this. And I will skip here the code. This is not really important what we are doing inside because most of the things here, you know, I just saw in the documentation how I need to implement them. But I'm going to show you in abstract what my tests are doing, right? So here with this computer vision service, I already initialize it. I provide the path to my PDF or an image. And as you can see here, basically the service when you call it is returning a list of strengths. And then here I'm initializing expected text snippets. And then we basically validate them. My method validates the only thing that it's doing here is that is making a collection of sort of subsetting the two lists, right? However, the problem with this approach is that even if you check, for example, this address here, you're not sure that the layout of the PDF is not broken, right? And you're not sure that this price is exactly here. And this was a problem for me. And I was continue digging how I can improve this. And I found another cognitive service AI from Microsoft. It's called form recognizer. The form recognizer, it's working in a similar way. But basically, since these are web services, they're returning JSON in both formats. And here this this analyzation, the thing that it's doing is that it's going to return the different boxes. And it's going to return their coordinates, right? And the information and the words and so on. And the thing that we can do in our code is that we can basically let me show you, we can further analyze the PDF. And then we can get these coordinates based on a template. Every time in my tests, I'm providing like an example PDF or an image. And I'm comparing the different XY and so forth on the different boxes with some kind of a delta. Usually for me, this 0.1 delta is working completely fine. And then when I'm checking the layout and it's fine, then I can I exposed basically the PDF or the image as a table. And I can get the different rows and columns. And I can check the exact text, which is perfect, right? Which is like working directly with the HTML table, right? And I further extend the solution to work on mobile desktop, etc. As you know, in APM and in Selenium, you can make a screenshot of a particular element. I had this problem in APM desktop for in one medical app. Because when app driver was too slow to get working with huge data sets, data tables. Basically, the thing that we did there is to make a screenshot of this table. And then I was using this approach to analyze only this particular image of this table of data. And then verify it in this session, which was really awesome, right? Maybe later I can in the rooms in the in-outs. If you're curious, I can show you more. Also, keep in mind that it's not only Azure. Google Cloud has this cloud vision API. I use it also in the past. Amazon also have their alternative. They are relatively similar in pricing in features. So, yeah. Now, the second thing that I wanted to show you is, as you know, is our part of our tests are good. And we write them. However, as you know, part of our job is to analyze the test failures. And this is almost half of our job. And I believe we should be able to ease this process, right? And in many projects, we started using this Docker based open source solution report portal. And I want to show it to you because it's free and open source. And it's awesome. You can check it on the report portal IO. I already hosted in Docker. So here with a single command, all of this will be created for you. I already set up the, I already walked into their demo website. You can check it. And why this dashboarding is, I really like it. Because it's real time. It's not just a report for a particular pipeline or a job in Jenkins or Azure or whatever continuous integration you use. It's a standalone application with different users, roles, et cetera. And again, when you run your tests, it's a date live. No matter how many parallel runs and pipelines you build, it's basically accumulating the data for all of the runs for the whole company. And these dashboards can be configured per team and have permissions, right? So it's really awesome. Also, you can see the launches. And then you can directly, the engineers can analyze the test failures here. And since this case integrations with JIRA and other similar systems, you can report books directly from here and mark certain tests for that it's automation, but that it's a system issue and so on. And also they have this built-in so-called machine learning. But basically, if you analyzed a few times a test with a particular stack trace, it's going to mark the next time. It will figure out that this was a problem and it's going to mark the test as an issue, right? Which is really great. You can check it.  You can deploy it on Kubernetes or on a virtual machine in one of the clouds. Also, in order to get started, they support all kinds of languages. You just need to click here on the particular technology and then you will see a completely example in GitHub, right? In C-Sharp, it's quite easy to use it. You just need, in most languages, actually, you just need configuration like this, where you provide a personal token project and some metadata. In C-Sharp, we install this new get package, report portal. Basically, this is like a plugin for the runner. Then everything starts working. It's completely the same in Java. For example, you need this report portal properties and you need a few dependencies in the form. That's it. Since we have only a few minutes, not so much time, I have one more thing to show you. It's called Key Vault. Especially about this wild stuff, like, for example, integration with Azure or SOSlabs, Lambda tests, no matter what it is, you need to store somewhere the endpoints and the subscription keys and the user names and so forth. As you know, one of the ways to do it is to put it into a configuration, not directly in the code, depending on the environment because these credentials and URLs can be different, right? One way is to put it here. However, this is not the best practice. Usually, the suggestion is to put it into environmental variables. That's fine. When you put it into environmental variables, we can get it like this. It's completely similar in Java and other languages. Get environmental variable, right? However, the big problem with this is that, for example, in our framework, when we are working on big enterprise projects, we have, I don't know, maybe 20 environmental variables to handle URLs, passwords, endpoints, subscription keys. If you're working on many machines, this can be tedious to set up so many environmental variables all the time, right? It's not in a centralized way. You cannot, for example, if your key has expired, it gets a problem to update them on all of the machines, right? So, of these cloud platforms, now they offered a solution in the cloud called Key Vault or in case of Google, this is called Secrets Manager. It's basically a centralized way of all where you keep all of your secrets there. And the only thing that you need, basically, is to initialize only two, three environmental variables to set up the Key Vault. That's it on the machine. And then, basically, you just ask for the different secrets, but you don't keep them on your machines and environments, which is great because you can, from a centralized point, keep them saved, and you can update them and so forth. And for most of our enterprises, this was actually a requirement. These big enterprises asked us to handle it this way. But, again, it's up to you. For example, here, this wrapper that I created, the Secrets Resolver, actually is working with both options. For example, in my configuration, if the value in my configuration starts with environment underscore, then I know that this environmental variable, other way, it checks the Vault. If I have it set up. We have three minutes. Maybe we can start with the questions. And I can show you many other stuff, but maybe it's time to answer a few questions. And later in the handouts, if you want, we can chat and discuss more. That's great, Anton. That was a very informative talk. We do have some questions from the participants. I'll read them out to you, and we can go from there. We have Stelios asking, which library, the library you were showing to leverage OCR and C-Sharp, is that available in other languages? I was showing this formula, the one of them for the layouts. It's called Form Recognizer. The other one is computer vision. Actually, you can check their documentation, Azure. Let me see. But I believe that they have examples in Java and other languages as well. Yeah, they have C-Sharp, Goal, Java, Node.js, and Python. And I believe it's the same for Google. Awesome. That's great. I hope that answers Stelios' question. We have another question about the image recognition, but someone anonymous is asking, how does this perform with subtitles, recognizing subtitles from a video screenshot? Is the dynamic background an issue? I haven't checked it. I have, by the way, played a little bit with captures, right, with these images that pop up sometimes on Google. It was partially okay. With subtitles, actually, I was playing a little bit with another service that they have. It's called Video Index. Basically, they have a video analysis service. I cannot find it right now, but they have a special service for the videos, and they can extract any kind of meta-information, including, I think, subtitles. But it's a little bit more expensive because they are parsing the whole video. But you can try it with screenshots, right? I have just used the video service. We have another question, probably the last one, but what we can take in this talk from Zon. I guess he's asking, is there anything that you can share more about the auto-analysing feature of Report Portal in other play? I haven't used it much. From what I saw, basically, it's just remembering the different, it's analyzing the exception messages that you get of the felt tests, right? It's remembering into the database, and then it's just looking quite in a dictionary or a map. What's going on, right? But it's not a real AI or machine learning, I think, it's just a really simple algorithm there. Most of the time, I'm double-checking the results that they are marking. At least this is my observation, of course. Maybe you can ask directly since it's on a GitHub. You can ask them directly. Makes sense. All right, so I guess we're almost out of time. Thanks, Anton, for sharing your experience with us today.


It's the Cube, covering HPE Big Data Conference 2016. Now, here are your hosts, Dave Vellante and Paul Gillan. Welcome back to Boston everybody. This is day two of the Hewlett Packard Enterprise Big Data Conference. This is the Cube, the worldwide leader in live tech coverage. We're here with Chris Selland, who's the Vice President of Business Development at Hewlett Packard Enterprise Big Data Platform. Chris, year four, great to see you again. Yes, it's always a pleasure. It's great to see you guys again here too. You've been with us from the beginning and we really appreciate that.  Well, thanks for having us.  You know, we started, I was one of them in the first year and you got to it. Well, yeah, but you know, Billy Bean was pretty good. Billy Bean was great, actually. That was the first year. The money ball guy. Yeah, but really, you had a great lineup throughout the years. Some terrific content today. We heard from folks from Uber. You had a great customer panel. I mean, yesterday Phil Black was fantastic. Yes. And so, yeah, it's all good. I mean, but you've preserved the roots of the conference, the customer and the practitioner roots. Well, honestly, and as I just said on stage, four years ago, a little over four years ago, Colin and I were walking through the hallway and we were talking about all the cool things our customers doing. You know, big data was becoming a hot topic, but it was really about the things that our customers were actually doing with our products and doing with the data. And we said, you know, we really need to get the customers talking to each other. So let's have a user group meeting. And honestly, the genesis of this event over four years was this was going to be basically the Vertica user group. And then it started kind of gathering loss and part of it was, you know, big data was a hot topic and talked to some of the folks at HB Corporate Marketing. And they said, oh, you're running a conference. I'm like, okay, it's a conference. And then, you know, it kind of went from there. But the core of it is and remains and always has been what are the customers doing? What are they, you know, and getting them communicating and working with each other. And of course we've got our partner ecosystem here as well. We've got, you know, folks like yourself, we've extended, we didn't do press and analysts, except for you guys the first year, we have all of that also. So it's really grown, but it's been great to see what it's become. Well, if you think about the really good conferences in the business that they retain those room, I mean, VMware, certainly EMC world, conferences like the, you know, the service now user conference, the heart of it is user driven content. And that much of what you see here, I was joking, not joking, commenting yesterday in the food line, somebody was talking about the problem you had with that algorithm is you needed to, you know, change the way in which you applied it. And I turned around and said, well, can you explain that in lay person's terms? And that's the kind of conversations that you're hearing in the hallways, right? Absolutely, absolutely. And it's been all along. You know, year one, we had, we actually had a reception with some customers in Billy Bean. I remember Billy Bean and Chris Weggersen who was the chief data guy from the Obama organization. We're geeking out about data talking and I mean talking for hours. And it's funny too, we're actually talking about Premier League Soccer, which is something that I'm really interested in. They weren't even talking baseball, they weren't talking politics, but just, you know, the cool conversations customers and others are having with each other and with us, obviously we participate, but we try to stay out of the way too, because this is not about marketing fluff.  You know, I've been saying for a long time, I think I'm sure I've said it on theCUBE, I've been on so many times, big data has been kind of crossing the chasm. It's gone from a technology topic.  Now it's more about solutions and applications of what companies are doing with it. And it doesn't mean IT is not in the room, there's still very much in the room, but it's so much become a business topic now and you know, the term's changing, the drivers are changing and we're seeing that. We've been seeing that over the years. And this morning's customer panel was really illuminating. I mean you had some great examples of how big data can be actually misused and what you have to get at is the why. Steve and John had best quote of the morning. They said it's easy to get the what, it's hard to get the why. And you had the example of Mercedes dealers, Mercedes dealer having a big local campaign, advertising campaign, not realizing that a lot of its customers are actually flying in to that dealer because they did something that no other dealer did and they had no marketing that was aimed at those people. And they weren't marketing that, but the customers were also finding it on their own and realizing that, I think it was a dealer, that was Paul Harrison from Simplify was talking about, the dealer in Washington DC had a certain type of car that other dealerships didn't have. They weren't talking about it, they weren't advertising it, but customers were responding to campaign based on information that they had found on their own. So, that's what we have to remember too. None of this happens in a vacuum, right? It's that we're doing things, we're looking at data, but it's the, what is it, the Heisenberg principle, that as you observe, it changes. And customers are aware and participating as well. So, that example of somebody flying from Fort Worth to DC to look at the Cardi, you asked why, because that's what you want to know to the point of the panel. Yeah, and there was also a great conversation about Brexit today, and in another case, here we have a current event where we thought we knew everything, right? All the polls had it nailed and they got it wrong and they got it wrong because they made assumptions. And I think it was Eric Dixon from Blue Labs was talking about, you have to ask the questions about where that data came from. How are you collected? What questions did you ask? We ignore that kind of stuff. Well, do we know?  Right, well, Brexit was a good example where, and there's some suggesting that even this election season too, right? I know we're not talking politics, but that what the polls are saying, we've got lakes and lakes and lakes full of polling data, but yet more of it is it telling us the right answer. And we're still, you know, not sure. I mean, clearly in Brexit, the polls were wrong. The polls are out and out wrong. There were plenty of polls, talked to plenty of people. They weren't right. So, but yet some got it right. I think there was some talk about a hedge fund nailed it. But they weren't necessarily sharing that data publicly. But it's true, you woke up the next morning and you were like, whoa, that's not what I expected. It's not what they're reporting on CNBC. It's not what any of the major news outlets report. As you mentioned, the hedge fund, as Eric said, some of the hedge funds, the private equity funds did know what was going to happen. Of course, they weren't telling anybody because there was a lot of money to be made in having that answer. You know, why were they thinking of things differently? And I think of all the research that is out there online right now that we just take it face value, we don't ask questions about how were these questions asked? What was the survey base? You know, who are these people? Right, right, exactly. Well, I think it goes to the theme of, you know, hedge funds aren't necessarily known for sharing their techniques in the public domain or putting them in the open source community, right? But, you know, it also goes more seriously to learning, you know, learning from the data. And, you know, Robert was talking this morning about why when we say machine learning, we like to say applied machine learning. Because it's, you know, what are we learning from it? But what are we also applying as people? There's so much in the news right now, for instance, about, you know, robots and you can apply that to big data, you're going to replace all these jobs and there's going to be no jobs because it's all going to be taken over from machines. But that's not necessarily true. It's that we can use what we learn from the data from machine. Believe me, we're not advocates of throwing away data. We do think more data is good. At the same time, just because you have a data lake or many data lakes full of more data, that doesn't necessarily imply you've got the answer. The answer is something that comes along with human participation, learning, you know, refining, modeling, and again, so much what we're doing, also, you know, building modeling capabilities into our products right now. Again, it's customer driven, right? It's what customers told us they wanted to do. It's not just about more data, faster data, but it's about running models on that data and seeing what affects what, you know, what impacts what. So that's really where the learning comes in. And people are still very much involved in that. Trial and error. As you're talking, one of your panelists is talking today about A-B testing and how much A-B testing is wrong. Or he said the third of it is just bad. It goes wrong. A third of it validates an assumption. And a third of it finds a completely different variable that they hadn't even expected. And that's what you, that's what strikes gold. Exactly, like penicillin. Yeah. So, precisely, it's what you discovered that you weren't necessarily looking for. But how did you need to at least know to look in that place? Or how did you start to realize there was correlation between things that, you know, factors and it might've been a time-based correlation. It could be based on another thing. I thought the example in the other panel, it's like, oh gee, you know, when we sell more ice cream, more people drown. It's like, you could draw that correlation if you just look at data, sure. But you realize that, no, we sell more ice cream because it's summertime, more people swim in the summertime. But those kinds of things, right? There's the causation effect and, you know, what impacts what. And when this variable changes, what does it do to that? And not just variables, behavior. Because we're talking about people here.   So, because that's the thing, you know, Vertica was always kind of an early adopter product. We had a lot of customers who would buy the product, love the product, but what we actually said, what are you doing with it? They didn't necessarily want to share, you know? And cause doing some pretty cool stuff and they're taking the technology and directions we never go out. Well, we had a logic bot guys on yesterday. Oh, that was crazy. I was like, what's your secret sauce? He's like, well, I'm not going to tell you. Right. You know, it's some cool stuff. Everything is a trade secret. So I want to talk about the market a little bit and then get into the ecosystem.  And what I was, we were talking about yesterday, Paul and I was that ROI in the early days of a dupe was sort of reduction of investment in my existing enterprise data warehouse. And Vertica, as Colin said yesterday, sort of always sat between the sort of open source Wild West and the, you know, traditional snake swallowing of basketball enterprise data warehouse. And you added value to that whole scenario. So take us through sort of what you see in the marketplace, what's happening, a lot of tumultuous times for some of the dupe vendors. Nobody seems to be making a lot of money there. Some of the high flyers have been knocked down a bit. What do you see and what's happening with Vertica? Yeah, well, part of it goes back to what I said earlier. I do think the market is now moving to solutions and what organizations are doing with the data as opposed to, you know, it's become more of a solutions market and the customer stories you hear now are around what we're doing, not just how much data we can load, how fast we can query it, how much we can store, how much it costs to store it. I mean, look, we think that Hadoop is a great technology for storing very large amounts of data, very inexpensively. But there's also, you know, there's a price performance trade-off and curve in this industry that different technologies for different places. I mean, as I said earlier, we are advocates of keeping all of your data. But, you know, we've had a couple of customers, for instance, in the predictive maintenance industry, you know, GE who's moving into, I was actually reading something based on that jet engine that blew out the other day. GE who's moving in, they're moving next door, right? You hear these stories about jet engines giving off a terabyte of data an hour. And I always say, I want that data to be really boring data on any flight of mine, right? Yeah, really. But at the same time, you want to look at it longitudinally over time to just see, you know, first of all, yes, you want to see in the near term are the anomalies or the things I need to look at. That's the data you need, like, right there, right real-time access to what's going on right now. So if there's an issue, you know, it's an airplane engine or, you know, my website, my e-commerce site or what have you, or, you know, just the traffic into my store volumes, all these things you want to know right now, you know, that's, cost isn't the main driving factor there. It's a performance thing. At the same time, do you want to keep all of it? Yes, you actually want to keep all of it. I heard a story the other day that one of the other, Natchi, and I won't name them because I heard it from a third party, but one of the other airline manufacturers, every time a flight lands that they have, the flight lands safely, they have to throw away all the data because they can't afford to keep it. Or, you know, at least this was in the past, that's kind of crazy because then you're losing some insight. Now, are you going to query that in real-time? Do you need real-time performance? No. And that's a lot of data. You know, a terabyte an hour per engine, I mean, you're talking about a lot of stuff to save. You don't want to throw it away. But, you know, as you kind of go up and down the curve, performance changes. So, we've kind of really architected what we do to really occupy, you know, a big, big chunk of that curve. But, you know, we're not necessarily trying to be the cheapest. We certainly, for the types of things we do, Chuck Bear was talking about use cases, try to be the highest performance, but you realize there's an ecosystem, there's different technologies for different problems as well. So.  Colin talked about it. Yesterday, your CTO talked about not moving the data combinations, which I want to talk about, is, you know, this new capability that allows you to combine different API functions without moving the data. The flip side of that is, data used to be, you remember, when you were an analyst, data was a liability, right? The general counselor was like, get rid of the data. After seven years, get rid of the data. Anything that's work in process, get rid of it, because there might be a smoking gun. Still is. And still is, right? So, when you guys are talking about keeping it forever, it's going to give the general counselor a jada. So, what are those conversations like? Well, it's not going to give the general counselor a jada if you do a good job securing it. Robert also talked about security, being one of the biggest use cases, actually, for big data these days, right? So, security compliance being another one. So, if you're protecting it effectively, it gives less adjada. But, yeah, it's still not an argument for throwing it away. Because, by the way, as they say, dumpster diving is probably one of the, throwing it away doesn't mean it's gone forever. It doesn't mean the bad guys can't still get at it either. So, keep it, learn from it, use it, but extract the nuggets. And sort of understand, I talk a lot, we like to talk a lot about hot, warm, cold data, kind of where price is more important, where performance is more important. And the problems you're trying to solve coming back to the customer stories, because that's what people really want to talk about right now. They don't want to talk about query speeds and data loading and so on and so forth.  But at the same time, we see like MapReduce being ripped out just about everywhere we're going. And currently being replaced by Spark, or at least attempting to replace it with Spark in many organizations. But, you know, there's a progression. And I mean, a year from now, there'll be new stuff. You know, I've talked to, you know, Merv Adrian at Gartner a lot about this. He likes to talk about, you know, flavor of the month, flavor of the year. There's a lot of new stuff, and it's good. It's good that we can use all of this technology. But it, you know, it both replaces and complements. And some of it sees, you know, long-term utilization. Other stuff goes away, but I do think, you know, you mentioned earlier that, you know, the market's adjusting to this is becoming much more of a business and also financially-driven thing right now. It's not just technology for technology. I want to ask you about that. I think it was Gartner that said last year that the big data was mostly being used for operational cost reduction purposes. That was really where the sweet spot was. Not so much strategic, you know, new ideas developing new markets. Are you seeing that change at all? Well, yeah, but I think that's where the insight stuff comes in too, right? That was like that Mercedes example we talked about earlier, is the, you know, what am I going to do to make it more likely that a customer's going to come into my store or what do I make it more likely that when they're in the store they're actually going to buy something? That's less about having a massive volume of data and that's more about the insights that we gain from it. So, and that's the turning point. Now, do we still need to keep everything for compliance and, you know, other types of purposes? Yes, we do, but it's really the learning. And that's, and again, that's where this whole concept of applied machine learning and a lot of what we're talking about here at the conference comes into play as well. And also, you know, moving the analytics to the data, that's the other thing, because I think it was Yop talked about how hard predictive is, and he's right. But one of the other things that's right, that's so hard about predictive is that most of the predictive products and capabilities out in the market today, they can't work. This is sort of the dirty little secret of the industry. They can't work on large volumes of data. You have to sample, and you have to reduce and sample and pull into these platforms. And, you know, so we've had predictive analytics products out in the market for a long time, but they can't necessarily run across these massive volumes of data, but that's where they're most valuable. So we're doing a lot of work to try to embed predictive machine learning capabilities in our core products, so you can actually run the models across large volumes of data, as opposed to, sure, do you want to run a model on the engine while it's running to see if there's any anomalies, it's running hot, something's wrong, yeah, there's high friction, of course, but at the same time, you also want to be able to take those models and run them on these massive volumes of data. And so that's something that I think you're going to see emerging as well, is doing predictive modeling on very, very large to find those insights that are the key things that sort of jump out. So. I remember the first year of this conference, I got into a little debate with Kurt Monash, which is never a good idea. And, but my contention with sampling is going to be dead. And, and, and I don't know if that's occurred yet, but it's happening in my view anyway. But I want to talk about the stack and the ecosystem. Steve yesterday from Ameripride. Yes. Was basically talking about the stack that they're building. We're taking a little stuff from the cloud stack. We're going to take some Vertica. We're going to take some Tableau and they're basically building their own big data pipeline. Yes. And the ecosystem is doing that as well. And that's something that, you know, you, you're watching things growing and it's building momentum. Talk about what both customers are doing, but also the ecosystem, they're sort of mirroring each other. That's absolutely what it takes. I mean, big data is, there's absolutely an ecosystem. That's one of the things that we've, you know, really made sort of a key part of what we're doing as well here too. Because I actually remember being at Strata three years ago, maybe it was three years ago, three, four years ago, you probably know the timing. And we had our booth and we were, you know, we were on the floor and right next to us on one side, EMC announced they were going into the Hadoop distribution. I think they announced Pivotal. On the other side, Intel announced they were going into the Hadoop business. They were kind of doing their own flavors, right? Well, yeah, but the thing was they were, everybody was going to do, we're going to have ours and ours and ours.  Let's not have ours. We remember asking you, are you guys going to do it? We want to know what we're doing well. And let's work with all of them. And that's really what we've tried to do. Now, all can be somewhat aspirational because we're really driven by, you know, what our customers tell us are the most important ones to them now, but like cloud, for instance. I mean, we've been on AWS cloud for a long time now. We've had Vertica on AWS. But when we launched Haven of Demand, you know, we launched it on Azure. And at the same time, we were definitely seeing demand from our Vertica base to be on Azure. So we're going to be on Azure. And, you know, we had a press conference yesterday and questions about other clouds. And yeah, we're certainly looking at and working with other clouds, but we see, you know, we need to work with the entire cloud ecosystem. We need to work with the entire BI and visualization ecosystem, ETL, data preparation, data loading. We need to work with that whole ecosystem, just the platform ecosystem. And of course, you know, and things like Hadoop and open source. And you heard the work that's going on with Spark, with Kafka, with Hadoop. So, you know, working with that entire ecosystem is key part. And yes, that's my, that's my day job is to work with that ecosystem. And, but it also ties into, you know, where our customers are telling us and what's important to them. So, I mean, it's got to be simple and transparent to developers. So when young Johns was speaking this morning, talking about the combinations and 70 APIs, and because we, for our own purposes, we're playing around with machine learning and AI, Watson. Yeah. And it's somewhat- We've got crowd chat. But so I'm listening to him. So I hit the Haven on Demand site, saw these things as video, as text analytics. I'm sending it to our development team. They say, check it out. You know, see how easy it is. Cause we've struggled, you know, as a developer integrating that, some of that stuff. It's not so easy. I'm hoping that Haven on Demand is easier. And it shouldn't matter what cloud it's on. You showed a video this morning, it certainly made it look easy. Are you going to, are you going to, is there going to be some sort of proof of concept or something that is easily accessible by developers where they can take it for a test drive? I mean, we build, you know, like in our marketplace, we build things called quick starts that are sort of demo applications. And we've got Haven on Demand, we've also got the Vertica marketplace where you can go see these examples. We've got some of the kind of, and a lot of them tend to be smaller companies, developers building on our products. But we're really also giving it out to customers to say, you know, you build it and bring it back if you choose to share it and learn from other customers. And by the way, come to our event and talk to other customers as well. So, you know, Eric from Blue Lab said something, and I don't think he got to it on stage today, but I was talking to him privately yesterday. He said, really, we were talking about the Brexit polling thing. He said, it's not looking at the data about the voters, customers, in his case, voters that's important. It's not looking at the data, but it's actually observing what they do. And so that's when some of the, you know, and like if they're on your website, that's clickstream analytics. But if they're out walking into a retail store, that's video analytics, audio analytics, you know, mobile analytics, so, and being able to sort of combine this stuff and put it together as some people like to say, mash it up. Is it simple? There's a lot of science behind it, but there's also a lot of art behind it too. And that's where the people come in. And we're trying to give the best of breed tools to the community, developer community, and that also means large corporate developers to let them do it in a way that wants to do it. And it's getting a lot less expensive. Yeah. Pretty much anybody can try it. Absolutely. And just the whole notion of bringing analytics to the data. I mean, that is the epiphany of Hadoop, right? Just leave the data where it is. Right. Just to throw it away. Right. Because it's so cheap to store it that there's no reason to get rid of it. So, from that perspective, Hadoop's been a great thing. That's the real game changer in that sort of bit flip. But, all right, we're out of time, but thanks very much for coming on. We'll give you the last word. As always. Put the bumper sticker on 2016 BDC. The bumper sticker on 2016 BDC. I like Chuck Barris' t-shirt this morning. What was his? Seizing is believing. Seize the data. So, I don't know if that's the last word, but I liked that one. Hashtag, seize the data. This is theCUBE, we're live from Boston, Massachusetts. Chris, thanks very much. Thanks, Dave, thanks, Paul. All right, CUBE right there, everybody will be back right after this short break.


Today we're opening McDonald's packs at McDonald's. Today I decided to open rare 2002 McDonald's booster packs into McDonald's, and you're about to see what happened. But first, I will be giving away this sealed McDonald's pack from 2002. These are pretty rare, it's like $40. All you have to do is like this video, be subscribed, and let me know if you have any memories from opening McDonald's pack back in the day. It doesn't have to be the 2002 McDonald's pack one, it can be even McDonald's pack two, one of the older GX versions. So just let me know in the comments. Now let's get into the adventure. What's up guys, Ruckson 34 here, back with another video at Mickey D's. Hello, how may I help you today? Do you guys have any Yu-Gi-Oh card toys? No, we do not. Oh, can I just get it? Well, we tried. Looks like we're gonna have to go inside to see if we can get anything. We've got a new addition to the toys here. We've got the McDonald's Yu-Gi-Oh pack. So sing to, you're out of here. This is the new, this is the new thing.  What, what? Where's the option? What's going on here? Wow, my kids' meal. McDonald's pack from 2002. 20 year old kids' meal. We have 10 of the original 2002 McDonald's packs. We could pull Millennium Shield, we could pull Cosmo Queen, others, there's some Super Airs in here. Only two cards per pack. So I figured we'd go to McDonald's, open these up and see if we can get some more. First pack, this one came out of the kids' meal, it's unbelievable. All right, here we go, our first pack. Can we pull the Millennium Shield? One of my favorite cards. In McDonald's, it's like the old school days. Ooh, the Yamadron, very cool. And there's gift, okay. Just a couple of comments, that's okay. I didn't get any chicken nuggets, I wanted to avoid the chicken nugget fingers, so here we go. It would have been more nostalgic to get the chicken nuggets though.  All right, next pack, let's see. Stuffed animal, okay. And the three-legged zombies, all right. We've got a couple of common packs so far, but that's okay. We're just looking for one ultra, that's all we want. Let's see what we can get here. All right, so far, four different comments. Only two cards per pack in these, so. Ooh, the Megasonic Eye. And a flying penguin, very good, very good. Okay, flying penguins are actually pretty good. We have seven more, seven more chances. You know, hit that like button if you guys want us to get it. See what you can do. Seven more packs. Get the Millennium Shield, the ultimate goal. Fairy's gift, we already get that. And stuffed animal, oh no. Oh no, it's all repeats, next pack. All right guys, this is the time. This is where we do it, this is where we pull it. Maybe, we'll see. Oh, the flying penguin, oh the Iran, so that's a different one. Okay, so far, this is what we've gotten.  These actually are still of some value though, because they are kind of hard to get. Stuffed animal, if you like it's on me. Yamadron's one of my favorites. We got what, the halfway point with five more packs. Let's see if these, what have they called? Kids Meals will give us some luck. Happy Meals, there we go. You're a lot of soft jals, it's okay. Turtleberry and the Yamadron again. Just one foil, that's all we want. Next pack. McDonald's in McDonald's, it's kind of crazy. It's like Inception. Except Yu-Gi-Oh! in McDonald's. Frog the Jam, the legend. We pulled the Frog the Jam, legendary. And the Ushioni, okay we've gotten different cards there, that's cool. Still no Super Rare or Ultra Rare though, that's what we're really looking for in this. In the McDonald's, 20 years later. Fairy's Gift. And the Yamadron, oh no. All Commons. What a disaster, two more packs. You gotta believe. We need a French Fry or something to save us. Those are our only chance. All right, here we go. Geronzo and the Turtlebird. Last pack magic, here's our last chance to pull something out of these. We'll open 10 at packs in at McDonald's 20 years later. Or they actually give us something. We'll find out here. Mega Sonic Eye. And the Flying Penguin, it's all Commons. But let's see what we got. Some really cool old school McDonald's Commons. We also get the Happy Meal, so that's a plus.


We are now going to segue to a presentation that's going to be delivered by Jeff Howe, our last questioner from the last session. Jeff is the, this segment is entitled Harnessing the Wisdom of the Crowd. Jeff, as he mentioned, is an assistant professor at Northeastern University. He's the author of the book Crowdsourcing. Why the power of the crowd is driving the future of business, a book that came out in 2008. Most of you, I think, picked up one of our programs that Alicia, who works here, was very creative in designing as an instruction manual. So this would be step three, the very important step that some of us often do not do, which is asking for help. So that's what we're going to hear about now, crowdsourcing. Jeff also runs the Twitter book club One Book 140 for the Atlantic. Jeff? Thanks a lot. Don't ask me for help. I just write about this stuff. All right, so first, this gets really anticlimactic because now I have to spend the next three minutes nattering while I plug in the system. So I want to thank Slate, the America Foundation, ASU for having me here. It was a lot of fun. So I coined the word crowdsourcing, which means I've wound up talking about it a lot for a long time. And I've always wanted to wind up in the same room with maker folk, because I write about it, and it's in my book, and I was really fascinated and enjoyed the last panel a lot, so thanks, guys. All right, so I like to start off with a movie because people like movies. And that way, if you've never heard of crowdsourcing before, which happened to be a lot in the beginning and happens to be very rarely, now it puts us all on the same page. All right, can we dim the house lights up here? So the video, is there an event person and otherwise it's going to be kind of lame. Just like that. That one has to stay on. Will you guys tell me if it's okay, it's okay. And we need sound. We need to gather physically in order to create a crowd. Suddenly, with the internet, we were able to create a virtual crowd. And what this allowed is it allowed people to get together through intent, through shared interests. This is new, this is a fundamentally new development in the course of human history, that communities are able to form simply out of shared interests, a shared passion for a hobby, a craft, and art. So crowdsourcing is when a company takes a job that was once performed by employees and outsources it in the form of an open call to a large, undefined group of people generally using the internet. There's a couple crucial terms in there. One is open call and the other is undefined and they both get the same idea that the person who you think would be best qualified to perform a job isn't always the best person to do it and the cocktail party version is very simple. It's crowdsourcing is Wikipedia with everything. Photography is a great example because it's what I think of as the canary and the coal mine. What's happened over recent years of photography is the advent of three separate developments. The first is the digital camera, the cheap, affordable SLR digital camera. The second is photo editing software which has become easier and easier to use. And the third of course is the internet. And so the quality level of stock photos produced by amateurs essentially reached an equilibrium with those created by professionals. What happened was that stock photos were no longer a scarce commodity. They became an abundant commodity. Instead of charging $300 per stock photo they charged a dollar and as you can imagine the demand for a commodity that had been marked down by 99% was huge. I don't think crowdsourcing eradicates business it just changes it dramatically. It forces companies to approach us as potential partners and that's much more interesting and much more exciting. We do buy things but we also participate meaningfully in the process by which those products are created. What we see with these successful forms of crowdsourcing is that they came up organically from the people formerly known as customers from the people formerly known as the audience. The technology is so good that it's become easier for people to become very good. And then finally the emergence of online communities. I like to think about the online community as kind of a building block of crowdsourcing. It's what the corporation is to the industrial era. It showed that people could come together and self-organize into productive units what once took managers and a corporate hierarchy can now be done in the context of the community. So now we know what it is, right? I'm always kind of embarrassed by that because Wired made that right before the book came out. I worked at Wired for a long time and I was working there when I coined the word it was for a Wired article. And the thing about those movies are they make you take like ten takes. So you start out really excited but like maybe too excited. Then you get really just like bored and you know we need that passion back. And so you wind up looking really funny. You're like, the thing about ground. It's a little embarrassed by that. All right, so welcome to what totally new. I get really bored with my decks after like a year so I just totally reinvent them and cook up new stuff. So this is what I've been looking at late. I've been following crowdsourcing on and off since 2005 because it certainly pre-existed the word. Although there weren't nearly as many of the incredible examples as there are now. This is some of the stuff I'm looking at. I'll tell you what you're looking at. I'll tell you what you're not looking at. You are not looking at a retro-viral protease from the Mason Pfizer monkey virus. What you're looking at is a proposed design for the retro-viral protease from the Mason Pfizer monkey virus. The reason it's a proposed design is for a long, long, long time. Over the years, no one could figure out what the design of it was. It's an enzyme, and enzymes are composed of proteins. How these proteins fold determine their function. And sometimes that function is to make us really, really sick. In fact, in the case of this one, it's to kill us. Figuring out obviously what that function is helps us cure such diseases. So it was something of a coup in September when a group of researchers published an article in the journal Nature, Structure and Molecular Biology that after years of a, quote, wide range of attempts, and, quote, they had successfully modeled the crystal structure of the, it's commonly called the MPMV or monkey retro-virus protease. Among the co-authors was a very unusual name, the Foldit-Voyd-Crusher's group. It was not often seen in the venerable pages of a nature journal. So what was the Foldit-Voyd-Crusher's group? It was a collection of video game players, many of them without even college degrees, much less backgrounds in nanobiotechnology. And they were from all around the world. And after years of all these wide range of attempts, the attempt that worked for the collection of University of Washington molecular biologists, computer scientists, and video game designers that had created Foldit was to turn the search for the MPMV retro-virus into a video game. And that's this. So we're going to return to nanobiotechnology.  I do like talking about the history of the term because it reveals something about the phenomenon itself and how it's come to be framed. I mean, that's certainly been a collaborative effort. But I think how that term first came about reveals something about the DNA of it. So I was a wired contributing editor from 2000 to 2010, roughly. And 2005 was what I was thought was my gear of the kid. I covered music for a long time in the music industry. And in the spring and summer of 2005, I was writing about Myspace. And to the extent anyone knew about Myspace and there weren't, it's hard to imagine. Now, it's meteoric rise and just as precipitous fall. But to the extent people knew about it, all they knew as it was disrupting the music industry because bands were going, they called it going band to fan. They would book their tour, find sofas to sleep on, sell merch, distribute their singles. So I was writing about the fact that Myspace was having on the music industry. And so as part of that, I basically spent the summer on the Vans Warped tour, which as anyone else been on the Vans Warped tour. I had a blast. It was awesome. I'm old, but I still skateboard. So I like doing stuff like this. And I spent a lot of time with kids in their tools. And their facility with those tools was so stunning. And this is not an epiphany for anyone in this room. But the next thought was it was that to them, you know, flash software or anything web design or their video cameras, their DSLRs, 2005 pre-DSLR, their cameras were what pencils were to us. And I mean, why are there this real culture? We're very focused on the technology. And the technology is the story. And I was like, this technology is just not the point for these kids. And the point wasn't that they were learning how to take cool videos because they wanted to be a videographer. That was just completely foreign to them. No, it's just I want to make a video. And that felt very new to me. At the time, we were very focused on user-generated content. And I thought that that was a loathsome term because, you know, we aren't users and no one sits down to generate. And when we do, we don't make content. I mean, content alone you could have me on for an hour about. But more than that, I was like, there's no way this is going to be restricted to content. Like, why are we setting up this weird artificial parameter around content? Because really what's happening is a revolution concerning the means of production. And we produce all sorts of stuff, not just content. So I didn't have a word. And the article came out and had none of this in it. But in that way, as the journalists in the room will relate to, you know, the seed for your next story is embedded in the ground for the first.  I just, it was all very inchoate. And so I called my editor at one point and what I had was this metaphor and friendly wired illustrator types that are willing to make my slides for me. The metaphor was that in focusing, and remember this is 2006 on user-generated content, everyone was staring at standing around old faithful saying, look, this is the phenomenon. This is this badass thing. It shoots up exactly every 43 minutes. And this is what we should all be focused on. I just had this idea that everyone was gathered around this and no one was looking at the fact that there were actually other little guys who were shooting up. Maybe that had nothing to do with content. And in fact, Eric Von Hippel was my first little geyser because I just started frantically calling people and one of them was Eric Von Hippel mentioned in the previous panel. He's at MIT and has actually been looking at user-generated innovation since the 80s. And so it was already happening and of all things the Integrated Circuit Board. Engineers who were buying integrated circuits and then saying, I can make this better than your engineers because I use this thing, you don't, you just make it. So I want to make it on my own. And so the company started saying, yeah, sure, you can do that. So I knew I had these little geysers that had nothing to do with content. A lot of the discussion I was having with Mark Robinson, my editor at Wired, revolved around the fact that this would be a disruption is also a great promise. Clearly you look at something like this and you recognize that there's a democratizing potential and an egalitarian aspect. But also a huge disruptive influence that I would argue actually we're only now beginning to see and we'll get worse before equilibrium is reached. And so he said it's like outsourcing the crowd. And I said, oh, crowdsourcing. So that was the birth of the word anyway. And the article came out in June 2006. So we took this slide right before the article came out. There were three hits on Google. They're all written by people who worked at Wired or slept with people who worked at Wired. And then this was like two weeks later, which I don't actually believe that 700,000 people were writing about crowdsourcing. I think that Google Analytics are really weird. But anyway, right now it's like 9 million. And certainly it did kind of take off. And by far not the best article I've ever written, although it was a pretty good feat of reporting. I spent six months. I probably talked to 50 people. There weren't that many examples to make my case that there were many geysers and what we should be looking at is the tectonic shift that was actually creating a situation in which you would see these surface manifestations. So it took me six months to find the geysers. And the big breakthrough, and I mentioned it in the video, but was stock photography, where you had a case that the crowd was undercutting, was completely total disruption of the stock photo industry. It's a small industry, $2 billion. It's about the same price I looked this up because I was curious for the global market for orchids. It's also $2 billion. But you had one very large public company, Getty, that had to find a private buyer. They found a private equity buyer and were smart enough to buy eye stock photo for around $52 million. After having said that these guys were barbarians at the gates, they're the amateurs. We don't need to worry about them. They turned out to worry about them because they saved Getty's business. It's now the most profitable part of Getty's business. So all of this is animated. Maybe this is old hat to everyone. I hope it is because it would say that these simple words, which deserve to be known by everyone, are known by everyone. If not, it is from Bill Joy, co-founder of Sun Microsystems, that no matter who you are, most of the smartest people work for someone else. Adam, how am I on time? So that's about right. So my favorite illustration of Joy's Law, just because it's so damn simple, is Innocentive. How many people in the room know about Innocentive? All right, good bit. I can still talk about. So Innocentive was a project started by Eli Lilly, by an Eli Lilly researcher named Alf Bingham in 2000. And the idea was, is even as many awesome PhD super smarties we have in our halls, we certainly, Bill Joy's Law, don't have enough. And we get stumped on projects and problems. So what would happen if we took a problem and we posted it to the world? Common idea now, the X Prize format, not so common then. So they created, well essentially it was just a bulletin board for scientists and they hired some people to go around the world and especially going around the world because a lot of the underutilized scientific intellectual capital exists in the former Soviet Union, India to some extent, China, and got scientists to sign up on this bulletin board so they would be able to see a problem. And lo, it was good. About one third of the problems were successfully solved. It doesn't sound like a lot except that when you think of that these are problems posted by Eli Lilly once they opened it up, a big consumer packaging goods firms like Colgate Palmolive, places that employ a lot of very smart people who can solve their problems. The problems that had stumped them, one third of them were being solved. Flash forward four or five years, Karim Lakhani, he's now at Harvard Business School, but at the time he was actually working with Eric at MIT, does a case study about an incentive trying to figure out how it works. He found three very, very interesting things. The first was that one third of all problems were solved. The second was that, and this is just amazing, that there's a positive correlation between success and distance from field, which is wonk speak for the idea that the less accredited you were for a problem, the better chance you had at solving the thing. So if you were, and this was anecdotally, I saw this over and over again, I would interview successful and incentive solvers. If they were chemists, they didn't bother looking at the chemistry problems. Why? Who can say why? Why would the chemists not look at the chemistry problem? It's already stumped all the chemists, right? Why are they going to look at that? They look at the crystallography problems, they look at stuff that they know nothing about, that they can apply their unique intelligence to a completely new domain. And finally, that the winners knew the answer. Again, this is, you know, think back to fold it. This 10 years of microbiologists stumped these answers that had stumped some of the best experts in their field. The winners knew whether or not they could solve it in 20 minutes. So it was knowledge that was already existing for them. In the interest of time, I won't tell you about Yuri Bodroff, although he exemplifies all the above lessons. Because I really want to go back to Atterna. Back to nanobiotechnology and leave you with that. So Atterna was started by Adrian Troy. He was one of the co-founders to fold it. And it is, for the sake of the brevity of this presentation, a lot like fold it, except with instead of looking for proteases, they're looking for synthetic RNA. The big difference is that, every week, they take the best gamers, right? So it's a video game, just like fold it. People of all ages, some as young as 11, are designing RNA. They take short tutorials. And then the difference is that at the end of every week, Adrian Troy and his team at Carnegie Mellon take the leaderboard, the top 10 designs for synthetic RNA, and they actually synthesize them in the test tube, see if they work for real or not, and then plug it back into the video game. And what I really love about this, I'll just let this play while I finish up, is that they utilize something that I saw again and again in the last six years of research, which we can call it competitive collaboration. It is being called competitive collaboration. You can come in, and if you don't like your design, you can take the winner, the one at the very top of the leaderboard, and just poach his complete design, and then you can try and improve it. And what this has meant is that it's given the collective the ability to accelerate their discovery and improve their results much faster than any individual could, and much faster, it's worth pointing out. Fold it, a rose out of Rosetta. Does anyone know what Rosetta at home was? It was millions... Remember how brilliant all this was going to be? Settie at home? Millions of computers that together were going to figure out our proteins fold. Guess what? All that magic computing power, it couldn't do it. That was one of the approaches that failed. It was three weeks of this stuff, of 11-year-olds with amazing pattern recognition. They were the ones who were able to do it. What are the applications? I don't know, but I found this, which is an RFP from DARPA, and they're going to... They want to use the same gamification principles to identify security flaws in our global middle-gov networks. You guys are from Washington. You know what that means I don't, but in our national security computer systems and who designed the RFP, fold it. So I would propose if it does not sound like hyperbole that the sky is the limit for this stuff. Looking forward to... Do we have time for Q&A? No. All right, well, I'm at crowdsourcing if anyone wants to ping me. Questions, answers, complaints. Thank you very much, guys.


I am Pritha, I am a fellow this year at the Berkman Klein Centre and I am also a research associate at the Howard School of Public Health in the Department of Social and Behavioral Sciences. So I am looking at how population health can be improved with the use of technology and in particular in low and middle income countries or the sort of disadvantaged populations in high income countries and in this I am looking especially at maternal health outcomes. So if I talk about recent events we have this universal identification system in India called ADHAR which is being linked to track people and that has potentially a lot of use in public health. So for example our TB program, tuberculosis program, the financial assistance that is provided is being linked through that. The privacy implications of this are really huge. So I guess what technology can do, we should also be wary of those very same things at the same time. So that balance is, I don't know how we are going to find it. I am working on it myself. So the potential is huge. But if you say like in a country like India if you say that you are not going to provide the services if a person does not have that ID yet, that is because you have to understand implementation is a huge challenge. So that is a problem. And secondly the privacy part of it really scares me because you are linking all sorts of data through this one ID and the government has access to all of it. So what I don't think enough is being done to address that or even research on that to sort of how to mitigate those concerns like how can you use the technology for the good but also reassure citizens, like there should be a mechanism to protect the privacy of citizens and I don't think enough is being done on that front yet.


How do you share your workshop with a friend across the country? Not totally a woodworking discussion today, but one that I think you will want to stay tuned for because through what I'm going to tell you about, you can indeed share your workshop with somebody who's across the country, whether that be friends or family. So stay tuned and I'll show you how. Woodworkers, Paul Carlson here, small workshop guy. My YouTube channel, my website, a lot of my social media, is all about the design of a one-car garage workshop and how to make that work for yourself as a weekend woodworker. I mean, how wonderful would it be if you could actually take not just the old panning 360-degree view and then it all looks really weird, but an actual picture that is three-dimensional and then you're able to share that with a friend and have them on their phone look at all aspects of your bubble. What if I can stand right here and do a complete view of my workshop and then share that with somebody? Wouldn't that be cool? Well, you can. There's an app. It was free so far as far as I could know. I can't see any downsides to it. Hopefully it's not run by the Russians or something and not just trying to collect your email address. But there's an app that I downloaded to my iPhone and my iPad called BUBBLI. Kind of like Bubbly and it has to do with shooting a bubble. So what I'm going to do is go ahead and show you the process of my shooting a bubble and then I'm going to share that with you in a link. You can go to that link and you can look around my workshop. But when you get done with that, think about it.  It is so cool because you can use your fingers on the phone to go all the way around. You can also move the phone around and look at different things. Look up, look down. Enough of an intro.  I'm going to hit the record button here and then the first thing I'm going to do is turn my hands up and down.  I've only got a certain amount of time so I want to get this preset out of the way. Then I'm going to turn around and I'm going to start filming so I'm going to go down, get my pop-up workbench in there, my samurai carpenter workbench, go all the way up, pan down.  So do me a favor, put that on your phone, go over to Facebook to Small Workshop Woodworking Community and join if you haven't already and then take a bubble and share it with the rest of us so we can see what your workshop looks like. That'll probably force you to clean it up, which is a good thing.


All right. Hello, everyone. We will head straight into our next session, which is a series of lightning talks. I do hope you will all stick around. My name is Fallen Modi, and I will be moderating this session. We will have four speakers and each will be speaking for no more than eight minutes. So with that, we're ready to start with Chris Aberson. He's a professor and chair of psychology at Humboldt State University. Over to you, Chris. Great. Thank you. I'll be talking today about some of my experiences as an editor, trying to move a journal toward open science practices. To give you a little background, I'm an editor for a journal called The Analyses of Social Issues and Public Policy. It's a journal that began in the 2001, and it's published by the Society for the Psychological Study of Social Issues, and it served the need for the society as the society's other outlet, which is called The Journal of Social Issues, which was only published. Basically, every issue was a special issue. It was like an edited volume, and it was like, we didn't have any space for standalone empirical articles. In 2017, I became the outlet's fifth editor. In my interview and all the documentation I submitted, I emphasized open science practices such as open access transparency and openness promotion guidelines, open science badges and things of that nature. Pretty early on, it became clear that open access wasn't going to happen because there are contracts with the publishers that we can't change. But all the other stuff, everyone on the committee that hired me was quite enthusiastic about. I became the editor the first day of 2017, and I adopted transparency and openness standards that day. I, as per my contract, I had cleared everything with our society's publication committee. They were on board with everything. Everything was fine until about six months in. Before our yearly council meeting, I got an email from the society president that had directed me to remove our journal as a signatory. This was a new issue, so it couldn't be discussed at our council meeting that year, so I had to wait until the next year to have a spot on the agenda. The lesson that I took away here is that I failed to have an understanding of both the formal and informal power structures in within a society. There was one influential person somebody very high up in the society, who is a critic of open science, and I didn't understand how just that one person could could actually derail my work. So this year, things came to a head at the council meeting where we had agenda items regarding top status for for the journal, and also whether I would be allowed to continue open science badges, which had just just recently begun. It was a contentious meeting. A lot of issues around data transparency statements people made points that those articles that said that data were available were going to be seen as more important for better work than those that don't have data that's available, and then there's very good reasons for not having data available as I certainly agree. In the end, the non signatory status on the top standards was upheld. I was also directed to retracted editorial I wrote on transparency and openness as kind of my introduction as as the editor badges were upheld by a single vote. The lesson I learned here was a lot of people in the field just really don't know in my own field and this this field would broadly be, you know, social psychology, a lot of people in the field simply don't know much about open science, and they kind of perceive as zealous, you know, everything is all or nothing. Lots of concerns about open data requirements, as I mentioned, but also lots of misunderstanding and basics. People didn't know what open access was versus open science versus open data, all of these things were mixed up in people's head so I think that there was an educational component that I that I really needed to deliver more on. In the third year I worked with the publication committee to revise the data ability data availability statements as directed by the council. This really just ended up in a situation where we changed, we require people to disclose data availability availability availability to language around it being encouraged. The publication committee wanted to let sleeping dogs lie on the retraction they said you know nobody resetted those editorials anyway. And then very interestingly, a few months later, widely publishers announced a new policy for all of their journals. All papers would be required to include an open data statement so this issue that had been at the core of what I had been fighting for, had now been completely resolved. They did how now it had now become incredibly mainstream. So the big lesson that I took from there is that it's really important to be patient, a change at high levels can be slow. They're often well behind where we want them to be, particularly the people who attend a conference and a conference called meta science, you know they're behind what we want. And it does catch up and I saw another example of this recently you know APA American Psychological Association became a signatory on the top standards. And that's that's a real dinosaur organization that's not that's that's that's come around finally. So, where have we ended up. Well, comparing pre 2000 to 2000 and pre 2020 to 2020. And later, because this is about the time where the where badges became something that was available for the whole year. The median sample size increase substantially from 64 to 304 a lot of that is my own focus on statistical power I'm sure. And adding open science badges about half of the articles that we publish have received at least one badge. We're trending closer to 60 70% for for for more recent submissions, we've been able to introduce registered reports, and our submissions have increased dramatically. We went from an average of 73 a year to now we're actually these numbers are inaccurate even up as of today, we've got about an average of about 200 per year. And of course 2021 is not not even done yet so there's been a big jump in in how much and how much how many submissions we're getting anecdotally some authors have told me that they selected us over other relevant journals because we were the only ones that promoted open science. What happened to my term, my term was about to end but I was I was extended for two years on the urging of that. Most negative about open science I really become a champion of my work. Again, change is slow, be patient, listen to those opposite people in opposition and talk to each other understand your each other's views like I did with the, the, the anti open science person. We found that we agreed on almost everything. There were just very, very, very minor differences. So, really do try to keep that dialogue open. Okay, thank you all very much. Thank you, Chris. That was fantastic. I will now move on to our next speaker who is shocker on and them shock is a PhD candidate in the Department of Psychiatry at Trinity College Dublin, giving a talk title cross cultural scale validation. Ready when you are. Thanks a lot. Yeah, I'm just going to share my video now. So, okay, please let me know if you can't hear anything. Imagine a bright PhD student starting off her first year. She's just finished master's degree that involved the cross cultural use of multiple scales. And now she's eager to spring into the exhilarating world of clinical scale validation. Little does she know she will end up questioning not only the very purpose and utility of the tool she's meant to produce, but also the foundations of the science to which she has devoted her entire career. I'm shock. Yeah, a third year PhD candidate at Trinity College Dublin, and the student I described was of course me. Over the past few months, I faced some challenges while designing a cross cultural validation of a trauma related shame scale. While many of these have been logistical such as keeping my PhD timeline in mind, finding collaborators in Asia and Europe, etc. There have also been some conceptual and methodological challenges to my perspective on conducting cross cultural research and on recruiting from non weird populations. So in the spirit of igniting a conversation on this topic, here are some of the issues and questions that I wish I had grappled with earlier on in the process of my own study. Sometimes research doesn't start out as cross cultural. This was the case with my study. Once the seed of having more diverse samples was planted, I let it lay dormant while preparing my study for an Irish population. In doing so, I missed out on having the input of collaborators in India, for example, from the get go. This input would have been advantageous in a few ways. Firstly, involving researchers from local target populations from the idea stage fosters a fairer relationship between PIs and collaborators, reducing the underlying power imbalance between PIs from largely Western resource rich institutions and researchers from non weird or not so well funded ones. Decisions about level of involvement, potential compensation and authorship can be made more respectfully in this way. Secondly, crucial choices in the study's design, such as ethical considerations, adapting scale items to the local cultural context, etc, should really be understood by PIs at a meaningful level. This requires time to learn about the cultural environment they will be entering and is where early involvement of local collaborators is key. I mean, imagine suddenly going into a community barely been exposed to and then hoping that they'll trust that you understand their culture. She doesn't even go here. Do you even go to this school? No. Ironically, if you're not familiar with early 2000s American pop culture, you might not understand why that clip was funny. Okay, so it's not possible for a scale to be valid in every subgroup of every cultural group you sample. But that's no reason to not try and improve. Even keeping the constraints of individual researchers in mind, there's always more we can do to get a slightly better diverse sample in cross cultural research. The reliance on convenience sampling in so-called non weird countries has resulted in many studies sampling subpopulations that have weird trades anyway. Think schools, universities and so on. These are educated, often westernized populations who are not part of minoritized groups. So can they really be considered a significantly different sample than, for example, the white Dutch population of the original study? I'm exaggerating, of course, but it's worth putting in the effort to recruit from a wider range of people in a region and not just the easiest to access population if at all possible. I'd also like to direct you to this talk on the validity of the term weird given by Sakshi Kai and colleagues at the CIPS 2021 conference. On the other hand, if there's little chance that a scale will be adopted within a community in the long term, at least in its current form, then maybe the most ethical course of action is recruiting from populations that have the most likelihood of using a validated scale later on. In other words, the weird trade populations I mentioned earlier.  However, given the sensitivity of the topic, particularly in small insular rural areas and the lack of psychological services there, it's pretty unlikely that participants will end up benefiting from my study. Perhaps the most crucial point I want to bring up is how do we discuss the utility and post-study uptake of cross-culturally validated material? Well, when it comes to scale validation, we need to consider some sub-questions. What is the end goal of your validation? Is it to strictly statistically test or confirm the validity of a particular scale by replicating the original study? Or do you also aim to encourage the uptake of a scale in a new range of populations? Although we may go into studies unconsciously assuming or hoping that the former will result in the latter, encouraging post-study adoption of a scale probably requires adjustments in how the study is carried out and almost inevitably in the properties of the scale itself. There's tons more that I couldn't include here, but if you want to continue the conversation, please consider following me at the T-searchers and consider following the Junior Researcher Program on Instagram and YouTube. Thank you so much for watching. Yeah, that's it for me. Thanks a lot. Thanks, Shakya. I'll just ask you to stop sharing the screen. Thanks. I did. Okay. Is it still sharing? No, I think it's fine, actually. Up next, we have Yuching Kai. Yuching is a Masters of Research student in Developmental Neuroscience and Psychopathology at UCAL and Yale University. Yuching is presenting a talk titled, Assessing Flexibility in the Measurement of Socioeconomic Status and Meta Research. Over to you, Yuching. Thank you. Hi, everyone. My name is Yuching Kai and today I'm going to present this research about the measurement flexibility of social coming status, which is by me and through other collaborators. So it is a matter of research. So what is social coming status? Social coming status or SCS is social standing or the class of individual or group of individuals. So it can represent for the accessibility of the resources for different units of individuals. SCS has been widely adopted in many different domains of studies. For example, in psychology or cognitive science, SCS has been found to be associated with many different outcome variables, including mental health, physical health, language development, and brain development of the children. You may think that social coming status is a quite straightforward concept, but the measurement of it is quite complicated and flexible. To begin with, we can use different indicators or resources to measure for SCS. So the most commonly used ones are education income and occupation, but you can also use not so conventional ones like political resources and subjective SCS. Even if we choose the same kind of indicator, the scoring of them can also be different. So for example, when measuring education, we can use levels of education or years of education. That is a categorical variable or continuous variable. Another thing to consider is whether to aggregate different indicators into a composite score. So for example, Honest Head Index is a very popular aggregated SCS score, which combine education, income, and occupation together. And all three of those indicators can be used as single indicator in other research. At last, we also need to consider what is the level of measurement for SCS in the study. So for example, individual SCS, parental and family SCS are the three of the most commonly used ones, but on a more extensive level, we can also measure for neighborhood SCS. For example, here is the neighborhood SCS of New Haven, we are currently living. On a higher level, we can also measure for things like gross national income of a whole country or region. So the current study wants to evaluate the flexibility of SCS of its effect on the results of cognitive neuroscience, specifically. So we first systematically review different ways of measuring SCS in this specific domain, and then we want to reproduce them using two public data sets, that is the PS from China and PSID from the US. Then we want to evaluate the impact of the flexibility of measurement on the possible outcomes in psychology and cognitive neuroscience. So we first did search, we'll first search the article and select the relevant ones, and the ones that can be potentially reproduced. So we use variables from CFPS and PSID to reproduce those different types of SCS. Then we evaluate the influence of the flexibility using the variance that can be explained due to the measurement itself using the index of ICC. And we also calculated for the associations between outcomes and SCS and also between different types of SCS. So the preliminary results has been pre-registered on OSF, and if you're interested in this study, you can take a look at it. So in this part of the analysis, we selected 53 papers, which used the 38 data sets, and we found that there are more than 40 different types of SCS, which is even larger than the number of data sets in this part of the analysis. About 20 to 30% of the variance can be explained by the measurement itself, and the correlation between SCS and targeted variables, and also between different SCS also vary to a good deal. So here is the correlation matrix of the different SCS, different types of SCS that is calculated from CFPS and PSID, and you can see that the number varies to a good deal. So what can we imply from the current result? As you may already know that the measurement issue in psychology has been discussed a lot recently. So for example, for depression and self-regulation, there are many different ways to measure for those concepts. Similarly, for social economic status, the flexibility of measurement has been found in the current study in the domain of cognitive neuroscience. So this could be explained by the complexity of SCS itself, but it can be measured by different indicators, but there are also problems in studies using SCS as a variable. So many studies arbitrarily choose different indicators, and they do not have citation or explanation over when, how they choose those indicators. And this could have an effect on the reliability and reproducibility of the findings in the cognitive neuroscience domain. That's all. Thank you for your attention. Thank you, Eugene. Up next, we will have a video that I will play for you on behalf of Hassan Khan, Hassan's research assistant at Ottawa Hospital Research Institute, and has recorded his talk for us on has open science penetrated academic hiring practices. Just bear with me for a minute while I share the video with you. Hello, everyone. Thank you to all who are attending today. My name is Hassan Khan, and today I will be talking about whether open science has penetrated academic hiring practices. Before I jump into my presentation, I just like to give a background about myself. I recently completed undergraduate degree in psychology from Carlton University. And for the past year I've been volunteering at the Ottawa Hospital Research Institute at the Center of Generalology. As many of you know the lack of transparent and reproducible research continues to be a worrying trend among the scientific community. For example, in the survey conducted in 2016, they found that more than half of the 15 hundred scientists reported being unsuccessful in reproducing their own research findings, which is quite concerning. Not surprisingly, these researchers believe that most often they're not the reason for the lack of reproducibility was due to the pressure to publish, as well as selective reporting. These factors are grounded in institutional practices, which continues to emphasize traditional methods for promotion and tenure, such as grant funding or the number of publications. Aside from that, the increased burden of bureaucracy often takes away time from doing and designing research. Now what can we do at the institutional level, we can shift our focus from a traditional method of assessment to an artificial method of assessment such as adopting open science practices to assess researchers. Although there is no agreed upon definition of open science, there are essentially a set of practices that look to promote transparency and credibility of scientific research. This can include registration of study protocol to publishing and open access journals, using preprints and making your study data publicly available. To promote this culture shift we have to get a sense of institution standard in terms of adopting open science practices when it comes to hiring faculty. So the purpose of this study was to evaluate the current hiring practices of academic institutions around the world, with regard to the mention of open science and research based faculty and postdoctoral positions. We conducted a cross sectional study of 192 institutions globally and gathered job postings for the past 30 to 60 days starting in February 2021. Our search strategy included obtaining job postings from the institution's career website, as well as any viable job boards. Job postings were assessed by the modified open science modular scheme. This is a self certification scheme that was made publicly available to us on the open science framework and is actually modeled off the transparency openness and promotion guidelines. Each reviewer assigned a level between zero to three to each job advertisement, a level of zero would indicate that the institution made no mention of open science practices in their job description, whereas a level three would indicate that the institution mentioned open side practices in the job description, as well as committed to including a proven track record of open science as an essential characteristic. So what did we find after examining 305 job advertisements in academic positions in 91 institutions, surprisingly only two had any specific mention of open science in their job advertisements. So it's clear that the institutions need to do more to promote open science if we are to deal with the reproducibility crisis, and there are some ways that they can address this. For one, they can make a commitment to open science in their job advertisements. They can provide examples of how open science is being promoted. They can share educational outputs. Last but not least, they can ask applicants to share how they have used open science at their professional capacity, and how they will continue to promote it going forward, if hired. I would just like to acknowledge the following collaborators for their ongoing support. Dr. David Moore, Burrida Franco, and Elab Malbole. Thank you all for attending today.