Hello everybody. My name is Barry Lynn and I run the Open Markets program here at New America and just wanted to start off by thanking you all for coming out today and braving the giant snowdrift to get here. And we kind of have a pretty packed agenda, so I'm just gonna try and start more or less on time, which is not always what we do in New America. So anyway, I'm just gonna have a couple of brief comments and then I'm gonna turn it over to Doug Preston. But, you know, in recent years, the reason we're here today is because in recent years two technological revolutions have entirely transformed how books are produced and sold in America. The internet has made it far easier for readers to buy any physical book they want, even the most obscure. The e-book has readily cut the cost of producing and distributing books by in essence eliminating the physical book entirely. On the surface, these two revolutions are great news for both the reader and the author. There's more information, more readily available, more ways for the author and reader to interact directly. Yet behind this heartening evolution is something, we see something very different entirely. What we see is extreme and growing concentration of power in the American market for books. A generation ago, the four largest retailers together held less than 12% of book sales in the United States. Among publishers, the top 75% of book sales were distributed among 50 independent companies. Today, by contrast, we see a single company that dominates retailing entirely, controlling 50%, 60%, 70%, 80% of different market sectors. This same company is increasingly powerful force in publishing. This one company indeed dominates this vital marketplace of ideas to a degree that we have never before seen in the United States, not in television or radio, not in film or newspapers, not in telephony or telegraphy. Now, there are many people who say, many people who believe that Amazon uses its monopoly powers in ways that are good for the reader, that are good for society. They say that Amazon wields its power in ways that result in more books at lower prices. I believe we're going to hear some stories today that will throw that idea that cheaper is always better into some doubt. But even if that were so, even if Amazon truly has lowered prices and truly continues to lower prices, we must also deal with the fact that Amazon actively, aggressively manipulates the flow of books between the author and reader. That Amazon favors some books and some authors and disfavors others. That Amazon routinely interferes in the ability of individual citizens to have their say or to find their truth. This is not a theoretical problem. Amazon's manipulation of the flow of books across its platform is not something that Amazon might do in the future. It is, on the contrary, something that Amazon has done repeatedly in the past. Amazon did so most dramatically in its effort in 2014 to force the publisher has shed to accept its terms of service. In that fight, Amazon simply cut off the sale of hashet books to American readers and I think we'll hear more about that later. This was a blunt tool and it's not one that Amazon uses every day. But I do think we'll hear later today about some of the other ways that Amazon warps the flow of books and information in America. Ways that may be more subtle but are perhaps just as effective. So anyway, we're here today to discuss whether this manipulation of the interaction between the citizen as author and the citizen as reader, whether this emergence of a master intermediary mediator in the U.S. book market is a problem that requires the government to act. And if so, what might the government do? We have a really excellent collection of folks to discuss this today. We're going to start off with a presentation by Doug Preston. He's the head of Authors United. He's a journalist. He's the author of numerous books of fiction and nonfiction. We're going to follow it up with a discussion between the author Scott Darrow and Chris Leonard of New America. Scott is a lawyer and he's the author of many bestselling books and he's also the former president of the Authors Guild. He knows this issue quite well. We then have two panels. One of the panels is composed of both traditional authors and self-published authors. And the other is mainly made up of antitrust lawyers and experts in data and discrimination. And I'm not going to introduce those people yet. The moderators of those panels will do so. Well, we're going to also hear a short presentation to give us some historical perspective from the author and legal scholar Jeffrey Rosen, who's the executive director of the National Constitution Center in Philadelphia. Just a few quick points before I turn it over to Doug. First, this event is not sponsored by any private company. We have not accepted any money from any publisher. We have not accepted any money from any booksellers or association of booksellers. We are holding this discussion only because we think this is a topic of vital importance for the American public. Second, many of the people we asked to stand with us today chose not to do so. They chose to not to do so not because they disagree with us as to whether Amazon's monopoly is a problem. They chose not to be here because they were afraid to speak in public about their experiences with Amazon because they were afraid to speak their opinions to their fellow citizens. They were afraid that Amazon would retaliate against them or people dear to them. And third, as I think we'll hear from our last panel, the various problems posed by Amazon's concentration of power of the American market for books may be a lot easier for us to fix than might now seem the case. America's past offers many lessons about how to deal with similar problems. Two simple principles stand out. First is that when you have a monopoly network, a monopolized network, you do not allow the people that run that monopoly to vertically integrate into lines of business that are run by people who depend on them. It creates a clear, simple, and inescapable conflict of interest. In this case, bluntly, Amazon probably should not be allowed to be in the publishing business. Second, monopoly networks should always be kept neutral. They should never be allowed to discriminate, not in service, not in pricing, not in quality. Amazon, in this case, should be required to deliver the exact same terms to all comers. So there's a little bit, for more information about this, there's an article that was in the New York Times last summer, last July 12th, about some of our efforts. It's by David Straitfeld. A couple little points. Paul Eid, who from Freshfields and antitrust lawyer who was on our second panel, is not going to be able to make it because of the snow storm. And Mark Coker, who's on our first panel, he's in California, we're going to be skyping him in. This is going to be available for streaming. So whatever you're saying, there are people out there watching it. It's also going to be recorded and available for people to watch in the future. And anyway, now I want to turn it over to Doug Preston, who is the head of Authors United. Thank you, Barry, and thank you, everyone, for coming, braving the snow, which doesn't look too bad, but thanks for coming anyway. I founded Authors United. I'm a Haschette author, and I discovered that along with all the 3000 other Haschette authors that our books were being targeted by Amazon for punitive action as a way to try to intimidate our publisher into agreeing to its terms. And some of the things they did, they delayed delivery by two to four weeks. They refused to pre-sell the books. And also, I remember going to my site, because I'm an author, I'm obsessively going to my site to look at my own books. And this big window popped up and it said, instead of buying this book, wouldn't you rather buy a book by Lee Child just as good an author and it's a lot cheaper and you'll get it right away. So I called up Lee Child and I said, what the hell's going on? He said, I didn't do it. I didn't do it. But these are examples of Amazon's dirty tactics. So we formed Authors United. We had over a thousand authors, many Nobel laureates, Pulitzer Prize winners. We took out a two-page open letter in the New York Times to Jeff Bezos protesting his tactics. And eventually the dispute was settled. But the problem has not gone away. Right now, Amazon, according to Paul Krugman, the economist, Amazon has as large a market share in the entire book business as Standard Oil did in 1911 right before it was broken up into 34 companies. Now this would be concerning if the company in question, even if the company in question were benign, but we all know that Amazon is not a benign company. I have some statistics here that I hope are not damned lies, but Amazon now controls 75% of online sales of physical books, not e-books. 65% of e-book sales, more than 40% of new book sales and about 85% of e-book sales of self-published authors. To give you a small example of that, the largest publisher, physical publisher in the world, Penguin Random House sells 15,000 titles a year. Amazon uploads 500,000 titles a year to its website. It has as many titles as there are books in the Weidner Library at Harvard. That just shows you what kind of market share we're talking about. Now I have some statistics here that I just picked up. They haven't been published by the Codex Group and it shows that, and I'm only talking about physical books now, the retail share of physical books sold in bookstores has gone from 72% in 2010 to 33% today. And this statistical chart created by the Codex Group is entitled, Amazon and online share move to total domination. Amazon is a monopoly in absolutely every sense of the word. And so we wrote, Authors United, the Authors Guild and the American Book Association wrote a report or a letter to the Justice Department asking it to investigate Amazon's monopoly and the ways that we believe it has abused that monopoly power in search of a greater market share. And we submitted the letter in August. We had a very good meeting with the Justice Department in September. They listened to us. We don't know if they're going to do anything, but we hope they will. Now I have some more statistics for you. And this is only about physical books. Amazon completely dominates e-books. Physical books, $5.6 billion in book sales. That's equivalent to 3,600 bookstore equivalents with 40,000 bookstore jobs lost because of this. And Amazon has avoided collecting $625 million in state sales taxes in the half of the states that it doesn't have to collect sales taxes in. So this is a company that is, has dominated, is completely, has a monopolistic domination of the market. Now in our letter to the Justice Department, we argued two things. One was we had a lot of antitrust arguments about specific antitrust laws that Amazon has broken. And you're going to hear a lot about that later on from the lawyers. But the second part of our argument was the freedom of information problem. Our country has never before allowed a company, a single corporation to dominate a vital marketplace of ideas the way Amazon does today. And even before there was even antitrust law in 1866 when the first telegraph wires were being strung across the continent, the Congress leapt in and prevented Western Union from having a monopoly over the information transmitted over those wires, the first example of electronic communication in history. In the 20th century, the Congress passed many laws that prevented a consolidation of power in the newspaper industry, in radio, and in television. And most recently in Internet, for example, the merger between Time Warner cable and Comcast was blocked because it would have resulted in a 57% domination of the broadband cable market. But Amazon now has monopolistic control of what maybe argued or could be argued as the most important informational market we have, which are books. They have, as Barry mentioned, they are also engaging in content control. They are interfering, we believe, already with a free flow of information of ideas in our society. And this is something that's very concerning because the biggest ideas, the most complex ideas, are first introduced in books usually. Thomas Paine, for example, starting with Thomas Paine and going on to the present day, some of the most important parts of our democratic discourse have been in books. But because of Amazon's market behavior, because of its sucking of money out of the publishing industry to itself, many books are not being published that would have been published because they are too risky for publishers. Publishers are now only publishing books by, or not only, but they're really favoring now books by bestselling authors and celebrities. They're not taking risks with unproven authors as much as they did. They're not taking risks with unpopular or quirky ideas. And as a result, since 2009, authors have seen their income, the average author has seen his or her income decline by 30%. And these are authors who have been in the business for 15 years. And a lot of that can be traced to Amazon. So that really is, I've thrown a lot of statistics at you. Thank you for listening, and we appreciate you all being here. Thank you.


The conversion of natural peat forests to plantations in Indonesian peatlands has severe impacts on hydrology, leading to amplified moisture stress and drought severity. This article was authored by M. Torfik, B. Minasini, A. B. McBratney, and others.


Hi everyone, Joe for Jaspy's casewrakes.com with another one box break of 2021 Upper Decks Skybox Metal Universe Champions, random letter break number five. We're going by the first letter of their first name. So if you end up with T, yeah, let me get all the Tiger Woodses. All card ship as well. And if you're one of the first 10 spots purchased in this break, got a chance at a couple extra spots. So here's the first 10 right here. Thank you for being early birds. Let's see if that results in a couple hits, couple extra spots for somebody. Two and a three, five times. Top two after five, one, two, three, four, and good luck fifth and final time. It's gonna be Ryan and Royle. Extra spots going your way after five.  You're still in the action. All right now, let's assign you a letter. So there are the letters. There's a new list for your names. And let's roll it and randomize it. Two and a one, three times. One, two, and three. Two and a one, three times for the letters. One, two, and three. First letter of their first name. We've got P down to EG. All right, James with P, Royle with N and R, Ryan with Z and any other letter not on this list and possibly any non-sports stuff if possible. I don't think we're gonna run into that situation, but if there's a card, like if it's like dinosaur, like it would go to you. Chino with D, Eric with J, Pichette with W and S, World Series, Royle with M, Dale, H, V, Lane, Ryan, A, Chino T, Eric's, K, C, Kansas City, Dale with B, Raymond L and Brian EG. Let's alphabetize by a name and we're gonna pause the video. When we come back, we're gonna see if there's any letter trades and then the break. Stick around, B-R-B.


This paper proposes a novel model of the neurovascular coupling which explains several important features of the coupling including the linear and non-linear regimes, the negative vascular response, undershoot, and the emergence of a calcium-driven hemodynamic response function. The model also shows how information is encoded and transmitted from the neuron to the vasculature, and how frequency modulation of astrocytic calcium dynamics plays a key role in this process. Additionally, the model links neuronal activity to the bold signal, allowing for the detection of neurodegenerative diseases based on changes in the neurovascular coupling. This article was authored by Federico Tesla, Margelina Lin, and Alain Distecx.


Some of those Sam's stuff order has not been tipping on the load.  On the load load, it ain't been tipping. Steady K and I'm back today with another Jordans video. Huge shout out to everybody tuning in. If you want to subscribe, make sure to smash that subscribe button. Give me a thumbs up and do not skip those ads. Today is March 23rd and we already get to this back. Look, it's raining outside. You guys can see it's very little people out here, baby. Very few people. I got the raincoat on. Again, I had to get the raincoat on, man. So I can get rain done, baby. But I hope you guys enjoyed the video. Also, don't make no excuses, baby. I never hear about, oh my God, it's raining. Oh my God, it's snowing. No, nobody.  So if we ain't making excuses, you shouldn't make none either. Order number one and two, what to do? Order number one, it's all the way sparked up 8.8 miles for 2117. We about to get this green. Order number two, what to do? Instacart order 41 items, 2.1 miles for 3627. Thank God for his blessings. I hope you guys are out there. I hope you're getting stressed, baby. I hope you guys enjoyed the video. Also, make sure to smash that subscribe button. Give me a thumbs up and do not skip those ads. We about to run up this bag. So, order number three. Get with me, baby. Are we still here, don't y'all? Walmart 13 items, 5 miles for 1150. Get with me. I'm going to drop off this wall. Make it the order, baby. I'm actually very glad she answered the phone, because guess what? They don't have the address on the app. It just says the street. That's it. That's it. That's all. So I'm glad she asked, because if not, I would have had to go through a lot of BS just to drop this order off. I'm talking outside. So that's good to ask, baby. I ain't going to have to call support and all that stuff, but that's the word. That's the last thing I'm going to do is call support.  I'm talking outside. Okay, there we go. Then with that photo of a jar, I'm going to smack, smack, smack. There we go. Let's go. We are here, baby. We got a lot of gallons of water. We're going to put this right here. I think I got a baby.  I don't know what kind of water baby is the truth. The steel? I don't know. I don't got no kids. I don't know. We'll have to come back for another trip. Grab all these. Let's go. Get this morning, baby. They got a big gallon. So I'm going to line the gallons up on the side right here. Boom, boom, boom. Sit this right here for now. Until we get the gallons situated. Last trip. Make it rip, baby. Get this photo. Dude, we did it. Order four and five. Stay alive. Let's drive. Order number four has always worked up two orders. 7.8 miles for 1708. It's time to skate. Order number five is Walmart. You see we at Walmart, baby. Walmart 18 items, 3.7 miles for 7.25. Stay alive. Let's drive. Order six and seven. Thank God for his blessings. Order number six is Panera braid two items. 1.3 miles for five. 25. Stay alive. Order number seven is all the way sparked up. One order. Sam's Club 5.3 miles for 1162. What'd it do? Now you old news. Got a couple of old schools. Let's see if they got it. Yeah, that's what I'm talking about, baby. By last time the dude had to come open the door for me. Because it's closed. And it's the same person always orders before it opens. So I don't know if that dude are here. He's gonna meet me in the park right on the app. But I don't see nobody. I don't see nobody. I don't see nobody. I don't see nobody. I'm gonna have to call them. The last time dude came up here. Hey, how you doing, Shanique? I'm outside with your order. All right, no problem. I'm gonna have a VIP. Here you go. You have a good one. Order number eight. We are here getting to this cake. Two eyes gone. Three orders. 47 items. 18.4 miles for 43. 99. We on the grind to the end of town. Y'all gonna see it, Shan. Let's go. So guess what? It is now 947. And they would not let us get the all the way sparked up order yet. So guess what I'm about to do? I'm about to go on here and try to get it early. 10 o'clock, I'm gonna log in so they can get busy. I'm gonna drop this order off and try to hurry up and get back. So we can get all the way sparked up again. Because they don't let us clock in early for the Sam's Club order for some reason. But we ain't complaining. We ain't talking about to get this money. Let's go. We're here baby. I bet you they got us together already. They didn't get this order and we gotta get busy. Hello. I'm back to vlog you guys. Again like always. But I'm gonna have a pick up for you. And I was wondering if I can get it early. Okay. Give me just a second. All right. Thank you. Huh? You didn't check in right? No, they won't let us check in until 10 o'clock.  Guess what baby? We are here. So we'll get all the way sparked up order. This is not a very good landing right here to put this box on. So I'm gonna have to put it like... Let's just put it right here. Step back. Oh, I need to step back. Hit him with that bottle. Hit him with that bottle. Knock on that bottle. Let's... Order number nine. We on the grind to the end of time. Y'all about to see us shine, baby. Order number nine is Pnebra at 3.5 miles for 8.11. Thank God for his blessing. We get money when he's stressing. Let's get busy. Nothing got us together. I see one order up there. I don't know if it's ours. Yes, it is. That's what I'm talking about. Get this. Get that. We got it. We are here to talk about this Pnebra. My question to you is are you getting to this brand, baby? Are you going to get this money? Are you going to get to it? Hit him with that photo. That's my smile. That's my smile. Bye. Let's go. Let's get it done. Last one is Pnebra at 7.4 miles for 17.47. Thank God for his blessings. We are already here, baby. But don't get these orders. We don't get busy. I hope you guys are with me. I hope you guys are enjoying the video. Make sure you smash that subscribe button. Give me a thumbs up and you are not skipping those ads. We're going to continue to get to this bag. Let's go. Back at it again with another Pnebra order, baby. We're liking some orders up here. I hope they got us together. Drop the feet, baby. Let's see. That's our tour. Got one right here. Bang. Bang. That's there's two. That's one. Oh, let me see. Hold on right here. We are here, baby. We are here to drop off order number two. What do they do? They take it to the back door. So that's exactly what we're going to do. They throw the yard back here, too. They're fixing up that portion of the thing. Let's talk it like it's hot. Hit him with that photo. Smash that smile. It's over, baby. We came. We saw me conquer. We made 201-17. We got to that green, baby. I hope you don't go to the video. Guess what? Wow. 11-14, baby. I mean, what else y'all want us to do? If we can get it down to two hours, I look. Why do we always want to take that, too? Oh, guess what? For order number one, we got to do it. For order number one, instead of 11-62, what to do? We got 36-93. They get a great tip. That's what I'm talking about. Tip buzz, baby. I've been noticing some of those SIM stuff order has not been tipping on the low.  No, they ain't been tipping. But either way, go win, complain them. We maintain it. And we do not thank it. I hope you guys are too. I'll see you guys tomorrow. We going back at it. Hey, it's back season all 2022. What to do? I'll see you guys on the next one. We going. We going. We throwing. You already know the DDK, and I'm on my way.


Thank you very much. So welcome everyone and I really appreciate you spending some of your Saturday here with us. I can talk about stream analytics all day long, but we won't today. We'll keep it to 40 minutes. And actually what I'm going to do in this session is actually two parts. The first part is part of a session that was run at San Francisco Next, where the product manager for Dataflow Sergey went through some of the new features. And then for the next part, we're going to actually run through a full end-to-end demo that was available at the San Francisco Next demo booths. So it is a bit tight on time, but we'll see how we go. Okay, so starting with stream analytics. What is stream analytics? For me, it is the continuous processing of data without having to materialize all of it first into a batch format. Now there's three, it's a little bit of an over-certification, but there are three broad categories of processing that I consider. The first one is transport. That is just taking a stream of data that's arriving, for example, from IoT devices to files or to your data warehouse, for example, BigQuery. The next category is where you're actually doing some enrichment of that stream. So imagine you have a point-of-sale system on your stores, it's sending information as you're making sales. With that, you have a product ID. And in the stream, you join that product ID with the product name and then land that denormalized value into your tables. The final piece, which is a bit that I get very passionate about, is doing in-stream analytics. So that's where you're actually doing aggregations and computations, statistical functions on the stream as it's happening and connecting that to the different parts of the business. So with those three categories in mind, what we're going to do is first go through the options that you have to build out a typical streaming pipeline within GCP. First of all, let's think about the data points. So we mentioned IoT devices already. And by that, imagine a factory that's full of sensors and every machine is sending information continuously about its state. The other one that is very common is clickstream. And by clickstream, we mean the events that are happening. One example, users are on an application in a mobile and they are clicking through various things like add to cart, etc. All of these are events that are clickstream that's being sent. And of course, the same will be true for web applications. So if you imagine this continuous stream of data flowing through, the first thing we need to do is absorb this data in some way. And some of the common tools, so Apache Kafka would be a choice, as well as Google Cloud Pub-Sub, which is a fully managed publish and subscribe API. We'll talk more about that a little bit later. Once you've absorbed this data, you then need to do some of the processing. So the three categories I described. Now to do that, the way that we're going to describe through it today, we're going to actually use Apache Beam as the programming model to describe the pipeline, describe the transformations that we want to do on that stream. And then we're going to make use of data flow as the execution engine for the pipeline we designed. We can make other choices as well. We could use Apache Flink or Spark as well to run that programming model. And we'll go through that in more detail as well. Downstream of this, we now need to absorb the data. Quite often, this will be into a data warehouse like BigQuery. Depending on some use cases, you may actually want to split deliver what the results. In other words, you may want to put it into your data warehouse. But you may also want to put it into something like Cloud Bigtable. One example, for example, would be Time Series, where we want the Time Series to go to BigQuery. But there are use cases where we want that Time Series data to be available in a key value store. Beyond that, obviously, there's then the visualization and the analysis you can do with that data. So let's go through some of these components. I'll pick the Google Cloud Platform ones here. So we have Google Cloud Pupsup. This is a global publish and subscribe API. It is serverless. In other words, all I need to do to set one up is to create a topic within the console. I don't need to do anything else. At that point, you can start sending messages into it. It will auto-scale. It will do all of the things that it needs to do to deal with the load coming in from day one. So whether it's 100 messages a second you need to send to it, or hundreds of thousands of messages a second. It can also act as a shock absorber for our system, because it will keep data for up to seven days. In other words, if there's any issues downstream in the processing, this absorbed data will stay on Pupsup for up to seven days for us to be able to pick up later. The final piece to think about with Pupsup, which I think is interesting is, we use Pupsup often as the ingestion, the first layer for this stream of data that's coming. But you can also use Pupsup as a glue between your data analysis systems and other departments and different parts of your business. For example, you can send alerts and stuff and use Pupsup as a way of sending those messages to those other systems. We'll actually go through that a little bit in the demo. In terms of Apache Beam and Dataflow, I'm going to talk about Apache Beam first. Apache Beam open source project about three years ago, Dataflow SDK, was donated to Apache Foundation to begin that project. Since then, we've had a really vibrant community. In 2018, I believe it was in the top three of a number of commits for an Apache project. It was also one of the most active in the dev lists. Actually, if you like technical content, there's some really senior Google engineers who work on this. They do a lot of really good technical, interesting discussions on the dev list. If you have a cup of coffee and a Saturday morning to spare, it's worth reading through those. In terms of what Beam does, it is a programming model that allows us to describe the pipeline. It works for both batch and stream. It has specific primitives that allows us to deal with some of the complexity when you want accurate below latency results from stream data. It comes with multiple runners and language. I'm going to go through that in a little bit more detail. This is one of the things folks like about Beam. The vision is to eventually have many languages that the users can use. Java and Python support it at the moment. There are more languages coming. Ultimately, for users to be able to pick the language of their choice and be able to design and build their pipelines. Then the other part is around the choice for the execution engine. Once you've built your pipeline, you can choose, for example, Apache Flink to be able to run and execute the processing or Google Cloud Dataflow and also the Apache Spark. The one we're going to talk about more today is Google Cloud Dataflow. Dataflow is a runner where you can run the Beam pipeline onto. It's serverless and fully managed. It has exactly one semantics, which is very important when you want accurate stream processing. It actually deals with all of the things like windows, state storage, shuffling for you. As a developer, there's a lot of stuff that it does for me that I don't need to care about anymore.  I will design the pipeline and then submit the job. At that point, I don't need to do anything else. I just wait for my results to show up. Underneath the covers, it's actually doing a lot of things. Just to give you a 30,000-foot view of what it does, I'll just walk through what are the things that happen once I submit my job. I type my code. I build my pipeline. I press P run. I go. It submits the job. At that point, it will schedule the work. It will spin up workers. It will then deploy my code to all of those workers. It will also deploy the code that it needs to be able to do things like shuffle and windowed state to those workers. It will auto-scale those workers to the right level, so it will increase them and decrease them based on the load that's coming in. It will take the inputs that I have, for example, if I've given it several streams and a database to read from. It will shard that data, move it to all of the workers. If the worker fails, it will move the work off that worker and put it to another worker. If there is too much work on a single worker, it can actually migrate some of the work to other areas. All of these things are happening, including niceties like when I've written log.info or log.error messages in my pipeline. It will push all of that to a central location from all workers. That location is for us Stackdriver, so I can actually go to Stackdriver and actually see all of the logs of what's happening on my pipeline. All of that is being taken care of. All I did is set p.run execute. I don't need to care about any of the stuff that's happening underneath the covers for me. When it first came out, it did have a very steep learning curve, and we've been continuously trying to improve that. Obviously, that kind of job is never done. There's always something else you can do to make it easier. Some of the things that folks have appreciated has been around a couple of items. One is the creation of a job UI, and the other one is SQL, which Jan's mentioned at the start, in terms of the job UI. If you recall the three categories that I mentioned of processing, one of them is transport, where you're just taking a stream or some content and moving it somewhere else. You're not doing any transformations. You're not doing any enrichment or calculation. Now, to have to write code to do that is kind of boring. It's tedious work, and that's why we have some templates that allow you, without code, to actually be able to submit a job just by going to a forum and filling out the details. There's some very common templates for common things that people want to move. For example, it pops up to BigQuery, and we'll show a quick demo of this in a moment. That makes life easier. One of the languages that is now supported in Beam is SQL, obviously a very well-used and well-liked language. It does things like... I'm a Java person, and if I needed to do a join in Beam, I need to write all that out, which is pretty painful. If you were doing it in Scala, it's a little bit less. Well, that's what we like to write. It's a simple join. We just want three lines of code. Let's keep it at that. With SQL support, we have the ability to write SQL within the pipeline. One of the things that was announced, it's currently Alpha, is the ability to write this SQL in the BigQuery user interface itself. Rather than choosing BigQuery, you can change to Dataflow, and it will submit the job for you. I have a show of hands. Who used BigQuery here before? Okay, a few folks. BigQuery is a good, petabyte-scale data warehouse in the cloud. Actually, I'm going to go through a quick demo of that. What we're going to do is we have sales information coming from a point of sale system. It is going to a PubSub topic. What I want to do is do the second type of work, which is enrich that stream. I want to take that information so that I can have the store, city, and location added to that piece of data for downstream processing. To do that, let's go over to... One of the things we can see here is in this mid-center pane is where I can actually normally type my SQL. I type my SQL, and if I was to hit Run, I would be using one of the datasets available in BigQuery.  To do that, I'm going to go to more query settings and switch the query engine from BigQuery to Google Cloud Dataflow. This will make some subtle changes to the UI. Now, I'm sure you don't want to watch me type a lot of SQL, so I'm going to cut and paste the SQL I prepared earlier. What we have different now with this UI is if I go to Add Data, in the previous BigQuery format you would only see the datasets that are available to your BigQuery. Here, we actually have a new one which is Cloud Dataflow Sources. In the Alpha, that is going to be PubSub. Now we have a stream as a source, and I could do a search, but I've actually set this up for us already. We can also see there's now Cloud Dataflow Sources available within the UI. If I look at Sales Per Store, we can see there is a schema attached. In that topic, I'm sending JSON messages, and I have defined the JSON schema in a YAML file and uploaded it to the system and said, this is what the JSON looks like. Now the system knows what it's going to get when it looks at that stream, and it's going to be able to use it within the SQL statement, including validating my SQL statement. So here, the SQL statement is making use of PubSub topic and joining that with BigQuery table. And the SQL statement is valid. I'm going to create Cloud Dataflow Job. Now I'm just going to hit next on this. It allows me to say where I want the job to, the table that is the output of this to be sent to. So let's just call it demo set and leave it at this. Now this will actually take a little while. It takes three, four minutes in alpha to actually get there. To save us time, I actually started one this morning. This is the Dataflow front page UI, which shows all of my jobs. The gray one is the one I just submitted, so that's queued up. That's going to take a few minutes to get started. I started one yesterday, which is this DF SQL inventory. So that's the monitoring interface with Dataflow. We can actually see the Dataflow job in the middle and some stats metadata on the side. It looks simple, but actually if I click on this expand, we're actually seeing what was interpreted from the SQL to the Dataflow pipeline. So what it's done is translated that SQL statement that I had into a pipeline DAG, a directly cycle graph, and this is that interpretation of what it's doing under the cover. So here we can actually see it's reading from up and further on down before the join. We can actually see the read from BigQuery. So that has allowed me to build a pipeline against the stream and the table in BigQuery, all without coming out of a user interface allowed to write SQL. Okay, so let's now move back to our slides. The next part is general availability of Dataflow, the stream scaling. So I described earlier all of the steps that happened underneath the covers for me when Dataflow does my work when I submit a job. It spins up those workers, it puts my user code in, it puts the Dataflow code in for things like shuffling and things like the window state. Now the new service, by the way, which is optional, you do not have to use this service, takes a large chunk of the stuff that's being, that's related to the Dataflow service away from the workers, leaving just my user code to be running those workers, and takes them into the streaming engine. So here there's some very direct benefits to a user who wants to make use of this. First of all, we have a much better idea in terms of utilization because it's now just mostly user code in those workers. So our auto scaling becomes much smoother. And there's a blog on the site around this which has some graphs that show some of that effect. It's better supportability. So before you needed to run an update on your Dataflow pipeline to get new features around these parts, the shuffling and the streaming, now because it's all in the service here, this is transparent to your pipeline. So you don't need to update it yourselves for us to get that benefit. And finally, there's less usage on the, there's less resource usage on the workers. So we're getting much more out of those CPUs. Now this service does, it is optional and there is a cost associated with it per gigabyte. It's currently available in four regions. Okay, so now I'm going to move over to an end to end demo. So I think if we're in a restaurant, this will be the point where they give us some glass of water to refresh our pellets. And we go straight into demo mode. Okay, for this demo, we are going to use a retail company as our sort of demo environment. And let's imagine we have some, a retail company with many stores and the VP of sales has asked us to improve the sales process specifically around couponing. So today her results of sales from stores comes at the end of the day or the next day because everything is batch processed. She wants real time data so that she can make couponing decisions based on things that are actually happening. For example, a store has got too much inventory of fresh produce and it's going to start deteriorating and we need to get rid of it. So that would be one option or making sure people come to special events at the stores, things like that. The other part is around the targeted couponing. Now if I have, I'm going to go through the real time one. If we have time, we'll do the targeted couponing as well. That touches on some of the ML pieces. The general pipeline that we will use to do this. So the point of sale system, this is the scanning of the devices, what's in the basket. That from the stores is going to be sent to Cloud PubSub to absorb the data stream. Then we're going to do the processing. Now we're going to do three things with every element that we have. The first thing that we do is we're going to send it every minute to Google Cloud Storage for our data lake as just Avro files. So this is kind of like raw, untouched, untransformed data. We're just going to keep a copy there. The second thing we're going to do is we are going to, without windowing, stream it directly into BigQuery. So there we're going to use BigQuery's ability to absorb streamed data and update tables fresh for our sales VP. The final piece is where we're getting into that third category of stream processing where we're going to do some basic analysis on the stream. So we're going to count the number of sales of the produce for a store and send it back to PubSub. And the reason we're doing that is then PubSub becomes a way of connecting our data pipeline to the other parts of the business, for example, the inventory department. And so the inventory department knows up to the minute or up to the almost real time what is happening in the stores and can start scheduling deliveries and stuff as the need arises. Okay, so going on to the demo itself. Now as we mentioned, PubSub is a fully serverless API so to show you the creation of one, it would be a few clicks, not that interesting. What I'm going to show you is some of the monitoring statistics from using Stackdriver. And on the top left here we have the stats from my orders topic. And here we are seeing around 200 orders per second coming into the system. Now obviously our stores are super successful because if you do the math, 200 a second of sales is something like 10 million a day. So these are really, really big stores. But we've jumped numbers up to make it more interesting. So there's 200 orders and every order will have multiple lines. So every sale will have a basket of say 10 items or 6 items. So there's actually quite a decent volume of data. What's really interesting is the graph on the right. Now this graph shows how long every message is actually staying in PubSub before it's being processed. And we can actually see the median is one second. So all of those messages are coming in. They're taking about a second before we're pulling them off with Dataflow. Now Dataflow, there's some graphs here underneath around Dataflow's performance metrics. Rather than talk through those, I'm actually going to go and show you Dataflow again. So here we have the monitoring user interface. This area, this is the description of the pipe line. So I've written this in code and it has interpreted that into these series of transforms. As you'll notice, it is a DAG, so there's no cycles allowed. But here everything is flowing downwards and we have those three activities that we were talking about. So the first one on the far left is the writing to Avro. So we do a one minute window and then we push the files to Avro. So this is the raw untouched data. In the middle path, there is no windowing. There is no batching. There's none of those pieces. We actually just stream it directly into BigQuery. And that's going in at a couple of hundred a second. Obviously BigQuery and Dataflow can cope with much, much higher numbers into the hundred thousands. And on the right hand side, we're doing stream analytics. Here we are doing a filter on fresh produce. If I click on that, we can actually see since this started, there was 12 million orders processed by this line. And what I'm doing in that piece of code is I'm taking the order, I'm expanding it to all of the lines that it has. So all of the items in the basket, it's now a bigger piece. And then I'm filtering all elements that aren't fresh produce. So there's a massive reduction. So we're going from 12 million to 4 million. That's actually much higher if you think that 12 million isn't multiplied by the number of lines. So if there was 10 line items per order, that's 120 million that we reduced to 4 million. We then do a 10-minute window and we start counting. The reason we have to do a window is in order to do a count and average, you need to stop at some point. And we need to tell the system when to stop. What's important is I'm using event time rather than processing time for this. So it's not based off the wall clock time of 10 minutes. It's based off the events 10-minute time, a little bit, it would take us too long to cover that in detail for this session. Next, I actually output this data back into PubSub. So here we could imagine that down here somewhere there is our inventory department who is actually picking up this information and able to keep inventory fresh at our stores. OK, now let's look at the BigQuery path. Now I'm back into BigQuery and I'm going to be using BigQuery standard without the Dataflow engine. Here I'm going to be looking at a specific table which happens to be the order with lines table. So this table is the table that Dataflow is pushing information into. The schema for that table is the order and there is a repeated field. This repeated field is the order lines. In other words, it's a collection that we have set up. If I look at the details for the table, it's not particularly huge. It's got 340 gig worth of data and there's around half a billion rows of information. It is partitioned and more interestingly at the bottom we can actually see that the streaming buffer is active. In other words, this table is now getting constant updates to it from that streaming pipeline. I'm going to run a query here and the query is going to do some enrichment of the data and more importantly, I have put in where the time is less than 10 minutes ago. Let's run that query. Now BigQuery uses a column or IO storage structure which means if my select statement doesn't choose all of the columns, it's not going to have to process all of the columns. It will prune those off. Also, because I had that time constraint and it's a partition table, the system automatically ignores partitions that aren't required. In other words, all the other partitions that were in the schema, so instead of having to process all 300 gigs, we only had to process 70 meg. And here we can actually see I put in this column for convenience the current time 34606 and we can see that the West Palm Beach, which is one of the highest in number of sales in the last 10 minutes, had some updates there about three seconds ago. The other one, which is, I won't even, I pronounce things badly, so let's say Houston is four at 04. So this table has got fresh real-time data for all of the analysis. And I haven't had to set up a single system. I haven't installed anything. I've just written code for that entire pipeline. Now that we have that data, we've solved the real-time problem. How are we doing for time? Are we okay? Okay, so I'll do the couponing piece. Okay, so now I'm using Data Studio. And with Data Studio, we're actually able to visualize some of the information that's in BigQuery. I did open this page up just before the talk, so the numbers at the top are going to be slightly off. But here we have, you know, a visualization of the sales in the last 10 minutes from our stores. We had the facilities data. So this ties it into the city. So we now have zip code and, therefore, that long locations. We have loyalty cards. So as you go to the supermarket and buy things, if you use your loyalty card, that ties that sale to you as a user and it's your ID. So we have loyalty card information for those sales that they did provide that card. And using GIS functionality in BigQuery, we have another table that actually gives us the loyalty program total lifetime sales for me as a user to stores within 20 kilometers of me. And that functionality is basically distance and that long we can use GIS functions in BigQuery to compute that. Okay, so we've got lots of interesting data. We have some data on our stores and stuff and we have the live fresh data. This is useful as one of the data points to start building more interesting couponing. Next, let's see what should we use as the item that we do couponing against. So in our stores, globally across all sales forever, which was half a billion rows of information, the top 20 items that are in a basket tend to be fresh goods, not surprising. And in the US, actually, it tends to be bananas, organic bananas, et cetera, that they will purchase. So that's a very common purchase when people are going to the grocery store. In this one, we've got some geographic stuff and done the New York state. But essentially, this is an item now that we know that most people want in the store. And so it's useful that earlier on in that data flow, we were checking for fresh produce so that we know we've always got goods like bananas available as that's one of the common things people will come to our stores to buy. And maybe this is something that we can use in our couponing to bring people to the store. So one way we could do that is we could start exploring the data with SQL. And I'm not a data scientist. So let's start with basic SQL. Let's have a look at the number of, for a specific user in our loyalty program, what time of day they will generally make purchases. So if I run the query, actually, I ran it before anyway, but let's run it again. What we see here is a customer ID. And these tuples of afternoon, F afternoon, T, F for false, T for true. Essentially, this user has bought that produce, bananas, et cetera, in afternoon ones, not in this afternoon. So all of this data together gives me some idea of when that user will buy something within a day. But what if I then want to add the frequency, the location, the weather, all of these other factors. You can keep writing more and more complex SQL to try and do this. Or as one of our folks in one of my data scientists colleagues did, he just used linear, sorry, logistic regression to actually solve the problem. Now, with BigQuery, because it supports ML within BigQuery itself, we don't have to export the data to another system, run tests, do notebooks, et cetera. Within BigQuery itself, we can run the ML that we need. So here, we have the ability to write in SQL with these extensions, create or replace model, a logistic regression model. And the other thing that we're doing is we're doing the feature extraction using SQL. Again, very easy language for a lot of people to use. So therefore, it makes it very accessible. This will go off and build a model. I'm not going to run that now because it will take a few minutes. And essentially, at the bottom, we will end up with a model on the left-hand side here. This will buy banana model. And if we look at the details, we can actually see the number of iterations that that process ran and some results. So the training, and this is also available in BigQuery, you can actually see some of the graphs of the trainings. And also, when you're evaluating the model itself, we can tweak the score threshold. Again, I'm not a scientist, so I'm not going to attempt to explain all of these pieces. But essentially, we have our confusion matrix of when it got it right in terms of guessing whether that particular user is likely to buy that particular produce. Finally, we have bringing it all together. So we can run a query against that table to give us a prediction of folks who are likely to buy that produce. In this case, we're going to use some bounds of smaller than 60% and bigger than 50%. The reason smaller than 60%, if they're most likely going to buy it anyway, then why send them the coupon? And bigger than 50% is if there is a cost associated with sending the coupons, they want to make it for the right target audience. But that's entirely a business decision. I can run that and get some results, but then you can imagine I can now run this with the real-time fresh data. And I can actually start making couponing decisions within a day, including tying it back to what inventory we have at different stores. So all of this we were able to do because we were doing everything in stream pipelines. Going back now. And so thank you very much. And again, I appreciate you guys coming on on Saturday to spend some time with us. Thank you.


So first of all, I want to thank everybody for coming and I'm very happy to be here to share with you what we have done and the field have accomplished and what we can do for the future. So the reason I am kind of mentioned that this disease is in the past was very very bad very very lethal disease, one of the worst disease, but kidney cancer has become a disease that's much easier to manage and currently for the metastatic clear cell kidney cancer, they will talk about it most of patients are contracted with the metastatic disease in the past, the medium survival is about a year now it's almost three and then I think we we are moving towards that in the next couple years I think we will move it to almost five years with all the other trials including immunotherapy or complex, more compound targeted therapy. So I think this disease is actually I think it's a poster child of what we can do for cancer and not just kidney cancer I think hopefully we can actually share the success with other disease time and people can start to think about not just so afraid of cancer in general and really think about how we fight it and how we get rid of it. So my name is James and I work at Memorial Sloan-Catering. I speak not for myself, I always speak they present the group that I work with and I'm very fortunate to have many people working with me at Memorial Sloan-Catering it's a group approach in team science and that we really try to understand the disease better I'm a physician scientist I see patient half a day a week as well around the laboratory having to you know Dana-Farber, Washu, Hopkins but I never crossed the midline so I never just came here about four years ago and the reason I came is really I really want to deal with this disease and that the I think I want to bring your attention to this is and I want you guys to be aware of this that kidney cancer although is like eighth most common disease in them in US but you can see that the funding from the government for kidney cancer is four into this other part okay and this other part maybe like maybe 10 percent of that other part belongs to 10 20 percent of the other part belong to kidney cancer research that's why the kidney cancer research kind of didn't move too much but but but for the past couple years I think we made a lots of a wave and and lots of progress and I hopefully that you know or some of you can you know be aware of this and then hopefully we can get more of the research funding for this disease so so think about it you know to me I feel like if you really want to cure cancer the first thing you have you know the patient most of the time nowadays go to radiology and they find something and then if they find something that it you know the surgical oncologist you know can take it out and try to take a look at it and that's so so now the where am I so this is double screen so I need to figure out okay so so there's small pieces or big pieces got cut out and be visualized by pathologists and pathology would make a diagnosis about what kind of disease that you have and once you have disease if you localize disease will be operated and cured fortunately and if for the patient that have a metastatic disease disease spread out then you make start to see medical oncologist and the radiation oncologist any any working with surgical oncologist hopefully that we can still make a cure or at least extend the lifespan or change improve the quality of life of cancer patients so this is all good and well and the death thing to this need to kind of all thing together you need to you know have scientific based studies and really understand the disease and really under in the past the trials all based on like okay in real life five thousand if a thousand patients and I try to see whether you respond or not nowadays I think this is totally irresponsible behavior if we really want to conduct of meaningful trials you need to know what patient to select first and that's why precision medicine personalized medicine I think that's very very important otherwise we're missing the big boat okay so today is what I want to talk about is from you know it's from the pathologist standpoint medical oncologist standpoint and the scientific standpoint of this part I want you to kind of help you kind of guide you through this and hopefully you can understand cancer a little bit better in a very very simple sense why do we develop cancer so so this is just a single cell so our body is made of like about 30 point 37.2 trillion cells in our body every single cell has the same DNA okay has three billion base pair of one copy and there's another copy of it what is this chromosomes so it's about six billion base pair in our DNA and I think about it and then the DNA make RNA RNA protein protein and then our structure and it make metabolites and that with all these they build up our body so the bodies consist of 37.2 billion cells they have like every single cell actually it's showing sharing the exactly same three billion base pair obviously six billion base pair because you have two copies of it and the reason why just Jen has talked about why this is so difficult to deal with for the past you know with the beginning of the human history is because it's just part of us and the reason it is part of it it's not coming it's basically we go through evolution then everybody you know evolves different ways and DNA is very very stable but if it evolve if we don't have the capacity to evolve we are not evolving okay so we'll be the same a couple hundred millions years ago so that's the that's the the capacity we inherit then we have the capacity to mutate unfortunately if the mutation go crazy go too fast and take on take a wrong route then the cells become try to be the host and that's proliferate like crazy without further control and that's taking over and it's that's from your body the cancer is from your own cells and that's why it's so difficult to to take care of it and the other layer of complexity is not just because it's all you is the other thing is really it's because it's you know everybody you know different people they have different genetic makeup and people get different kind of cancer even within the so you can see that different people they could have different cancer and the other thing I will try to show you that is within the same patient this is a kidney cancer of mine so he has cancer here this is the kidney and he has metastasis here this is spleen this is liver are they are they the same disease it is a big question has been puzzling for us but no kidney cancer it just taught me a lot is many of them even you take from here to here from here they look they start from the same origin but they start to mutate and acquire different mutation and the truth acquiring different mutation they become behave a little bit different so that's why disease it is a layer of complexity and heterogeneity that that make disease is very difficult to treat and again so as I said that we start with cells and then we start with DNA so the you know so-called you know analysis analysis of big wholesale analysis of DNA is called you know genomics and then wholesale analysis of you know the RNA made from DNA is transcriptomics and then it you know RNA the purpose of it mainly is to make protein and protein is proteomics and then proteins mainly try to kind of build up our body conduct or an enzymatic reaction every time I'm moving my hands my consuming as lots of ATP and those kinds of that's all the enzyme is doing it so so this is complex it is very very intricate of phenomena so but the current focus in the past we all saw our focus on transcriptomics because the transition is easier to obtain but but it's flexible but generally a lot of noise so nowadays because we sort of go back and because of the sequencing technology available to us at this moment in in past in 1995 when this whole human genome was sequencing cost of billion dollars to sequence 20 times of your genome every single base pair sequence 20 times now it's about thousand dollars so that's a million times difference and that's why enable us to understand the disease is at life speed now compared to five ten years ago that's the major difference and that that's not only it and that's what we will build on it and we'll start to understand the other thing like proteomics and metabolomics hopefully by by knowing from DNA and through this we start from the genotype and the ultimate phenotype we can understand how cancer is built and how do we deal with cancer so this is a for for a cancer biologist it would you know everybody will know this so-called cancer hormones have been proposed for many but 10 20 years and it's been highly cited it's just because cancer is a cell within our cell try to be you know dominant dominant and try to gain all the capacity that we have by the differentiated and and by acquiring multiple activity and and causing a disease so so this is just a pretty a background information for you guys to understand it has changed so much from very beginning when I was seeing cancer patient when I was an intern trying to to help them and it was just like horribly disheartened because the only thing I can say that you have cancer I cannot do anything about it okay so things change a lot so in terms of kidney cancer the you know the most known kidney cancer and most of the people contracted it's so-called clear cell kidney cancer and and is it we notice a BHL disease and for suggesting me BHL disease for a long time but now with all the genomics I can tell you this is really really a BHL disease and and we make most progress of kidney cancer treatment on this disease and that's exactly what I say one year to three and then likely in five years and then the goal of my and and I was talking to Dr. Moser was that we should try to actually cure a significant amount of this disease before then we can actually say we make some progress then the other part of the disease sorry so the other non-clear cells so we'll call non-clear so because it's just so many different of them they have papillary and they have papillary type one type two they have chromophore they have a TFE3 tumor they have medullary type they're collecting type and there's something that we actually classifying so unclassified so these are all based on histology these are non-molecular characterization these are all just based on histology and there are other disease these are based on molecular characterization so this the kidney cancer is fascinating that's because that I'm studying and you know the patient I'm taking here it's just because it's a disease of many many many other diseases not like prostate cases one disease maybe a few disease long maybe three four disease this is like a disease of maybe 10 disease and in individual one of them we involve many many different kind of a mutation like SDHB patient I've seen of three of them for the past two three years they're only like maybe 20 family being identified in the whole world so you know so and the problem is that we we know how to treat this disease and it you know just genus talk about you know in old days the chemotherapy was the only way that I can that we can treat patient and that that you know some patient you know we need to know better how to use them but at a time the medium survival for five it's about a year and now it's much better we move the curve and it passed the clear silk disease is the bad disease but now you can see here the non-clear silk become a bad disease because we remove this curve from here and here and this disease the other 25 percent then basically left no trial and no meaningful trial and try to understand that but but hopefully by understanding the molecular makeup of the disease hopefully you can figure how to treat them more specifically instead of just say okay you have kidney cancer even you are non-clear silk kidney cancer but we just treat you as clear silk kidney cancer because we have no drug for it okay so that's the problem that we are facing so the the genomic studies now has been very clearly suggested clear-sealed type kidney cancer the majority of you know the patient here are patient or family and the type that they involve is called clear-sealed and it is a one-up the key feature is a chromosome 3p loss and the chromosome loss is because the first event is a loss of a gene called VHL and this and VHL is on 3p so we need to lose both good copies of the so-called tumor suppressor so that cancer can cancer take place and the key feature of these complexes regulate a phenomenon a protein it's called hip family school hypoxia inducible factor and these family proteins mainly induce hypoxia vessel formation and think about it this is it's part of our body responses if you have under hypoxia condition this gene need to be stabilized and they stimulate blood vessel formation that's why clear-sealed kidney cancer is full of blood vessel and that's why the drug they work not a chemotherapy agent they work the best they change the way we treat kidney cancer is the targeted therapy targeting blood vessels okay so that's what changed the practice and then the other thing that we know is about m-tore inhibitors it's been proposed maybe working here maybe working there nobody really understand what's really going on I think we may start to learn more about this and as you got kidney cancer clear-sealed kidney cancer very interesting for some reason it has to go to some reason to activate Torah pathway that's why Torah inhibitors are important and that's why Torah inhibitors is the first is the the kidney cancer is the first in a proof indication for m-tore inhibitors so we have made a lot of progress and these are the type nine that you can see from you know from 2005 or after they approved to 2012 accidentally to prove we moved the curve from one to two and half almost three but the problem is that we lots of it's about 20 to 30 percent better patient like this tumor strength about 50 percent of patients are stabilized for like a year or so then start to progress but it has about 20 percent of them even within a clear so they behave pooling in some way they die very very fast and then these are the patients that I try to learn more and and try to make them look like this and instead of they are dying like within three to six months but haven't said that that's why the first layer of heterogeneity and complexity and I think they are genomic determinants to determine which one is this which one is that which one is this I think I have some evidence for that but I don't want to bring it up to you guys kind of complete you could eat everything's way way too complicated but one thing I can tell you about is two real genomic studies now we begin to understand kidney cancer it clear cell kidney cancer is a VHO disease because VHO is mainly mutated and there are some other genes you know also mutated at high frequency like PBI 1 said D2 but these are all names one of the two 20 000 genes that we have in our bodies and the reason they are important because they when they mutated they make the cell give them more power give them more of the unwanted phenotype for our whole organism and they become try to become they differentiate they're probably afraid they don't die and take over the host it's a lose-lose situation because the cancer never probably get become another human being they all died so anyway so so we know much more about this and they actually tell us who will do better who will do worse because the problem that the issue what I have was maybe like 60 70 percent of patients will be cured by just surgery the problem is maybe maybe 20 percent of them will develop you see 20 years later or some of them what is it two years three years four years five years so spend millions billions of dollars trying to figure out who will recur who or not and then we develop all kind of the nomogram try to predict but here but I did study that we have done and it's not included here you should we should be able to kind of build a molecular feature and try to understand who will likely develop recurrence who will not instead of monitoring everybody exactly the same and that's that's to me that's um I hate to say that I kind of say this so in the past and I think it's very I don't want to say it's stupid but but it's exactly what I'm trying to say um because and then because it's the reason it's stupid is because you spend billions of dollars you will take 10 000 patient and you are not going to get the answer because it's just not the right way to do it and that's why the that's what health cost is so much because you don't select out the right patient to do the right thing and that's cost that's why it's so expensive okay so the you know I talk about you know the genomic studies that you know I talk about clear cell a little bit but now you know we we actually started to learn more about you know chromophobic type uh kidney cancer you know and and it's it's another type of kidney cancer it's very fascinating because this disease has multiple chromosomal loss it lost chromosome you know one two six ten thirteen seventeen it's kind of very very defective tumor by the point why would it this so defective tumor losing chromosomes so it should be a very wimpy tumor how come they if some of them about five to ten percent of this disease actually develop metastasis so we have some idea about maybe what's happening and and and and so on and so forth so the other things like you know papillary you know TCGA efforts has been trying to study this disease and we you know we like we'll be publishing this this year in 2015 and then it's some other disease I think I began to know more about this disease and then we have classified this we think this is about three to four different disease at least molecularity and these disease we have sort of pretty much better understand molecular handling of what these are these two are the one that at this moment really don't know much and this one happened imaginary happened in most of the African American who has a sickle cell trait and those patients always come if a patient come to see me it's about 20 some years so early 20s or early 30s and an African American patient before they came working at all I said this patient most likely have this bad disease is super aggressive okay and the most of the kidney cancer treatment nowadays that we have do not touch it so this is again so this is the genomic studies that was telling you that you know the kidney clear cell is losing 3p but the chromophore is losing 1 2 6 10 13 17 it's it's very crazy that you it's very effective how come you've come so you know so that's something that that needs to be studied in the future so the the cancer is it's genetically a Darwinian way where to think about a star which is exactly the single cell but it started a quite a different mutation and if you develop you know evolve a quality of mutation you can consider is a tree and maybe a different kind of tree but it before even they are evolving if they are it's within you know it's a localized disease you know surgeon can just sort of cut it out cut the tree whatever the tree to come whatever complex the tree is they cut it out you know then the tree is gone why do you need more about the intertumor heterogenesis of the primary side there was a big paper published by Charlie Rohmaston 2012 and triggered this whole wave of understanding the the heterogeneity but to me the heterogeneity of the primary tumor is it's it's important in predicting who is going to respond or who is going to relapse down the road but it's not as important in terms of do we really know you know how complex it is if you if you see many many mutations they are these these are passenger mutations have nothing to do with the pathogenesis of disease why would you care so surgery can take care of it the problem is that when the disease spread then it started then we started in color problem we need to deal with the heterogeneity and in the past it's very you know treating cancer is really you know old days like before even before immunotherapy is chemotherapy just just like a new period of war like bomb and everything did normal tissue get hurt tumor tissue get hurt then hopefully that you know normal tissue can recover better tumor tumor tissue get killed more and that's the that's the the way of treating cancer in the very beginning and then they'll target this therapy that you know if you try to find out where the the problem is and target it and hopefully you can get rid of the disease and now that the immunotherapy and something else that they're in a pipeline so this is the paper I was talking about that you know if you take kidney cancer it's like half fascinating disease if you take this is one patient and if you take tumor from there here different region of the primary or from the metastasis you're going to figure out it are carrying different kind of mutations and the problem is that interesting thing that is trying to teach us it diverge out crazily but they actually for example in this tumor this this different regions they all start with VHL mutation and this guy here branched out they mutate 32 and this totally different tumor also mutate 32 but different ways to mutating it so it's kind of fascinating right but we use always one mutation why would why would you acquire mutation but it also on the same gene suggesting there is a reason that you need to do that so it's not really just like a tree and I'll try to illustrate this way the way I think about cancer that is you know genetically it looks like a tree but phenotypically as I said the end output is like you need to acquire all the cancer phenotype that is they need to acquire different mutation on gene and gene translate into the phenotype that the cancer phenotype but once you started and actually you need you have for some disease for kidney cancer like for example they want to go through VHL but they don't they have to go to some gene to gain additional activity but it's wondering honestly started it has to go through the pathway it has to go through the same river and it is already predetermined it's not it's complex but it's not complex it's complex to a degree that we think is just too too bad we cannot do anything because they have to converge on certain genes and the reason I come around genes because they want to converge on function the reason I want to converge on functional pathways is because they want to gain common additional tumor phenotype okay so the easiest way I see it is this is my own personal you know view of cancer is I think it's it's it's a river you start with something and then you know you start to branch out but you don't have to converge and then converge against a gene level it can be on a pathway level but ultimately it is the phenotype level and that's why if you take if it's a clear cell cancer you start with the BHL mutation you deal with the BHL it works if you you know you started you know maybe it converts at mTOR somewhere so if block mTOR it works but whenever it gets too late I call you but later they start to acquire all kinds of way of mutation that's why you cannot even deal with it and because lots of them start to replace the the activity that you know rendered by the early mutation and it's become very difficult but that's the the opportunity of immunotherapy the immunotherapy actually does this maybe actually give you an immune system more target to deal with that's why immunotherapy is part of the game and I always you know and the reason immunotherapy is kind of interesting is you know we have our own immunity we were we born with our immunity and the immunity is really trying to tease out what use yourself so it's called tolerance I own okay this is my I don't want to attack it and if it's from foreign is a viral infection or bacterial infection you get rid of it so the strongest infection is acute infection you get rid of it and and then you live with your body right every single cell but you need to live with them through your lifespan so you need to develop a lot of tolerance and the cancer cell is it's part of it but it's just a little bit different because they may have cryolis mutation or that you have mutation maybe a little bit different it's a little bit dysregulated so so the immune the the immune checkpoint basically is just dialed down the activity okay so now you really try to recognize this by increasing your immunity and decrease the tolerance so the toxicity you can imagine is autoimmunity so it's very simple you think about you dial down your immune system and what happened what you're going to see is the immune autoimmunity autoimmunity disease so you know I will talk to about a vehicle and and and and then like I can say they all facts immune therapy is great it's important and I think it is correct but but I almost also reminded them that that we need to take all things into consideration you know therapy you know targeted therapy maybe even chemotherapy and treat figure out who should get what then get and then you are in the trial that way would be much much more meaningful and then the last thing I was trying to tell you that in the future there's another another thing that that people start to explore and then and that maybe actually contribute to future treatment is some of the metabolic pathway so again you start with DNA RNA protein metabolize and then this is basically trying to do something build up the DNA building DNA building protein building all those kind of things so some disease especially kidney cancer I think will be very range kidney cancer and pancreatic cancer very very metabolic driven disease I didn't know it's time to talk about it but it's really really a metabolic driven disease so I think metabolic treatment will be kind of interesting thing to look into for the future for some kind of cancer type just like immunotherapy will work on some cancer type not all cancer type you know the therapy work on smoker alone because it mutated heavily work on uv melanoma because it mutated heavily that generated a lot of new antigen the way you work on kidney cancer I've been thinking so much recently so it be really could be kidney cancer is really a very immunogenic disease it's not the mutation we know it's not a mutation because mutation is not that high but anyway so so that's my take on it and so so that's my 25 30 minutes


Hi, my name is Cesar Saavedra, Technical Marketing Manager with GitLab. In this demo, I'd like to take you on what a typical flow would be like for a developer using GitLab. I will be using Amazon EKS as a deployment environment. We have a lot of material, so let's get started. So as a developer, I track my work via boards. Here you can see my three boards open to do undoing. And let's say that there's an application already in production. And it's right here, this is a web app. And as you can see, the spring is here and the background is green. And let's assume that someone would like to make that more customizable. So there is a request that will be created by someone. It could be created from the issues screen right here. Or simply they can just come to my to-do board and enter customize web app landing page. And create and then submit issue. This will create the issue under my to-do board. Notice that the label has been assigned automatically. And I will see this obviously appear on my board. And then I'm going to start working on it. So I'll go ahead and assign it to myself. And then I'll move it to my doing board. Now at this point, I can actually open the issue. And the issue is what the problem is. And the MR, or merge request, is the center of changes of the work that I'm doing and the place where stakeholders collaborate on the resolution of the issue. So here I'm going to say create merge request. Then since I'm starting to work on this, I will go ahead and open web ID and navigate to where the source file is. Let's just say I want to change the background color and I'm going to say AWS. Also, I know that there is a unit test in this project that will test the color of the screen and the string that is being returned. Actually it's going to test what's being returned as the call to that web app. So I have to make the changes here too.  Now, if we go back to the MR, now notice here in the MR, you can actually click on changes and see side by side the changes that took place in the files of that project that I applied in this case. This is how the stakeholders on this MR can collaborate down to the source code line changes and say, you know, it looks good to me or something like that. For now I'm not going to put a comment in there, but you know, you can collaborate all the way down to a single source line. And this is good because, you know, through this collaboration, what you get is you get better code quality. It also speeds up the development of features and also increases developer productivity. This next screen, which is the pipeline. So a pipeline has been started as a result of committing the changes to the newly created branch. Since this pipeline will take a few minutes to complete, instead of waiting, I already have the same pipeline already completed. So let's go see it. Here is the pipeline. It's already been run all the way to the end. And this pipeline is part of our auto DevOps pipeline, which comes out of the box and is based on lessons learned and best practices for continuous integration and continuous delivery. These pipelines can jumpstart your development work, saving you time and shortening your development lifecycle. As you can see, they also shift work left as much as possible so that you can identify and fix problems and preempt issues from occurring in production. However, if these pipelines don't quite fit your needs, you can create your own pipeline or modify an existing one. This specific pipeline includes the review app, which is right here. And this spins up its own environment, in this case, in the EKS cluster, so that updates can be validated by the stakeholders of this MR. So let's briefly now go over each of the jobs included in this pipeline. We'll start with a build stage and also the job, which creates a build of the application using an existing Docker file. The resulting Docker file image is pushed to the container registry and tagged with a commit SHA or TAG. Then in the test stage, you have a few jobs here. In general, test analyzes your project to detect a language and framework that is written in or implemented in, and then it runs the appropriate test for your application. It will also use the test you already have in your application. In this case, it will run the single unit test that I showed you earlier. And if there are no tests, it's up to you to add them. Code quality, it runs a static analysis and other code checks on the current code. It creates a report, which is then uploaded as an artifact that you can later download and check out. Container scanning is a vulnerability static analysis for containers to check for potential security issues on Docker images. Gymnasium may have a dependency scanning, runs an analysis on the project dependencies and checks for potential security issues. License scanning searches the project dependencies for their license and checks them for compliance in your project. Secret SAS scans the content of the repository to find API keys, another information that should not be there. Another area related to secrets is the ability to mask and protect variables. GitLab provides this capability via our CICD variable settings, which I will show you in a minute. Spotbugs SAS, which is also part of our static application security testing suite. It runs a static analysis on the current code and checks for potential security issues, for example, buffer overflow. The review app, it actually creates, like I mentioned before, a container and deploys the application on EKS. And the stakeholders actually at this point can review updates to the app before it gets merged into the main branch, improving the fidelity of the solution that will eventually get deployed to production. At this point, I'd like to show you the way to protect variables, sensitive information. So for example, in this case, you can see some variables in this project related to AWS access key, and for example, ECS service and task definitions, et cetera. But if you were to change this, you would just click on these checkboxes. And obviously masking is useful so that you don't show the actual value in any of the jobs or output that is generated from running the jobs. So the ability to protect and mask these variables, shields organizations from potential security issues. So let's go back to where we were here and let's continue going over right here to the next stage is DAST. And that's our dynamic application security testing capabilities. DAST analyzes the current code and checks for potential security issues. For example, cross-site scripting. And then the performance measures the performance of a webpage. It creates a JSON report as well, including the overall performance score for each of the pages. And it uploads this report as an artifact. So let's go back to the MR here. One thing I like to bring up to your attention is that this MR is related to the issue that created it. And when the MR is completed, it will also close the issue. As the jobs run, data is generated and reports are generated that are uploaded as artifacts. You can click on the full report if you wanna get more in-depth information about that specific report. And in here you can, in this specific case is showing you the different types of security issues that it found. And you can actually dismiss the vulnerability. You can create an issue or you can get more information about that specific vulnerability. This specific MR has been, the pipeline ran already. And now we are ready to, it's waiting to be merged. And the assumption here is the stakeholders have had a chance also to run the review environment. So the review environment is actually a running environment that has all the updates applied to it. And it's the point or an environment that the stakeholders can use to check that all the changes that were made are correct. Let's merge the MR. Now notice that the merge button here is not available. It's grayed out. And the reason is that this MR has been marked in draft mode. Okay. So this is something that is built in within GitLab. And it's an extra safety mechanism to prevent an accidental merge and to indicate also that this is still being worked on. So first, we need to clear draft mode by clicking on the markers ready button. And now the prefix draft in the MR title will be gone. And now the merge button is activated. Also, you will see that there is a delete source branch check mark here that is checked. What this will do is it will clean up all the resources used by the review app, including the EKS container that was spun up. This actually helps with container sprawl. Let's merge. And this merge is going to kick off another pipeline that will run all the checks that you saw earlier and it'll take the deployment to a staging environment that will also be running on EKS. By this point, this is gonna take a few minutes. So let's go to an MR that has already completed. So this is the exact same MR that is executed on the other project, but it's already completed. Notice here that the changes were merged into master and the source branch has been deleted and the issue has been closed. So let's go to the pipeline. Now notice here now that this is showing some jobs here that were not there before. And this is basically the continuous delivery portion of the pipeline. You can see here that there is a staging environment. So again, like I mentioned before, once the updates have been merged, an environment is spun up on EKS. And this is called the staging environment. And at this point, you don't see it in this case, but these are manual steps. So these will be grayed out. These actually were grayed out. And you have to manually click on these to do an incremental rollout. You can choose 10, 25, 50 or 100%. An incremental rollout into production reduces the risk and contributes to a better user experience and reduces the danger of an outage. And after the application has been rolled out to production and to the production environment, another performance test is run in production. So let's go back to the completed merged MR. And I wanna click on production here. This is showing the production environment. This list is actually an auditable sequence of changes that have been applied to the production environment. This is the last one that was run. The nice thing about this auditable sequence of changes is that at any point in time, if you introduce a change and you notice that this is showing an unexpected behavior in production, you can always roll back the environment by clicking on this button right here, the rollback environment, which will take you back to a previous known state. At this point, you can click on view deployments and this is actually the application running in production. As you notice, it has a new color that we applied orange and it has the AWS prefix. So the two changes that we introduced into production. The other thing I wanted to show you was the monitoring. We have Prometheus running on the EKS cluster, collecting metrics. And this is showing you metrics, not just for the cluster, but also for the running application here. Let's really deploy this environment and then let's go back to the monitoring window. The rockets that you see here are actually a change that was just rolled out to production. And it's, you can actually see the job itself that executed as a result of that update to production. And the nice thing about these indicators is that they can help you troubleshoot production issues because there's a correlation between the behavior of, for example, total memory here and the introduction of that change. So these demos so far have used EKS and ECS as a deployment platform. However, you can also use GitLab with bare metal and virtualized deployment environments as well as different operating systems like Windows. All right, so to wrap up, we have gone over how GitLab can help organizations speed up their software development and delivery. We discussed auto DevOps and how it can get you started quickly with pipelines that are germane to your project language and deployment environment. Whether it is EKS, ECS, bare metal, virtualized environments or even Windows. Thank you for watching.