Before I start, I want to ask you, do you really understand what this picture means? This elephant is our totem animal and a trunk. And this is an impossible tourist. It's a logo of Jason. So this... So when I ask our artist to make a picture for this presentation, she asked me what's the history about this. And I said that for many years, the Polish community tried to stay separate from the Nova Square World. We say that we are relational database. And there are some small world of some hipster people who do some Jason stuff and so on. But real life is complex and our users start asking us about that.  And another group of people, they ask us about Jason in Postgres, they came from the Nova Square World. They start using MongoDB, start using other Nova Square databases and quickly realize that they need something more sophisticated, more flexible database, which keep... which say very robust about the data and so on. So eventually Postgres has got Jason support Jason now is a first-class citizen between data types of Postgres SQL. So now I will talk about the history of Nova Square features. Actually, 2003, we invented HStore, IN Theodore. We invented HStore for one very specific, one of the case of the Nova Square database. We was working on the system, which collected aggregate metadata from different objects. They have a similar but slightly different structure of metadata. And we have about 50 kinds of objects. And we have to combine it into one table. And this table becomes very wide, like 500 columns. I was very, I would say, angry to work in PeerSQL with so wide table.  So it was very sparse table. And we decided, we were working on application using Perl. So we said, okay, there is a hash in Perl. Why not use hash and implement hash storage in Postgres? And then we got just, I remember, 6 columns. And one 8 columns was just HStore data type, which contains everything inside. And this was very one of the first use case for the JSON. It's to aggregate some data with different structure. Another, we start using this HStore. And our friends give us, talk us about another case. They use HStore for storing user profiles or personification. And this was another use case. In 2006, in Toronto, the first meeting where all Postgres developers really see each other, because before we just exchanged email, email list. We presented HStore as a demonstration of genindex. And people say, wow, this is what we need, and they start using HStore. In 2012, JSON was eventually appeared in the 9.2. But this JSON was just a text, textual data type. And was useful only for storing and verification of JSON. People was a bit upset. And I started thinking about that we need to do something more usable. And we presented at PGCon nested HStore. So it was HStore with arrays, with objects. So it looks very similar to the JSON. But nested HStore has a binary storage, index support, bunch of operators and so on. Next conference we generalized our storage. I call it binary storage for nested data. And then we forgot nested HStore, because people said we don't need HStore, nested HStore JSON, we need just JSON. So we forgot nested HStore, just renamed to the JSONB. JSONB means some people say better, some people say binary. So what do you like? And it was committed to 9.4. And since then we see that on this graph, on this plot, it's a Google trend. I made it yesterday that JSONB becomes very, very popular. But HStore is still in use. So this is a very short history of... Yeah? Yeah. And another one is... We expect that in 2018 we will submit some new features to the progress. 11 or maybe 10 something. Who knows? We need your help. We need your votes. So after we got JSONB, we have... People said that this is NoSQL on ACID. And I made this picture, you know. NoSQL databases also evolved. They become... First they want just a simple key value database. Then it becomes order key value. And so on, it becomes more complicated. But here we have a very good relational database with very strong JSON support. We have binary storage for them. We have indexing. And I will show that actually NoSQL in Postgres can compete with MongoDB. And I see no reason to use MongoDB. Yeah. Okay. But now we have two JSON data types. You see there? We have a JSON and we have JSONB. So the difference between the... Illustrated just one SQL query. So the difference. JSON is a textual storage as is.  And JSONB has no white space. You see this? No white space. No duplicate case. So last key with win. And JSONB have a key sorted by length and then key. Very important. And it has a binary storage, so you don't need to parse again and again and has index support. Overall, JSONB about 10 times faster than JSON, just for sequential... If you do sequential scan. And if you use index, we can get like a thousand times faster. JSONB, really, really helpful and very useful. I will not talk about all details about JSONB, because two years ago, Christoph Petters made a very, very detailed talk about all these features. So I recommend to look on this. Also, did you understand this? You know about JSON and Argonomics. Yeah. So this is very nice picture, very nice presentation. Very good job. And here is the conclusion of his talk. Use relational data for the basic set of attributes and all other stuff stored in JSON. Forget about JSON and use JSONB. So this is... I completely agree, fully agree with this conclusion. So I save time for the more interesting stuff for the future. Yes. JSONB is great, but, you know, we have no good query language. JSONB actually is a black box for the SQL. But sometimes we want something to talk to JSON and to navigate inside the structure using SQL. For example, let's consider this very simple data here. This is just some artificial data set. It contains colors and some... on different levels. And you may think about this as a supermarket of some shop which describes the colors and some goods. And you want to find... to buy something red. Sometimes it happens, you know. You want to buy something red. So, with SQL you have to write this query. So not everyone can write this query. It's very complicated. It's not easy. It's because SQL was invented for atomic data type. It works with something like integer, string. Very good. But if you have a data type which has some structure, SQL can't work inside, you know. This was a problem. But fortunately, Polaris has a very good support of SQL and writing this complex query. You can do something. But it's not our way. So, at PGcon, we... I and Theodore, we made... we presented vodka and Jeff query. I would not talk about vodka now, but Jeff vodka is a new access method. We are still working on it. And Jeff query is a JSON query language. So, with JS query, you can write query like this. You see? Very simple. Very simple and very effective, because efficient, because it also has index support of the separators. So, if you don't know what is JS query, it is a GitHub repository, and you can download. It is extension to the Postgres. It works fine. Many people use JS query. It provides more comparison operator, index support, and search in nested object and arrays. So, it's very good. Why we did not committed JS query to the core of Postgres? People asking us. Because we know that JS query is not the answer. JS query is just a tool, which is temporary tool. We need some SQL level.  Postgres will not... JS query is data type. Yeah. So, we were looking for the better solution, some extensible SQL language. So, and fortunately, in December, SQL standard, JSON appears in the SQL standard. So, now it is official. I know that Oracle syntax is very similar to this proposed standard. So, I think that it's Oracle pushed JSON to the SQL standard. And we started working on implementation of this SQL JSON to the Postgres. Unfortunately, so, we discussed this in January, end of January in Brussels. Yes, and after the month, we submitted the patch to the community. But community said that it's too big. It's about 16,000 lines of code. And 5,000 of regression tests. And so, nobody will, you know, able to check all this stuff and committed for Postgres 10. So, unfortunately, we will not have this in Postgres 10. We will work for Postgres 11. SQL JSON, it's a new data type. It's not a data type. It is just a data model for SQL. The SQL standard doesn't require you have a data type. For example, in Oracle, or MySQL, or SQL server, they don't have JSON data type. They have a string, nvarchar, or something like this, some container. And do or and apply model to this container, to this container. But the problem is that it's very slow. They have to parse every time. Postgres, made because Postgres has extensibility, extendable by database. So, we can implement a new data type easily. We have all interfaces. So, that's why we have a JSON B as a data type, which already parsed binary storage in a very fast. Much faster than in Oracle, than in MySQL, or SQL server. We don't need to parse every time. So, one question was, we have JSON, JSON B, should we implement something, you know, like new SQL JSON? Three data types, too much. So, we decided to work, to continue to stay with JSON B, because it's already usable, very good data type. It's a subset of standard. It is an ordered and unique case. So, we will not implement new data type. We will implement new features from standard to JSON B. SQL JSON standard contains also specification for JSON pass. JSON pass data type for SQL pass language. It's very similar to JS query, actually. Different syntaxes, but easy to understand. If you know JS query, it's easy to work with JSON pass. So, JSON pass allowed you to navigate through the structure of JSON. You can construct very, very interesting queries. You can do everything with JSON pass. And also it specifies nine functions implemented as SQL clause. It's not a function, it's a SQL clause. So, we have, instead of something like 60 functions for JSON and JSON B, SQL standard just specifies nine functions, because it has a JSON pass. So, but for example, how it looks. So, it uses dot notation. So, dollar sign means the root element. And then, normal, everybody understand this. Dollar dot ABC and so on. You can use arrays as a bracket square and filters, which is very, very powerful. So, you can specify that the value of this key at sign should be greater than 10 inside the pass. It's very handful. And also in JSON pass you can specify a method, conversion method. So, you can convert to date time, to integer to some. Take a absolute value and so on. So, JSON pass is very, very powerful. You can see that this is a query from all this, which use contains operator. And this is query, the same query, which use SQL JSON syntax and JSON pass. It looks similar, but I agree that sometimes you need to type too much. But the good is that this is standard. SQL JSON, you see the SQL exist. This example illustrates question sign filter and some expression. Expression use name variable. The name variable pass quite later. Just example. You can imagine that, a very simple example, but you can imagine, you can write more interesting queries. And we have a SQL JSON construction function. I will not talk about them, because they look very similar to Alva. JSON B, RACE, JSON B, Object and so on. Looks very similar. And we have a JSON retrieval function. JSON value. It extracts a SQL value of predefined type from JSON value. And JSON query, it's a JSON text from JSON. It can, very powerful, it's returned many, many values. JSON table, it's think about unnest for JSON. So it's convert JSON to the relational forum. It's very powerful, also function. No testing function, JSON exist function. It's also useful. I will show next, for example, this how to use JSON value. In JSON value, you can specify also returning, returning integer. So we default minus 1 on empty. So if there is no value, then by default we will return minus 1. If there is an error, so we don't default minus 2. So here is an example. And JSON query use another example and the example of using wrapper. So we can specify should we wrap value result to the array, to the arrays. Conditional, unconditional, without wrapping and so on. It's very simple examples, because we have 5000 lines of examples. Yeah, this is just for easy understanding. Yes, that it is very powerful. And constraints, finally we can do constraints. This is what people ask about. Now we can do. Let's demonstrate the power of JSON table. So we have a delicious bookmark. And we convert JSON data. It's about 1 and 3 GB today relational data. So the schema is very simple. Just column name of type JSONB. And it looks like this one. So the structure is it's complete. So how to convert it to relational format. With the JSON table, it's easy. Not very easy, but it looks manageable, you know, the query. So here we use JSON table can be used only from page, from cloud. So we can specify the necessary path. So we can make some cycles inside. And we specify what we do with the result. Inner join, cross join, and so on. You understand it. You have to create new relational row with JSON value converted to a scalar value. So you have to if you have, for example, one array, you have to use join inside. If you have several arrays, you have to join between all of them. It becomes very it's very difficult. And so now we got relational data. And it's relational data in this case becomes about 2 times bigger. And this schema. It's just example of JSON table. So an SQL JSON finds something red looks like this. Over the JS query, but also understandable query more or less. You can play with SQL JSON already. We have a GitHub repository. If you want to understand the grammars, BNF is available also. Somebody published it, I don't know who. And we need your feedback bug reports and suggestions because SQL standard was written on very specific dialect of English. We Russians we very difficult to understand sometimes. So we need your help if we miss something. And especially we have problem writing documentation. Imagine how to describe document all these features. It will be very difficult for non native English speaker. Yeah. So the road map we try to push SQL JSON to Postgres 11. We have our fork Postgres Pro will be available maybe in May. So you can download and play already. Yes. And another is dictionary compression. We also plan to push to Postgres 11. Yes. And also will be available in Postgres Pro 10. Yes. And when we will have a sharding we will have a real SQL database. And sharding will be in several years I believe in Postgres. So now we will talk about JSONB compression. The idea is to have a transparent compression of JSONB and access to the child elements without full decompression of the JSON. That was the goal. And motivation was that I've seen JSON documents with very, very, very long key and the value is just true false. Or just integer. Or just something. And I understand people. They want to have something normal, understandable. Not just F1, F2, K1, K2. But JSON But it takes so long, so big overhead inside. And people asking us about could you please make it something like compression with JSONB. And the idea is very simple to use dictionary compression. So dictionary compression means that you have keys, key names stored in some external place and in JSON inside use just ID of keys. And then you can save a lot. For example if you have key name like 20 bytes you replace by integer, by 4 bytes and you have 100 such keys in the JSON and you have 1 million of rows by 1 million JSON you can save several gigabytes just on dictionary compression. Of course if you have long values the saving will not be so big. Also we try to compress values because we we try to use to you know object and array entries we have overhead like 4 bytes per array element. If you have 8 elements in array you'll spend just 32 bytes just for overhead. 8 bytes for object field. So we can save it using variable byte compression but I don't believe it will help too much. Dictionary compression is the best what we expect. But you see that what we use inside. So keys replaced by the ID we use delta coding if we have sorted ID arrays so we can use delta coding we have variable length encoding entries instead of 4 bytes entries for arrays we use chunk encoding for entry arrays and store integer numeric into int32 and use variable length encoding. It looks very strange but it works. Believe me I will show you some examples. We implement create compression method so you can specify your own user compression method and the syntax now looks like create table column name data type and you can specify compress username and suboptions but you can make alter table alter type and information about compression method stored in the pg attributes we have two additional fields it's compression method and compression options and one implementation of this infrastructure is JSONBC compression method type using dictionary compression so we have two possibilities so you understand there is an infrastructure and one implementation actually we try something like 10 different compressors we use lz for dictionary we use snap many many compressors but they are not they work well sometimes but they are too slow you have to decompress all JSON to get the value of specified keys so we decided to stay just with JSONBC which is a dictionary compression and it's you can have dictionaries identified by sequence ID and you know with sequence ID you have a fully transparent updates of dictionaries but and another if you know predefined key names you can use just Yenom so we have two methods for the dictionaries here is example we have create table JSONBC compress JSONBC and can manually create dictionary sequence and create table and specify this you know manually updating this and now the results of compression we have two data sets it's a delicious bookmarks it's 1.2 million rows they are mostly string values relatively short case and have two arrays of tags and links of three field objects so it should be not very good for the compression and another is a situs dataset customer reviews from amazon it's a 3 million rows rather long case and a lot of short integer numbers so it's a very good candidate for the compression yes another test we made with compressed file system it's a page-level compression which we implemented in our posguest pro enterprise it was also interesting how JSONBC compared with just use compression file system and here is the results so this is three two data set and the diction bookmarks here is a relational table so it's a JSON converted to relational data and here is the size and slide with about 700 megabytes and this will be different in all colors because so this is a compressed file system so we see that compressed is about 250 something like this so it's come up CFS is good for this JSON compressed also good JSON JSONB here is a JSONBC so we see that JSONBC has a very low overhead overhead to the relational data but with deletion bookmarks it's not so good so we have ah, I forgot to say that here is a wiretider the same data we load the Mongo so we see this is a Mongo size and Mongo Mongo developers they make very good job they have a very good compression so our JSONBC has a bigger size there is Mongo wiretider but still the idea of this plot is to show that JSONBC is very good compressed it's very close to the size of relational table and also interesting to see that compression file system works very well so if you have CFS you don't need maybe JSONBC yes the same for ZFS we did not test ZFS because we just don't have experience and our engineer said that it doesn't work but I saw that some people said that it works very well but I think that we will try of course but compression is another one side another side is performance because if you have a very good compression but it's very slow you don't need it so here is the performance it just contains query and we see that for relational data we have very slow performance but JSONBC and JSONBC have about the same performance and even better than Mongo so Mongo is slower here again JSONBC looks very good it has a good compression and very good performance so what does this mean what does it mean first we run all our tests in non-parallel mode then we turn on parallel mode and we got this number so you see that JSONBC and JSONBC several times faster than Mongo in parallel mode another query how to get several keys we see that JSONBC is faster than JSONBC we did not test MongoDB because we don't know how to measure time in Mongo they use a cursor impossible to check how long it takes to get several keys maybe you know you save me and very also interesting aggregate we use this average and of course relational table shines of course very fast but JSONBC still have comparable results and also is faster than Mongo and again we can use a parallel mode which parallel mode is that the parallel mode now in 96 yes what is the slide showing I understood the previous one oops getting several keys so key not just one field but three fields so you see that this performance and we have of course one problems and one problem is transactional dictionary updates of course sorry I will ask so currently automatic dictionary updates we use background workers but autonomous transaction would be much better so maybe it's posgres 11 there will be some autonomous transaction we can use them and also we did not implement cascading deletion of dictionaries because for example if you drop some column we need to track the dependency if there is some dictionary associated with this column this is easy, we just did not implement it and also we did not implement a user compression method not fully supported but I don't know should we do this summary is that jsonbc can reduce jsonb column size to its relational equivalent size so it's very good and it has very low CPU overhead and sometimes even faster than jsonb and compression can be significantly lower than in page level compression on cfs we need of course to do some work and availability you can download from github and play check on your data now we need to do some benchmarks how no school is fast because there are many many myths and the first myth made myself in 2014 it was my first non scientific benchmark which people I first show that jsonb can be faster than monger because before people just didn't believe before people said that monger is very good, very fast but when I did this benchmark very simple, very non scientific I was surprised that we are faster we are faster and you see that time monger was very slow for loading 10 minutes compared to the 40 seconds it was that time then enterprise db published the result no school benchmark it was already it was now it was a bit more complex than my non scientific but people still believe that it's not good because all this benchmark was written by positive people of course and people tend to believe some independent and scientific benchmark the benchmark should be reproducible they should support more databases many work laws it should be open source and so on so what we did we implemented we run ycsv benchmark do you know what this is benchmark available there is a scientific paper and it's a de facto standard for the measuring nose to a performer so we run ycsv for the postgres master and monger db 3.2 it's the latest version we use a two servers one server to generate load another server for database and they connected with 2 gigabit ethernet and ycsv benchmark provides 6 core workloads workload A it's update heavy 50% of updates and 50% of read workload B it's read mostly it's 90% and 5% workload C is read only 100% read workload D read latest so you insert new record new rows inserted and then get all this so you get hot workload E it's a short range and workload F it's read modify write so you read record modify and write back changes and all workloads used zip file distribution for record selection so this is not written by postgres people this is written by some independent people there are published results for many many databases so now we have for postgres we use a postgres 9.6 sometimes and master sometimes with asynchronous commit on why? because we want to match the monger db monger db g0 because if we use a g1 it becomes too slow very slow so you know what this asynchronous commit so we use 2 sets of 1 and 5 million rows and with asynchronous commit off we were able to run just 100 kilobytes of rows because monger db becomes very slow progress is optimized for this but not monger also we use fast update for gene index this is a problem of gene we know this and we tested a lot we tested functional d3 index for jsonb jsonb cd sql json functional d3 index for jsonb on compressed file system gene index for jsonb for jsonb build object monger db wiretiger this is default and we returned a whole json and another test we returned just one field we used 2 kinds of json just 10 fields and 200 fields this is very important because with 200 fields the average size about 7 kilobytes so it's toasted and you can see all variants and here is the result first we identify the problem we identify problem with hot you know what is hot update workload A hemi update 50% of updates and we found that progress becomes after some 40 clients it becomes the performance drops and we found the problem because hot doesn't support all structure all the data types like json, hstore, arrays because we have a functional index on the one key but updates different key and hot doesn't understand this so this is a problem we made some quick fix of course for the hot and then it becomes better so all other results will be with this fix but we need to make more sophisticated fix it was quick and dirty and here is the result for the six workloads workload A, D, C, D, E, N and then the monger is yellow monger is yellow and v3 json, v3, json, v3 we see that on all workloads functional v3 json, v3 is better monger is always lower I remind you this is for 10 fields so short rather short, not toasted json another results it's a 10, 1 million rows 10 fields and we see that progress is better in all workloads again here monger is crimson vilec vilec monger and all other is progress so we see that again progress is better than monger and we see that djin is not good for updates this is understood djin should be not good for update index but it's okay for select for workload C for example workload C djin is the same as v3 but djin has very interesting feature djin index all keys functional v3 key index just one key like in monger so djin sometimes could be very useful but not for updates now we have results for 5 million rows, 10 fields we also see the same results that progress is better progress is blue and monger is green sorry for the colors colors is why we should be green but it looks like yellow that's my objective yeah maybe but believe me it's green here yeah but we see that all this progress 1 million rows 5 billion rows 10 fields progress is better than monger okay so we made 1 million not 1 million 100 keys rows it's just my mistakes 10 fields and we use asynchronous commit off and we see that progress also better this is monger okay yes but situation is not very good when we take 200 fields so json becomes toasted and we see that monger db is better on somewhere close we need to understand what we can do what we tested here progress is toasted and we use and we identify that 50% spend to the pglz to uncompress it so we create another test with no pglz it becomes better but still sometimes we monger db is better so for long json b monger db has some chance in somewhere close of course so the conclusion is that pglz still beats monger db on one server that's true because we don't have built in sharding some distributed model so we use just one server i hope that in several years we can repeat this benchmark and see how we are good in distributed model but i believe that 90% of projects could be run on one server you don't need many servers so the summary of my presentation is that pglz is already good no squal database yes and the squal json provide better flexibility and interoperability that means that queries from Oracle will run the same as in pglz because it's a part of squal standard this is very good and squal json is just just the beginning just starting i expect this year or next year they will release something more because squal json doesn't specify any updates, any modified operator just build and query and json b compression is really useful we also expect in pglz 11 pglz beats MongoDB in one node configuration so we need to figure out how to make a hot support for json b and hstore and other arrays maybe we need it's time to introduce type specific compression so for now we use pglz for any data types this is not good because for json b we need some another type of compression for example so this is also direction of future work to introduce type specific so data type should understand which compression is better for them not just pglz and maybe we can run ycsb benchmark in distribute mode using cytos cytos demonstrated the cytos mx how to run in distribute maybe we can run using cytos and posrace will help no squal users to avoid nightmare yes we need sharding and community is working on it we have two different using fdw and using cytos approach so we'll see so expected in 2-3 years and the last the slide is about credit to people behind json b who help us with json b andrew who also made the first version of json and help us with json b roger class who work with andrew this is admitted although who also work for implementing some functions he ported actually hstore functions to the json b functions and now he implemented full tech search for json and subscription syntaxes like arrays for json b so i don't know how andrew will commit this to posrace 10 full tech search will go to the posrace 10 and 3 companies engineer was the first company posrace was interesting this is how our community works they just sent me email and said would you like to experiment with money to work and do something then hero cook and our company posrace professional still supported the development of posrace yes and the last slide sells to think what is the birthday of mincent van goch this is my favorite artist and this is a very fun but not by van goch of course somebody emulate the style of van goch but i like this picture so celebrating van goch okay thanks and attention yes i am here so you can ask me and we can discuss everything


All right, such an honor and pleasure to be hosting Lynn Langit for this closing keynote of Agile India Conference. Lynn is someone that I've been trying to get to the Agile India Conference for many years now. I first met her in the Yaw Conference in Sydney.  Some of you would have met Dr. Dennis in the past Agile India and ODAC conferences that we've hosted. You know, they've been doing some pretty cutting edge amazing work with, you know, sequencing genomes and doing all of that with, you know, basically function as a service architecture and just the amount of cost savings that you've been able to demonstrate has just been amazingly impressive. So, and then of course you've been very influential in all your work with the cloud and data work that you've been doing. And I also see that you now built a lot of LinkedIn courses with over 4 million, you know, students so that's a that's a huge impact. Congratulations on that. And without much delay, I want to get straight to the topic. So request you to please share your screen. Great, thank you and thank you and welcome to everyone. So let's go ahead and get started on our topic today. All right, well welcome to everyone. And this talk is on cloud adoption patterns. And my name is Lynn Langett. So to set context, it's interesting to note that there is a group that studies success and failure of software projects they do this every year and they collect information and they publish it and I have a link. I can make these slides available. And what they find is 75% of software projects each year, every year have some or all of these characteristics. The first characteristic is the project is late. It's delivered not on time. The second is it's over budget costs too much. The next is it lacks key features. And unfortunately the last one is it sometimes doesn't even work. And, you know, one of the challenges of our industry is it moves very, very quickly, particularly in the domain I'm going to talk about, which increases the likelihood of these situations occurring. So how do you counteract that while you collect information about what works as patterns. And that's really what has driven the creation of this talk. It might surprise you that when I considered the cloud work that I had done as an independent consultant over the past 11 years. I'm really not making this up 95% of the projects that I've worked on at were successful. Now, I'll put a little asterisk here. I'm often brought into projects that are in a failed state. So it's not a true metric, but it's certainly better than the previous one. So again, how and why do the teams that I work with do this. Well, to understand that you have to understand a little bit about me first. So, first I came out of the corporate world 2007 until 2011 I worked at Microsoft. And I was a global employee, focusing mostly on data. That's my background data warehousing and databases. So when I left Microsoft to launch my consultancy. Although I started working with the cloud because literally the year I left is the year that Microsoft launched Azure and Amazon was coming into the market. The first focus that I had was big data, and it was in the advertising in the financial area. And I, you know, started moving people in really core patterns lift and shift kind of patterns, but had a lot of success. Cloud data was quickly on the scene with offerings from really the all three of them. I was working more with AWS and GCP. I still do work with Azure and sometimes even Alibaba. But you know you can't know all the services and so AWS was first mover. And so I often was focusing on AWS and then more and more GCP. In 2017, because of a personal situation, and opportunity, I pivoted my work professionally from financial and advertising to human health, specifically cancer research. And I started my first project with a group out in Australia that was mentioned in my introduction, and really found that there was a lot of opportunity in cancer genomic research for optimizations around the cloud. And so I have subsequently worked only in that domain. Now up in 2022, I've had over 20 clients and have had substantial impact in human health in cancer and in COVID particularly since 2020, and have elevated and escalated my journey because of some of the characteristics of genomics. And the first characteristic is there's data, there's big data, and then there's genomic data. Before the pandemic, one of my key customers before the pandemic was putting in 45 terabytes per day into it was Google Cloud. And because they were actually part of in the United States testing around COVID, that just increased rapidly during COVID. These are data volumes that I had never seen in my data career. And so working with customers to build solutions that work effectively in cloud with these volumes of data, and with the complexity of computation that genomics requires because of the complexity of the questions that are asked across the human genome, drives new patterns, which not only are going to impact genomics, but are also going to impact all of cloud solutions. So let's look in terms of patterns. And in this talk, I'm going to identify a number of key cloud patterns that will drive your teams to success. It's interesting where this talk came from. I'm working with a big customer in the United States right now, moving a number of projects to the cloud. And they've had, unfortunately, sort of the typical situation, the 75%, where a lot of the projects are challenged. And so some of the teams have come to me and said, you know, have you observed these patterns? And so about a month ago, I actually wrote an article about this. And then just coincidentally, I was invited to this keynote. And so this is a new talk that I've created for you, but also for my customer. The first cloud pattern is, in my opinion, the most important. So we're going to spend the most time on it. And this is particularly for customers who are brand new to cloud. A key pillar of success is understanding and using elastic services. So in addition to talking about patterns, I also want to introduce this concept that I call cloud smells, probably you're familiar with code smells. And this is just something that I have used informally that I'm, you know, sharing with with you all here so hopefully it will be helpful. So one of the cloud smells around this area of elastic services is when teams are new to the cloud, they have a very difficult time understanding that compute should be ephemeral, which means it should go away. Compute should come up and go down because, you know, they're used to servers and racks. And so it's very, very difficult and scary to think about VMs that are properly sized do a job and then go away. And this is an example from this customer, who's they're trying to move a batch cluster from, well, it's a Colo to the cloud. And as they're rewriting the code base. He is spending inordinate amounts of time because the log files are currently written to the nodes, which go away. So he doesn't have any log files. So you can see here he spent one week and basically it doesn't really matter the details but to figure out he just needed more ram for a task, which is just, you know, this is what causes those 75%. So where should the logs be the logs should be written to the persistent area of the cloud, which is buckets in this case, but this idea that services are, you know, going to be always there is a cloud smell. So the other aspect of these patterns that's difficult is cloud has evolved in terms of the services. When cloud was first introduced back in 2011 2012 and this is an Amazon architecture, you know, you had like five things to worry about five services. So as you know they were configurable, it was a much easier problem space. And as we'll see this is sort of the second dimension that adds to the complexity. But let's just start with 2012. So we're talking about elastic services and that would be easy to, you know, or whatever VM right. So in 2006, Amazon, you know, launched really the concept of public cloud with EC2 and Azure and and a GCP came along later with optimized sizable VMs. The second big aspect of working with elastic services and compute the big releases came in 2014. And that's Amazon Lambda and Google open source and Kubernetes and Docker. There's also including another key service in here that I use on GCP which is cloud run, which is a way to quickly test out Docker containers. And it, it's been a stepping stone. The diagram is designed deliberately, you know, if you're moving, for example, in GCP from Compute Engine, the next step in terms of elasticity in my opinion is not Kubernetes. It's too complicated. You want something simpler that's more like Lambda, and that's cloud run. So from a pragmatic point, all right, you should have elastic properly sized instances and this is case of EC2. How do you understand the instances. This is a measure of cloud fluency around elasticity that I talked to my customers about, you know, Amazon does not make this information easily available. This is the use of tools, right. This is an open source tool. It's available on GitHub and again linked in the slide where you can filter your EC2 instance types by your needs. And it also has the pricing. It's kind of cut off on here. But you need to have some sort of capacity for understanding and I listed Amazon because they're the most difficult to do this with. Google is a little bit easier. They actually display the pricing in the console. And I think Azure does that too. Amazon's notorious.  And I have had this case in genomics. I had a customer actually last year, that's they were going cloud native and they should have had a cluster, and they just simply picked the largest EC2 instance that you could get and left it on. And I just said, oh, no, this is not what you want to do because it's core concept of elasticity is key to success with getting value out of cloud. Speaking of that, if you picked that biggest EC2 instance and just left it on, why would that be bad in terms of your cloud costs? Well, it would be bad because the cloud constantly evolves and you probably don't need it on at all times. And interestingly, for my work, despite the importance of EC2 or the appropriately sized VM in genomics, because of the data volume, compute is just a small part of getting value from cloud. Now, I realize that those of you listening are probably not genomics. So your compute bill could very well be the biggest part and that, you know, pre genomics, that was the case for a lot of my customers. But showing you the future where we need to optimize is compute yes, but data very much yes, because of the volumes of data. I had one customer back before the pandemic, they were spending $250,000 a month on bucket storage. And by simply changing the bucket storage from the default of multi regional to single region, which was met their needs, we reduced that bill by 100,000 a month. And again, the reason I tell you this is because I think genomics on the cloud is the future for many data cases because of the vast volumes of data we're collecting across devices. So speaking of cloud evolving, the next evolution really is a reflection of that data. This is work that I did as was mentioned with that group out in Australia to take an on-prem genomic process that took 500 hours to run and to cloudify it. This is the intermediate level of work where, again, you don't really have to understand the architecture, but the key point is all those little red buckets. The cloud optimization was around data. So a second pillar is to use the correct service for your particular use case. Data is moving in the genomic scale off of file systems into buckets because that storage is exponentially more scalable and cheaper. And in this case, we had great success. We took this process from 500 hours to like five hours. Now we did subsequent improvement that took it down to 10 minutes. But this going from 500 hours to five hours was a really important intermediate step and an example of cloud fluency. I actually wrote all this up and this is one of the first examples of a data lake for genomics back in 2017 and it's on my medium. So again, linked. So what is elastic data? You know, I come out of data warehousing. So, you know, say, well, where's your databases? Well, they're still there and there's still cases where they're best fit. But in my genomic world, the big release, of course, was S3 buckets, which really, in my opinion, launched the cloud as much as EC2 did. And then the secondary releases are around pass or data as a service. So the big one in 2011 was BigQuery from Google, which is a sequel on top of files, which I think to this day, and what is it, 10 years later, customers don't fully understand. Subsequent really important releases were spark on Kubernetes, which is actually what we used in that previous architecture to get it down to 10 minutes. And then of particular interest, because I know it's always great to know what's relevant right now is this newly released set of services and to my understanding, Google's out first with data plex around data mesh, which is a data catalog across data lakes. And this in my domain of genomic scale data is a game changer. Because if you think about it, if you have data sprawled across your organization and the internet in file buckets, basically, how are you going to find that. And, you know, the challenges during the pandemic of collaborating within and more importantly between global health organizations really escalated the need for this, because with the amount of genomic data that is now being sequenced because of the decreasing price and the health needs. This is a really exciting area to work in and there's high demand. So again, if you're watching this and you're thinking about what you should do for your career to increase your career growth. If you have interest in the data area, I would say you really want to start exploring implementing data mesh with customers. And because of that this is actually just an example of something that I did I made this for myself and my customers and this is the Google version. It's important with this growth of services on cloud that you create these maps for yourself and your customers so this is my mind map of different data capabilities on Google cloud that again I created for this customer so that we can figure out how the parts of it are going to be providing the most value and are the most priority to implement because the challenge you know most these vendors have over 300 services now is how do you pick the right one. And the data area to me is the most interesting area probably because my background but it's similar and networking and security and the other areas as well. So I kind of that's a high level look, but the challenge, you know, to get to that 95% success rate is you have to pick the right set of services but then once you get into the services, you have to understand them deeply, because what that impacts is the, you know, service that you provide to the customer and how much it costs. So this is an example from Google and there's a developer advocate that's referenced here, where she is drawing these sketch notes that drill to help you to quickly look at services that are core this of course is persistent disks that you would associate to VM so it's kind of the opposite of a data mesh, it's like an individual disk. And if you look at it, you know, it is challenging that you have, what is it five different ways that you can select it at the highest level, and then you have another dimension of three different ways whether it's local SSD persistent and all that. So the key pattern on this is for the services that you select. Do you understand the implication of the configuration options. And again, I'm going to go back to that client I had where we save them $100,000 a month by changing one setting on a bucket. It really matters. It really matters in terms of cloud fluency. So speaking of cloud fluency where are we now. So, again, it doesn't matter so much the architecture this is an architecture I was working with genomics customer this year. What matters is if you take away nothing else there's more squares which is the more services. So first you have to pick the best fit services, so you have to understand from a high level and then you have to configure them correctly. An additional aspect of this is almost all of my work really for several years in data processing also includes machine learning and that's reflected here on the bottom right with the green square for Sage maker. So that really should be a number of squares kind of ran out of room here, because elastic machine learning is I think at the tip of the cloud fluency iceberg. So what does that look like in 2007 it was really just open source libraries. The big release for me was 2015 with TensorFlow open sourcing and then Amazon built Sage maker. And to that Google has continued to make neural networks and machine learning more usable in particular, I mean all of them are but Google in particular with auto ML. Also notable releases are Google collabs, which allows you to have Jupyter notebooks in a browser. And it's a path to get to full cloud for data science, and Amazon has a pretty new release called Sage maker labs which does the same thing. It allows this elasticity in that you have data scientists who have some or no cloud, and they can go from their desktop. In the case of collabs and Sage maker labs to a browser based SAS tool, where they can try out their notebooks that can then be built in a machine learning lifecycle on the cloud. So that was a lot on the first pattern, but it's the most important pattern, but there are more. And the third pattern, by contrast, is a little bit more, I don't know, straightforward, simple, but as important in a lot of ways. So the cloud smell associated to this is how do you get somebody working on the cloud, whether it's a developer and end user, and it continues to astound me that people do not work on the cloud for the cloud. In 2013, and you can see it's a third part of a three part series of blog posts of how to set up for development for an AWS service elastic beanstalk install this configure this update this to do to do to do. Clients I've had recently, I mean I took me as a consultant, I don't know two weeks to be able to access the console. It's kind of ridiculous. So I realized you have to have controls, but the sort of antithesis of this is you want to work in the cloud on the cloud. And again because I just happen to be doing a lot with you see P. This is the console which is fine to do use when you're learning and teaching. Of course when you're deploying to production you're going to use script because you have the concept of infrastructure as code. But on the bottom left you see the terminal, or cloud shell, which is a VM that has all the tools you would need installed and configured. So again, rather than spending days or weeks and setting things up you just click on that basically. And if you click the open editor, you get a ID that's browser based that again has all the tools. It is just crazy to me that people do not use these tools more often. I've actually again written medium article about this, because for years now, I install nothing on my laptop.  So the third cloud pattern for successful deployment is organizational success and again this probably reflects the nature of my work which is an independent consultancy. So although I come in sometimes at the beginning of projects for proof of concepts. I'm often brought in I call myself a cloud plumber when things don't work. So, so the cloud smell is doesn't work. So I am continually surprised by when I asked the question, which end users are using this solution. The answer is all too often none. In my opinion, software is not usable, unless it's usable by end users, if a developer can run it on his or her desktop, it's not usable. It's given end user. And then the next question is how many end users. And I find more often than not, there are assumptions that are not measured. Again, a very easy thing to do is in the genomics use case where everything goes into file buckets. What percentage of your, in my case, genomic researchers have uploaded a file to a bucket. If you can't answer that question you don't know how many people are using your solution. So the, the, and the counter to this is you want to figure out your minimum viable anything if you don't have any software working and deploy it as soon as possible. And this is something that I've actually been working on I literally was called into this project. I've been working on it about four months now where millions of dollars were spent to write genomic pipelines for a customer. And they were never deployed. And so it's been two years. And the very first thing I said is, okay, let's take this pipeline that runs for 14 hours and let's get a one hour version of it. You have to get the, this is, you know, an agile principle, you have to get rapid feedback, right. And in whatever domain you're in, no matter how complicated it is, it's possible to slice off a minimum so that you can get something working and build from there. And I'm really happy to say that just coincidentally, the end user ran the full pipeline yesterday wasn't been a good week in production. So it works. This comes out of lean manufacturing minimum viable concept. When I, you know, old school when I was a DBA it was a minimum viable report and now it's a minimum viable pipeline, but the key is a small version, all the way to the end user and then iterate. So, to get that going.  And so, to me a picture is the way to communicate much faster and much more globally than words. And oftentimes when you have these stuck projects, you have an assumption of understanding through lots of words and lots of documents and lots of code that nobody reads. So it's, you know, it's might seem sort of childlike, but it works. So this is one that I drew for this particular project that we want to take this, you know, complex pipeline that has 54 tasks in it and we want to just get it down to three tasks and have an in and an out. And I do think that this thinking is part of getting to MVP I've seen it many times over the years. So we looked at cloud patterns as it relates to technology, the other aspect of successful software is people. And so, the fourth cloud pattern and the subsequent patterns are under the people sphere and the first one is technical manager. So the association here for the challenge is we just use lift and shift. And you might say, Well, how's that relate to technical manager. That means the manager is not technical, because they don't understand the different capabilities that are available in cloud. Yes, you might want to use lift and shift. But you might also want to use serverless. This article that I've quoted here from this person in the industry Lee Atkinson, he actually is saying here that in cases where you use lift and shift. It's becoming increasingly the case if you don't know what you're doing that the overall bill is higher in cloud. And that's because of the amount of services you have now. So there are cases to, for example, just go ahead and put your database in the cloud. But it's more often the case that it's going to be a better fit to use parts and pieces or services that are more granular and a better fit and that requires more sophistication on understanding how to work with the cloud. So I often find that managers are people who are developers who have don't have a lot of experiences managers, I find that or I find managers who were not developers. So the middle managers that I work with often are kind of having skill deficiencies and so part of getting success in these projects is to help the managers with these five habits that I think reflect a successful technical manager. The first one is to get very frequent updates that can be short and make sure the manager is listening more than they're talking to encourage and support the manager in asking clarifying questions that help the manager to communicate to whoever they're managing, whether up or down. So act as a translator basically if the manager isn't understanding the technology. The next is involving all the way up stakeholders so they understand what's happening and a level that of detail they need and all the way to the users so that they understand what's happening and why it's happening. I mean I'll make an example from genomics. I tell every user over and over. I know changes challenging and it's disruptive to your workflow, but you're out of space. You can't not go to the cloud in this particular customer and just being that straightforward about it and explaining that the existing solution is literally out of space now and they're already queuing helps the users to understand why the change is important and helps them to want to participate in it. You want to measure deployed software and deployed software in my mind is all the way working for end users. And they want to be part of the technical training. Again, have had a really good week professionally. Not only did the end user run the pipeline on production after two years of nothing. The manager ran it. And I think the manager, well, I think they're both really happy but I think the manager was happier and more surprised than the end user. It was a good, good day. So speaking of managers, I pulled a page from my work as a data warehouse person, building dashboards with KPIs for business metrics. And a technique that I use to communicate with managers is I use code or solution dashboards, and this is one again from a current project. It doesn't really matter so much the dimension across the top. What matters is you can look and see there are two pipelines. It's DNA and RNA in this case. And you can see quickly the status so you can see the DNA pipeline is not as good a shape as the RNA. And as a manager, you can see this optimized issue is a blocker because it's got the two red X's. And managers need to have technical detail summarized in a way they can action. So red, yellow, green works.  So I another version of this is I will take cloud architectures and I will color them over the top red, yellow, green, like this section of the architecture doesn't work. So red, this section we are testing but we're not sure yellow. This section fully functions and we're testing as MVP with all the way to the customer green. Again, it might seem overly simplified, but you know, cutting through the complexity is part of a higher level pattern for cloud success. So, you know, I've been saying it all through probably I should have put this first. It's right up there with elastic services. You know this end user focus. It's an agile principle. So it shouldn't be too surprising, but it is, you know, key in my work as a technical professional in cloud.  But as I work with different teams. I mean, if I see this. And this happened with my current team, you know, it's a well known thing that the structure of the team reflects the structure of the software that's presented the conways law. I had a person who I literally have never seen her and I've been working on this project for six months. And it's not that she has children or dogs or whatever. It's just and we have this is the case with this particular client we have meeting after meeting after meeting where no one turns on their cameras. I mean they're checked out.  Well, the first thing is don't have meetings if you don't need them. So send emails or other types of communication. The second thing is get the group to be smaller and turn your camera on be that person. So build with the users. You know, I have I've kind of decomposed to this previous slide into having one on one working meetings with basically all the people on the slide. And, you know, when it's just the two of us, we turn the cameras on and we work together and involving users from the beginning for that MVP. From day one, even before anything worked on the pipeline, we built something using the technology so that they would understand the technology. So these cloud patterns for people technical manager and user focus. The third one is learning culture. And again this is, you know, core agile stuff but applied to cloud. So this is a pundit from Gartner Lydia Leon. And she wrote this again conveniently all the stuff is conveniently this month. And basically she said cloud skills gap has reached a crisis level in many organizations. The smell is when the manager or the team says oh yeah we don't really do training we just figure it out. Well you don't, because you don't just figure out 300 services. I personally spend 25% of my professional time studying, and I have for 11 years, and that's one of the key reasons I'm successful. When you are in an industry that moves as quickly as cloud. Having formal time for learning is part of the job. And if you don't have that you're going to have that 75% number. So what's tricky is to figure out how that implements for the team, because learning is evolving, you know in the old days we went to a class and sat in the room for eight hours well you don't do that anymore. And now you don't even go anywhere and how do you get people to pay attention when for their work they're already on zoom calls. So you know the key I think is interactivity. And so that's actually something I'm working on with this client. So this animation is crazy on purpose, because that's one thing I have found teaching cloud is not linear. It's more of a just in time. And I was trying in my sort of bad graphics sort of way to animate how it goes you know like you probably saw some of those boxes pop up and go away, because teams aren't ready. Right, so as a technical trainer which is some aspect of my work with teams, you have to have be embedded and you have to understand at what point, will they for example be interested in automating their deployments to use something like terraform rather than scripts.  It's too much. Another aspect of this interactivity is building a sandbox and this is something I'm working with on this big client right now because because they've had the pain of failure. They now are more interested in spending some time and learning. Well, how are we going to do it. Right. So, again, I like to be pragmatic. So this is from the earlier drawing of data mesh where I was trying to understand the services that could provide value, which is something I'm actually working on right now I'm trying to learn how to do it myself because it's a new set of services. Well, once I can make professional recommendations and share what I've learned, then I will as a next step, create a little demo that I can work with the team and say okay this is where I think these services can add value. We'll explore them a bit together and then once we get sort of a baseline, then we can figure out at what level we want to expand this interactive discussion slash training with the rest of the team and we can build a sandbox environment that they can then experiment with. I've learned a lot of information at you in this talk and I'm hoping that at least some of it or most of it will be useful to you in however you're implementing cloud because I want you to move from that 75% to the 95% but it's not without concentrated and thoughtful effort that you get there. I know I've been repetitive, but it's a pattern that if you take nothing else away from this talk, I implore you to get closer to your users. Yes, I work with the developers and infrastructure team and they are my partners and collaborators, but the people who I serve in my work are the genomic researchers, the medical doctors, and the people who then publish on the data. I work with the people who I focus my conversations and I ask questions like what do you dream about being able to do from a research point, because that's how we not only get success in cloud, but we drive cloud forward in our particular domain. We look at what's most interesting to our users. And to kind of kick in this with aspirational. This is a tweet, and this is very domain specific, but I'll just read it available in six new tarot workflows users can now analyze the whole genome in over one hour with this technology, compared to 24 hours in the CPU based environment and reduce the compute cost by more than half. So let's do great things together. So those are our patterns. And there's my article, and I like to give you something to learn with. So here's an repo where I have over 100 resources. And there's the address of the repo, and I'm Lynn Lang it. Thanks so much for listening. All right. Thanks a lot, Lynn. That was amazing. All those resources and the patterns. I've tried to capture them, but I'm sure I'll have to go back and watch the video again. There's a lot of information you were able to pack and appreciate that. Cool. Just want to see if there are any questions from the audience we could, you know, maybe spend 10 minutes, take the questions. Folks, if you have any question, please use the Q&A section and fire away. Don't be shy. All right. Maybe while we wait, Lynn, I'll give, I'll start off with one of the questions and I think we were briefly talking about this backstage in terms of, you know, more and more companies are realizing that the costs on cloud is getting quite high for them, especially as they start scaling. And maybe it's not a fair comparison to what they used to have us as all the benefits they get and stuff like that. So from purely from a cost perspective, if someone's still contemplating either moving to the cloud or is already on cloud and thinking of moving off the cloud, what would be your recommendation to them. It really depends on the volume. You know, again, my lens is very skewed because of working in genomics. Genomics really can't not use the cloud because of the extreme data volumes. It would not be practical. But now there are other domains. I think government is a really good example of this, where there's other needs such as extremely tight security. The cloud, of course, has great security, but whether they're going to use the government version of the cloud like Amazon's Gov cloud in the US, for example. But there are some cases that for security, you would want to have either redundancy or not on the cloud, some cases for financial as well, because you have the resources. So it's always an analysis of what is your set of business needs and, you know, where are you going to get the most value. You know, in the case that we were talking about in India before the before I came on, when you have the ability to have resources, data centers. When you have, you know, companies like like those that you're working with in India that have the infrastructure that people can work with. That's it's sort of a different case. I see less of that in the US. And I think it's because the public cloud, you know, originated in the US. So, you know, it's the case here, but it's more often it would be hybrid, rather than hybrid between clouds rather than hybrid on prem. To go to add on prem. Now if you have existing on prem to extend your on prem. Okay, that will happen. But I don't see that as often so that was like really interesting to me to hear about you know that situation in in India. Cool. All right. So Gilb is in the audience and he's saying it's a great talk thanks so much at many leavens Lynn, and he also, I think has a question or a session about, you know, and Tom earlier in his keynote and talked about kind of broadening the term to stakeholders as opposed to just users, because of, you know, we do serve multiple stakeholders. So he's I think he's just kind of probably question asking that would you consider broadening the term to stakeholders, as opposed to just users.  And if we don't think that way I think that's one of the reasons there's so much bad software out there, you know what I mean. Because, again, I just maybe it's the way my consultancy is but it is shocking to me how often that answer to who, which doctors which researchers whatever use the software, we don't know, we think a lot. We're not sure we wish more would. There's a question from Aliya she's saying what are the basic factors you would consider when you asked to migrate from on Prem to cloud native. So what you know, again, like any software project that you what are the what are your core, you know, business needs like all this I'll put it in genomics because that's what I know right, so I would say, you know what is the storage capacity of your solution right now, and what and what percentage is it increasing. So when are you going to run out of storage, how much are you paying for it. And how is that, is it exceeding your budget. What is the performance, I mean here's another example, you know what fails. So like there is a pipeline that this client that I have is running, and I asked the researcher, how often you have to rerun it because it fails because the cluster runs out of capacity, 50% of the time. So then you prioritize, because obviously you would prioritize that, because that's not acceptable, based on business needs, right, because again sometimes you have to explain. Well, if this. And again I'm getting very specific but I think that might be helpful for the audience because I was more general and talk. The pipeline fails 50% of the time and it takes one day to run.  Do you know what I mean. And so you have to do that math, because, unfortunately, and it's again it's one of the reasons I'm doing this talk. I have not seen the success of projects improve over my 11 years in cloud. I've seen it get worse as the number of services is out there, which drives things like we'll just stay on pram or we'll just go back to on pram because it's the complexities it's blowing people's minds so again this idea of like focusing narrowly on where you can get value, which is lean applied lean cloud. One, one other question for me from my silent is that in your experience working with the genome researchers and other folks that you work with, what has been like your hardest part of trying to get them to adopt, you know, some of the cloud patterns that you talked about to acknowledge that they have to spend time learning the technology because when you work with scientists, their brains are already so busy working on their research problems, and trying to keep up with their domain I mean they're going to classes all the time. Like the two that I'm working with now for this project. Like, we're going to classes about new types of, you know, genomic data and new types of sequencing, and we're kind of out of time we're not doing any actual work anymore between learning cloud with you and between learning about genomics, you know, it's interesting genomics because the sequencing technologies, well, like, here's a new thing, I'll just tell you. So then, one of the new things that they really want to do is, they're looking a lot at RNA, because RNA is you know what's expressed downstream and it looks, it shows you what is the effectiveness of the cancer right so you have DNA, which is what you inherit right and then RNA is like diseases you get. Well, the quality of the RNA data gets better and better and better and there's this huge thing it's called spatial multi omics, where you can now pinpoint on the tumor, where the RNA is, which I'm getting very detailed but it's the quality of the data is going like, and so you know the challenge is the data quality is just increasing dramatically. And these researchers have to understand like okay what can I do with spatial RNA data now right what is it how do I work with it, and then they have to deal with. How do I use the cloud, right, it's a lot. It's so it's a very, I would say they're right up there with, you know, quants with financial analysts, like it's a really, really front of the pipeline difficult job to be a bioinformatics researcher right now. It's rewarding, but it's very champ. So again as a cloud architect and cloud practitioner I'm like, how can I simplify.  Right. And that's not going to that's not going to help their cloud journey, whereas if they can click on the console. That's going to work. It's a simple things. Yeah, I think the context here is very important is having that empathy for your for your stakeholders or users in terms of what is that they're trying to accomplish and trying to provide them the simplest parts to accomplish this thing rather than just drown them into all the things that you are excited about movies like this new cool service, you know we should try out and you know that just maybe overwhelms them and might might and back to my question and that's where people may, you know, find it very hard to stay up to speed and just acknowledge that they need to invest time to learn some of these techniques. I will take one last question I know, you know, we kind of shooting a little over here but this is this one question I think this is gets asked a lot. And so I apologize for that but I think generally when when people hear about, you know, genetic data or genome information being available in the cloud. You know, there's a natural concern around the security of the data, and how do you ensure, you know, it's, it's, it's in the right hands of people. So we just love to hear what's your take on that. First of all, security is extremely difficult. And I find that most people don't do security well enough in their environments they control. I am not a security expert by any means but through my years, I have always been sort of a, I don't know, like I'll just look for the common sense thing like is, is your door locked, you know layer zero, right. Do people share passwords right. And what I find is that it security is so difficult that the cloud as a baseline is more secure than any on prem, because they have more people like Google and Microsoft to do it right. That's number one what they provide. Number two is broken record. As you have more abilities with services, you have to spend time to learn them. You know when you first started it was get I am you know users and roles and policies and you're done right. Well it's not that simple anymore. Excuse me, I mean you have, you know, different levels of VPC and different levels of network traffic. And then if you have a pipe you have to, you know blah blah blah you have to set up all this. It's a specialty, right. And so it's an area that if you have, first of all on prem knowledge. That's great. And second of all, I cannot find enough people like in my boutique consultancy, I have a few people that I will bring on for jobs. But it's again if you're trying to advance your career and you're interested in it. Spend time learning it get certified, learn all the nuances of it, you know work with security testing pen testing group so you can really understand how to respond to that make it your expertise. So the short answer is, it's more secure by default in cloud, but to implement it properly, do all the other things I talked about, get trained, be an expert, find out the business needs and balance. Because if you have this draconian security, then your end users will rebel against it and they will share passwords and they will and you'll have no security. So it's like any other aspect of cloud, whether it's you know your data storage or your speed of your application, you have to, it's a continuum, how much security is appropriate. So it's a skill combined with experience so that you can understand that, because if you lock things down too much, then you don't have adoption. Yeah, I think very well put I often keep joking with folks that you know this the most secure thing is, is something that's disconnected buried under the earth, and nobody. That's the most secure thing. But as you start opening up things you need to figure out like, you know, what's the appropriate level because it's a spectrum and you can, you know, bounce off to either extremes of the spectrum, and, you know, run into issues, one one side the adoption issues on the other side, you know, losing all the data and other kinds of issues, but finding that balance I think is is the key. And just additional point because this literally just came up recently in a client is, I thought it was very, very smart in cloud, the person the security lead said, we are not going to stop all the threats, what we're going to do is we're going to alert on them. And I thought that really shows a maturity that it that you know, that's what we can do, we can alert and we can respond, but we cannot because of the parts and pieces in this solution and that's not right for every solution, but I think that is a good approach for usability. The security and usability balance. Yeah. Yeah, cool. I think we've, I think I've answered, I've got picked up most of the questions here. And again, I think we just about running out of time but I want to thank you again Lynn for joining us this morning. I think the patterns you've shared and that the new term code smells, sorry, cloud smells from code smells. That's, that's pretty neat. Is this somewhere you're curating all these cloud smells, could other folks contribute to it. I actually came up with it for this talk, I was like, because I want to, and so we kind of interesting. So my sort of repository is, you know, learning cloud, right, on GitHub, and I have a section in there called patterns so that's actually thanks that's a great suggestion. Maybe I will do that will be like the, the Kevlin Henny of cloud. You can send me pictures. I don't know if I want to go down there. Anyway, thank you very much for having me and thanks to the audience. I hope that it was, you know, good use of your time. All right, thanks Lynn, and thanks everyone for joining in greatly appreciate you saying and with that we're going to actually wrap up the conference greatly appreciate everyone for being part of this wonderful three days. And hope it has been a great learning experience and a great networking experience for all of you. Goodbye. Thank you.


Assalamualaikum! Hello! I'm Daud K. Do you know Sillat? Sillat is the fourth-largest city in Bangladesh, which is famous with soupies and tea heart. Then today, I will show you the beauty of the Sillat. Let's... I'm a Sillat Achi. I'm a Sillat Balabai. Let's go! Hey, Assalamualaikum! How are you? Assalamualaikum! Welcome to Sillat. Thank you so much. I'm a Sillat T4.  Let's get it. This is a famous restaurant here? Yeah, yeah. Paz Bhai. Paz Bhai? Yeah. Oh, I'm so hungry right now.  So, here is really famous restaurant in Sillat. Paz Bhai. So, let's try. I'm so hungry, so it will be super, super delicious. Let's go. Dry fish, sea dry fish, river dry fish. Oh, so Bangladesh people like rice and fish together. Rice, fish and dal. Oh, dal. Lentil. Lentil is dal. Mace, patay and bangkali. Let's try. Yeah, let's try. It's really good. It's really good. Oneg Muza. These are. These are. These are. No meal. These are. But oneg should have them. So, Alhamdulillah, it was really good meal. And you know, Sillat is tea half of Bangladesh. So, I want to taste it. It's for you. You can let's try. It's really good. It's really good. Organic tea in Sillat. Assalamualaikum warahmatullahi wabarakatuh. Hello, everyone. So today, I came to the Sillat. And I came here. Hazra Sa Jalla. Lama Tula Mosque. Which is the most famous place in Sillat. Let's go. I'm so looking forward to it. Let's go. This is so huge and big. So many people come here. So, this is the top of the hill. The Sheen Center. Hazra Lama Tula's graveyard. Ah, graveyard. Very interesting place. So, I've heard this is kind of Sufi? Yes, this is Sufi pilgrim. Sufi pilgrim. So, this is so old. Older building. Older building. Like 400 years ago. It makes you feel very peaceful. Interesting. Okay. Okay. Wow, awesome. Yeah, okay. So, I want to say something shortly. So, inside I couldn't film because they don't allow to take the videos. But I saw something a little bit strange. Some people pray in front of the phone. And it seems like they wish something. And they even give some money to the phone. So, I thought it's a little bit strange. So, I asked Imam. And he said this is kind of bidah. Making a new thing of Islam. Only wish and ask help to Allah. Not for certain person or scholar. Because Islam is already perfect. So, we don't need to less or more. So, that's it. Then let's go to the other place. Let's go. Let's go, let's go. You are Hafiz, right? Yes. Alhamdulillah. You teach Quran? Yes. In online? Alhamdulillah. From Bangladesh to Texas city. Texas? Yes. Alhamdulillah. You are very nice. Masha Allah. Amir victim. Thank you so much. Thanks to Allah. Wa a'aikum salam. Thank you. You're welcome. This is my friend. How are you? You look so good. Masha Allah. Masha Allah. You look so good. Thank you, Salam. Peace be upon you. How are you? And for a moment, we heard a little bit about Madrasa. There is none worthy of worship but Allah. Muhammad Sallallahu Alaihi Wasallam is the messenger of Allah. He is the final messenger. He is not only messenger. He is the final messenger. Allah is the only created.  Thank you so much. Why not? This tea garden is so big and so huge and that's one of the reason that Bangladesh has a wonderful and tasty tea in the world. Wow, so huge. Wow, this is a tea garden. So big, so huge and the air is so good and smells so good. How much big is it? I don't know. I can't see it. It's endless. It's all the garden. It feels like prince and princess. So we came to Shadabat Torot White Stone and the liver is so big and beautiful and there you can see the mountain and that side is Indian, not Bangladesh so this is really interesting place. So we will take the boat in this liver and let's go. I'm so excited. Let's go. Hurry up please. The time is short. Let's hurry up. Hurry up. This is the boat in Shadabat Torot. Let's go. Let's go. Let's go to India. So today we came to Silat and which is the most beautiful and also very religious city and there are so many beautiful nature here so I was really happy and thank you for introducing beautiful Silat. Very very good. Thank you for watching my video. Take care and be safe. Thank you. Bye. Islam is not only that accept light in the name of Allah. This is the word of Islam but the main means of Islam the heart of Islam is the practicing. Practicing and salat and making life with the Sunnah prophets. Allah said in the Holy Quran so many times so please memorize this word fair Allah subhanahu wa ta'ala in every time if you fair Allah subhanahu wa ta'ala in every time Allah will reward you in this dunya and help you in your life. Peace be upon you. Peace be upon you.


Hey So I'm really I'm going to talk about some gen work with hotek and Daniel that we call like an extension evaluation So a standard problem at script. Oh is to have a general chair say tall that wants to distribute socks among Sox among attendees The problem is that we're not happy with just picking socks We want each of us to have the best circle available So after years of research at IBM they managed to produce some algorithm the best possible to pitch The best of four each of us. It turns out that the algorithm is pretty weird first the Yeah, the input to the algorithm is the sorry is the dating history of the attendees So it's pretty weird, but it turned out that this was the best option possible and also another problem was that The algorithm is actually extremely complicated. So it actually requires IBM supercomputers to run it So only talk and I can run it So first she puts at the registration page She asked every attendee to put the dating history, but for some reason people didn't want to do that So as cryptographer, how can we help? So we introduce a new primitive that we call iconic function evaluation Where we have some private data x and we want to to let tall compute f of x on our data So using like a new function evaluation tall can Compress the huge algorithm into some short digest that can be seen as a public key for the function and Your regular crypto attendee will be able to use this public key to generate a safer text of the Of its dating history after that how can recover the cipher text and using her knowledge of the circuit Will be able to recover the best socks so she can put the other so that's nice But more formally what happens is that? We want pretty much or Primitive to only to let tall learn about ffx and certainly means that she only Learns ffx and not the whole dating history and For the iconic part as our name suggests we want the cipher text to be succinct in the sense that Encryption time should be smaller than the Set of distance to our computers. We are not at IBM And we manage to construct such a primitive from LW more seriously Application I'm pretty excited about is on MPC where we have and parties with input X1 up to Xn and a large function F The question is whether we can compute Such a protocol to compute f of The X1 up to Xn in time much less than the structure So that seems paradoxical at first because well you have to compute the function at some points But we actually managed to achieve something somewhat similar Where there's a first an offline preprocessing phase? so for the picture above Disconsisting compressing the function f and so everybody can do that offline. This is costly, but we don't care Then there's an online phase where people compute the ciphertext and that's our laconic Property suggests this is small. So the online complexity of the of the protocol here Of the part here is small and then people can individually decrypt the Decrypt the ciphertext. So in the end that this just gives an MPC where the online communication is small But that I think is pretty cool so you can More about that on the imprint or catch either hot take Daniel or me if you have any questions. Thanks for your attention


Yeah, we on boss talk one-on-one a lot of ideas. It's a lot of we haven't we haven't we haven't even got a dope horror Movie, you don't have a dope you're right Western.  It made it You when I watch it one time one time years later on and I'll watch it again. I can watch Me too me and you the same He's the first cowboy black cowboy from the plate it like you know And he's straight out of 30 minutes away material Hey, listen man that that movie there. No, no trust me That's what he did Ray and we that is how I love I was coming out of universal I looked at There's a big thing said Ray And I was like, I thought it was Ray. Yeah, he's like, that's Jamie So I went to his house that night and I was like niggie about to get a Oscar You knew it when I saw the poster Because I thought it was Ray He didn't Jamie as Ray. Yeah, he killed that.  Oh, yeah His first day in comedy was in San Diego a place called Smokies. We always talk about. Oh, yeah. Oh, Jamie You know Jamie did comedy, right? Yeah, like I like they know tomorrow. I love it It's more of a used to a joke about it His grandma Anybody walk like that got to have a Anybody cough like that got to have a I mean it was just Yeah, brilliant. I know Jane for a long time. You use at them parties. I heard about the parties I never got to go to one, but I heard they was off the chain. Yeah His best friend live down there and he would come into the shop for you seven shopping I'm sure I'll come in in there and buy like a shirt. I'm like, yeah, I'm going up here to go see Yeah, nigga. Good luck man. God day. You might see anybody at Jamie's house Anybody anybody, but we've been going there Since you know, here's a great personality. Oh, no, he's good dude. He just it's solid He's solid. We know as Eric, right? Yeah, that's the question I had on that's you why'd you change the name?  Oh, you shorten it. Okay, what is your phone? Cuz it's easier to say It's just long, you know Richard Pryor's naming Richard Pryor. Yeah, no, y'all always change That's when y'all hit y'all start them when you change the name.   I need one level but I need one level. I just want to save That's it Yeah, we on boss talk