has like three questions and we've got like, now we're at 15. Hello everyone. I'm Dean. I work at Status. I'm a security researcher. I used to do the entire auditing thing. I haven't done it in years. It's going to be super interesting to see how this space has progressed and I have a really competent panel. So I think it's probably best if everyone just goes around and introduces themselves, maybe says what they're working on and what some of the most interesting things that have happened in the past year in the security space have been. Maybe events, encounters, changes, things which have progressed in the right direction or things which we were still seeing which we didn't expect we'd see one year old. Hey guys. My name is Gonzalo. I'm here representing the consensus diligence team we're part of the consensus but we're also squashing bots. We've been squashing bots for the past three years, I guess. I'd like to say that a lot has progressed but we were just talking outside about how it kind of has not. I think it's normal for stuff to progress slowly. Even as we mature as the ecosystem matures, things will start progressing but it's still slowly, right? I'm here to talk about it. I'm a mountain monster vendor and I work with Ethereum Foundation with security staff regarding Ethereum and not so much smart contract development through auditing but more the entire ecosystem and the entire infrastructure of Ethereum and so last month I thought the big change in security scene was that the security awareness and the entire security community around Ethereum had really started to grow. There's been this ETH security group which really become large and there have been a lot of new companies formed and building new cool tools for analysis of smart contracts and this last year the evolution has kind of continued on a path and the tooling has become better and there is more tooling and I think that the maybe biggest change in the last year is that all these security folks who started coming to the scene through smart contracts then also started looking into the base there and looking into actually how the EVM works, how protocol works and they've given inputs to the hard forks and the potential changes in the EVM and things like that. So that's a good development I think. Hi, I'm Yohei, I work for Monstown, I'm based in Tokyo. I bring audits and also kind of the life-changing security consulting for more enterprise-focused solutions. Over the last year I think the manual audit process hasn't changed that much. The tools are getting better but that's still kind of a bottleneck to how we can kind of transition from manual audits to fully automated and I don't think it's going to happen for a couple of years or maybe a couple of decades. It's just a very hard question. In terms of the stuff we've seen this year, I think a lot of stuff related to gas costs. Obviously there was the concept of the hard fork which was kind of delayed because of some changes related to kind of gas pricing. There's been a lot of stuff that we've learned as a company. One thing that we're kind of focusing on is how we mitigate risks even after they happen. So as a security company we're aware that nothing's perfect and all our clients and everyone should be aware that even if you get a security audit, nothing's going to be perfect. So people are I guess becoming more aware of that and trying to plan for what happens if actually a known vulnerability comes up or an involuntary vulnerability comes up. We've been kind of fixing our kind of rethinking through our audit process and how to kind of learn from our mistakes. So I think that's been kind of a good trend. There's been a couple of pretty big security vulnerability findings in the space here thanks to kind of like the security groups. And I think just as an industry in general we're kind of being more proactive and kind of responding more rapidly to kind of these vulnerability findings. I think that's good and we'll continue to progress. Hi, I'm Aleppo. I'm with Open Disabling. We do the tools that people know as form contracts and we also do security audits and we do security research team. And I agree on the tools thing. I think there's been a move forward, a push for more automated tools and these have been developed a lot recently. I also think that there's a bit of over reliance sometimes in tools and what you just said that it's going to be a long time still reliant on them fully. One other thing maybe that we saw is a move not only to a lower level but also to a higher level like incentives and for instance we had the Compa audit this year which created a lot of controversy because we found that actually they could like withdraw entire money so there was a question like should we report these kind of things? Often times we work with no clear threat models so it's like you have to be conservative or the threat model and like in what we report over for our own so I think it's, and sometimes like it has gone past we look at the incentive systems for some reason but we think it's super important and so we do it anyway so I think that's an interesting direction which it has developed a little and we see that we want to see a lot of development for it. Hi, my name is Subin. I work for chain security also doing audits and as Martin said I've been looking a bit more into low level security also this year and yeah I totally agree with your last point that you just mentioned so I'm going to pick it up there so we also see a lot of fights with people over trust models with clients on trust models and so on so like that's always a place where there's tension because the client never thinks of themselves as malicious so that's of course what decentralized can bring to us so that's very interesting. Otherwise in general in terms of the year I would say it's been an interesting year, I would say last year Defconn OpenSephyn released or really announced Defconn OS and I think we really felt the impact. We are seeing a lot more delicate calls than we were seeing one year ago due to all of the proxies and so on so that definitely has happened and still of course we're seeing a lot of bugs but I think we're seeing bugs on a different level now and at the Lightning Talks there was an interesting graph how now the average call depth of contracts has gone up a lot like we're seeing contracts that have call depths up to ten deep or something like some contract that calls Kyber Network then later calls Compound and so on like the complexity is going up a lot so I think security has generally gone up because complexity has also gone up it has still sufficiently many bugs so I would say Germany it's been a positive year. Yeah so with the whole call depth thing that you mentioned I think what we realized there is that a year ago a lot of us were mainly auditing ICO contracts and token contracts and now people are actually building those products and I think some of you here have probably retained some of the people in the ICOs that you were auditing and are now actually auditing the products which they're shipping do you think that they've learned from the things you were teaching them a year ago and the general maturity of their code bases is going up or do you think we're still making the same mistakes? I definitely think that they've learned yeah so I think that I mean for example thanks to the great work by OpenSatman that a certain standard has been set and like it's become a lot easier to write good code and there's kind of like there has been like kind of a template for good code now and that's why I think in general it hasn't moved. Yeah I agree and just like to that for instance we could have like a guide to a checklist for what we expect from the code before starting an audit and I think this has worked and so we still didn't see a code in a much more material stage that we used to see before. What are some of the things you throw in the checklist what do you expect from your clients? Documentation, we put a lot of emphasis in the documentation testing things that talk about the health of the code for instance and we had the solidity and compiler audit last year and a lot of the issues we found were like the bus factor there were two people working on like a language the bus factor is like a factor or like how many people have to be in a bus that rushes for the project to end in time and it was like two guys running this whole industry essentially so this kind of things we now have in this checklist and I think the purchase is doing like a much better job at assessing those. I think the bus factor is probably something interesting for you as well from like what things that happened in Shanghai when everyone was on the same flight and no one was able to respond to things but as the EF kind of like developed new strategies to ensure that if things are found domain that there is always someone who's able to respond or are we still in a two man, three man point of failure there? So there is no like structured way that it's ensured that there are always people on board but as all of it is pretty spread out and our developers are pretty spread out but if an earthquake was to take out Europe then yeah, it would probably be pretty bad for the man. But there are still some people in the USA who cannot vote so there are probably some candidates in that case. Well, I'll speak a little bit to the maturity topic. We've seen people write, better go, people sheet products that we audited their ICOs before or not and I think it's also due to the bear market to be honest everybody was in a, there was a clear look like nothing was happening and people were actually building and it was a meme but it was also true like people were building products and now they're sheeting those products and we see maturity, we see a bigger call that that shows in the code quality that reaches us as a company the audits that we, the clients that we would engage with like they have good teams and like that shows in the code. I kind of want to talk about templating. Obviously Open Zeppelin and all the work around templating has been a major contribution and I think it's a great path to continue down. At the same time as the contracts become more complex there are these custom, there's these features that become more general that people want to use but might not be available on Open Zeppelin so at the same time we've been seeing a lot more copy and pastes of these very complex functions and then when a vulnerability is found in one of these and you just find these cross-contracts so being able to kind of maybe, I don't know, submit these kind of complex contracts for more audits to happen and kind of turning them into templates might be one way to move forward but I mean it sucks that the first company to actually develop it has to kind of carry on the burden of paying for the auditing and the people who just copy it are basically free writers. A lot of people like to use templates but not contribute in any way to it. Yeah, not just templates but like other people's code because some contracts are kind of open source so people just take what's good and move what's working and then they don't have to pay for the security afterwards. So I mean that's a challenging problem to solve kind of distributing the costs across the first person's market and everyone who just copies the code but that might be something to kind of think about. Okay, so now that we've kind of moved away as said from ICOs, from auditing simple token contracts to more complex systems how has that affected your practices? Are you guys finding yourself using tools more in addition to doing manual audits or are you focusing more on doing manual audits or what's changed in your workflow over the past due to the changes we're seeing? I think having a solid understanding of the business model and what they're trying to achieve has been a crucial part of the audit process. Just understanding the business logic and figuring out how that's translated into code has obviously consumed a lot more of our time. If it's like a simple token contract or an ICO contract then it's fairly straightforward what kind of features are available but once the contracts start talking to other contracts and other projects then it becomes a lot more harder to kind of map out what does what and that scope gets a lot bigger. So I think in terms of the stuff we did last year now the kind of unwrapping of the project has consumed a lot more time than it did last year. I can touch on a topic that we were asking Martin before and say that we, like I personally found out that we were, or that we were not that prepared for instant response, honestly. Like as people have shipped more products more stuff has happened like in real time, right? There's a lot more and the groups that were in the security groups that were in like this like people are always discovering stuff and like incident response has become more and more important and I've honestly found out that we're not that prepared that we need really, really, really fucking needed the step of our game. I think for you guys we're also at the security chat. We've noticed that there's people now like shout out to Sam, whoever he is finding bugs every three days in like major all the peace here, like you're amazing. I don't know how you do it. You need to sleep a little more. But I guess with the incident response we can learn from two companies here who've done audits and all of us miss things sometimes for zero X for consensus diligence that was the zero X audit and for chain security it was DAO stack where both of them, I don't remember the DAO stack vulnerability the consensus diligence one missed a a return size check in the zero X contracts thankfully none of the trades were affected but when things like these are discovered what have you learned from that how has that affected your practices what will you do further on to try and minimize that? Yeah, that's also a very good and comfortable question we learned something so that's good and mostly it was processes it's not like Sam is probably an AI I don't know how we got that that's good that's good that he's a good guy and not just exploiting everything from that but internally we discovered that we needed to again we needed to up our game like in processes process wise and mainly in a very specific part of an audit which is mitigations so what happened like for this reason in the static call it was like a problem with the call data return and we discovered that what happened was that we were not treating the mitigation space like the part after the audit after we provide a report after we provide the artifact to the client they obviously fixed the bluff that we find the vulnerabilities now we were probably not treating these mitigation space as a new audit which it should be because like it's new codes so it's a completely different thing that you gotta check I guess and it's a new audit it's just not fixed it's right it's not like new code is new code that's it so that's mostly the most important thing that came out of that that we learned internally is that new code is new code there's no two different ways to treat it there's no mitigation you cannot care less that's just because it's after the audit that comes after the audit that's finished basically so we are now just realizing that we are like we all collectively have been capable of just like to like share these teachings and learnings honestly so we've learned a lot from all the other guys in the community as well so there's been quite a lot of interest during the last three years from my part and what I can say about those is that they've all been different and they can't really we can't apply a template okay so we have an incident to do gather in this channel and communicate with these people because they're all comparing so widely like for example if we can't discover that there's a vulnerability in the Solidity compiler then that's kind of how you privilege information and first a couple of poor people need to find out exactly how what triggers this bug to appear and kind of scan the the entire office of the contract and see what are affected and how bad is it that it actually is going there and can reach out to them approach it by the way it's a really good idea to put some kind of contact information in your verified source code and then we have these other kind of instance where there's vulnerability in GAP which is just a GAP theme where there's like apparently have a number of wallets which anyone can take ownership on where there actually needs to be a group of people who are willing to basically exploit these which can be seen as kind of invasive and and in that part this white hat group has done a lot of good for the community in general the question was also very yeah so a very good question so for us it actually was what was mentioned before trust model so the trust model wasn't entirely clear which contract came for whether that contract can be replaced with an untrustable contract and that's more cost and so yeah it's already well extended it's definitely a matter of processes making sure that trust model is clear making sure documentation is clear making sure that business model is very clear which contract can be replaced with untrustable code which contracts cannot be replaced with untrusted code and then yeah it was a very subtle bug in the end as with every bug when you see it then you're like how could we have missed this but there are just two repetitive calls to a gator which then which is the constant but then of course you can limit it as another constant any more than everything breaks yeah I think a general lesson too is we are all leaders we will all miss bugs eventually so what another means with that another is not a guarantee of security but just an assessment of security we will all miss bugs eventually and yeah so I think this is an important message too that it's important that these things are visible that we acknowledge them when they happen and so that people know what to expect and I don't think that they're absolutely safe because of the paradigm and security I'm going back also to your previous question and I think too deep on the the things we do have on the third ball I was going to say something but I forgot so sorry so speaking about how audits aren't actually a guarantee of security it might be interesting to hear from you guys what is an audit actually and who are they written for so with crypto it's kind of with this industry which we're working with it's kind of interesting where we open source a lot of our audits and that feel like Martin coming from a more traditional thing before Ethereum how did that work there what's different here and do you think those differences are good for the user no matter like should users have to care about a company having gotten an audit is it really that important who's really the client of one of these audits right yeah and I think it ties into what you guys were talking about earlier so to give some context from the regular security industry it's very clear who is the customer or the audit it's the company who wants to be audited and to be you go in and you do a security audit and it's highly classified and you deliver it to the customer and the customer there might be probably only like five people who read it and they probably don't spread it internally because they kind of want to forget all about it but they want to fix it and then later on you confirm that it was fixed and it's a dream of like publishing it and then in this open source world it's kind of expected that these audit results are published and in some cases there's even kind of an expectation that the audit should be phrased in a way that really describes how good this thing is I haven't done audits for the last couple of years I mean in smart contracts but before that I noticed several customers who like wanted to pick out some snippets of the audit where it was really positive work and that's not how I ever wrote the audit so I was put into the audit that this only highlights all the bad stuff I've had that experience where I've written and noted and people came back to me to say that they were sad about the audit result because it didn't highlight the things which they did good so it was like I don't care about what you did good and I care about what you did bad and that's the only thing you pay me to highlight I'm not going to give you two thumbs up on like wow your C20 was really beautifully written exactly so I think and I think that's a good thing to smart contracts even to the open source world but it's better not like Pat's on the back saying you did good it highlights you suck here and here and here and that's it I forgot what the press was I don't know if it's going to take a question do you want to take a question? I have a question so how do you guys identify the vulnerability so how you can make decision that this piece of logic is definitely bad the security part and this is not yeah I guess the question was how we identify the vulnerability I would phrase it quite concisely I mean as has been said at the beginning you need specification and if this matches the specification it's a bad probably also a lot of experience I think a lot of us have looked at code over excessive amounts of hours and at one point you start to get a feel for which lines start to look dingy and then you start analyzing those lines and then at one point you're like yes this is definitely dingy but it's probably for most of us it's a multi or multi-stage process it's definitely about expectations right the mismatch of expectations of the specification then we will try to match the code to that specification like it's about expectations like your expectations if you don't provide us with the specification we will have to come up with one ourselves and this is what Dean was talking about we look at code a lot so we try to mainly just like we try to come up with a specification that we think you want but that might not exactly be so yeah just write your specifications so we would know what to check for otherwise it's really really hard for us to do the job that you want us to and by specification you mean the white paper I need documentation of any sort I can talk about from one part there's this high level thing where you have these specifications maybe written in like flow or business use case kind of thing but then what Dean was referring to I think is more like the lower level assumption you can look at just a snip.code and see what are the assumptions being made here here's an assumption that this value will not be negative and here's an assumption that this loop will end on before a certain time I ask you because I'm also to secure the audits and sometimes that happens that that contract is without any bugs so it's beautiful it's written very good quality code but it works different than something that just write in the white paper so is this a bug or not yes so here's how my process used to work when I was still auditing what I would do is I would expect a specification from my client and then I would go off and look at the code and write a specification for myself and then while looking at that code I look at general bugs and everything and I write my specification and then I compare it to the specification which Dave handed me and I try to analyze do the behaviors which I have found interact with what they've given me and the reason why I write my own specification before reading theirs is because I don't want to cloud my judgment by their predefined assumptions so I go in completely blindly I do that I look at that and then I go off of that and I continue off of that so I think for everyone it's probably a multi-phase process which starts off by just first it's the quick scan for the stupid bugs like stupid calculation re-entrancy stuff like that and then you continue you always go one layer further just to quickly follow up I think that really ties into your question before who's the audience of the audit because actually what we started to do is the whole ICO hype that you mentioned when we started doing a lot of audits was we always clearly wrote what the token does or what the ICO does like for example how many percent of the tokens does the creator get because that was sometimes not so clear on the website let's say and we found this was really important for the people and also if we don't write it in the report even though it might clearly be said on the website now it might not be on the website anymore tomorrow and then people will say like oh but you said this is correct as is and then so we found it's important to document certain core features because to us the general public is also like an audience of the audit report I think it's stuff I agree with everyone but one thing that I think auditors talk about is the centralization issue and how sometimes it's not documented in the white paper and if that's something we should highlight on these reports obviously I think the reports are kind of targeted towards the community in some way but not it's very technical so it's very hard for someone who doesn't understand solidity to kind of go through the report and see how good the code is obviously we're pointing out where the mistakes are so it is going to sound negative but it doesn't mean that the code is absolutely trash even good projects have vulnerabilities that might have an address too and we phrased it that way so there wasn't vulnerability we fixed it but readers might think that oh it had a vulnerability in the first place it might not be a good code so kind of managing this vacations is kind of also important we try to do that we try to make it fair so we point out the mistakes but we also address how they fixed it and obviously kind of like the centralization and also kind of upgradability so the thing with smart contracts is that it's not supposed to be upgradable but there are ways to kind of implement upgradability so in that case the code might be safe for now when the audit was conducted but we can't actually guarantee that in the future so we tend to kind of talk about that kind of work with the clients figure out how to phrase that what they expect and how they can mitigate vulnerabilities that might come across from these upgradability and centralization issues one thing we started doing recently and actually in consensus we were doing it before us and we would like to make a summary post aside from the full report which is less technical and putting a summary of how the project works and things like this bread model we either have right by the client or we assume and a summary of the main vulnerabilities we found and then leading to the actual report where that's the technical so that's a much more digestible thing and thanks guys for coming up with that because I think that's very useful and it's a good way to inform the community without having the technical high bar of reading a full audit report yeah in regard to question earlier which I forgot to actually answer so I think it belongs to good form and should be included in the security audit is basically the trust assumptions of a contract so there is an admin who can do whatever he wants and there are these kinds of admins which can do these kinds of actions so if the report is published then that should be published with it yeah because it should be communicated to the community and I think it is the responsibility of the auditors to outline the trust model I do have a question for you guys so when you talk about the trust model do you highlight that even the specification of the client says that that's what he wants do you feel that there is like an ethical obligation to auditors somehow is that like where you guys are arriving at is there like an ethical obligation to flag that even if it's the specification that's the way they want it yeah I think so because it also yeah highlights that this is how it works these are the built in costs I think there is a question over there so I just want to go back to the public reporting it might sound a bit cynical but there is I think an incentive to disclose every bug obviously on behalf of the auditor especially since these reports are typically a good PR to fund a lot of bugs and they are very severe so I think there is an incentive to exaggerate the severity of the bug or maybe explain it in a way in the report that makes it seem like it was more difficult to find or anything and also if that incentive does exist how do you manage your relationship with clients so they don't proceed this I think the easiest method is not having any marketing people on the team making sure the reports were written by technical people yeah but actually that's not a new problem this is the same thing from classic security and what typically happens is that you deliver a report communication with the client and they say hey that's not really high because there is only like blah blah you can only access the something that's not important and then you can say oh ok I might never put it out to medium so this is and then you deliver a final report that's usually what happens but those reports were public right in the classic world no they aren't public but I mean it's basically the same back and forth you agree with your clients and at the end of the day it's up to the auditors to deliver a report and it's up to the client to decide do we want to publish this or not and if we publish it we can make no changes to it but we can say hey we think that the report actually sucks because we don't think it's that bad yeah going back to the severity classification we try to be as objective as possible like if someone else who wasn't involved in a different process looks at it how superior what they think it is so we do have I think every company has kind of standards on what they classify as high, medium, low informational at the end of the day it kind of depends on how much impact it might have on the end user the people who use the smart contracts so there's going to be significant financial loss and obviously it's high severity but again we do work with the clients to figure out if this will be exploited what would be the impact and how easy it would be to kind of exploit it we don't take it into account like how far it was for us to discover it I think it's more about the impact on what happens to the users so it is a battle we do upgrading and kind of downgrading severity during the other process is common and it really we try to be as objective as possible where are the mics I think there's a question here I think I can, my voice carries two quick questions one is around the tooling and in relation to that with like vulnerabilities that appear only once the contract has been running for a long time so what what type of tooling it's like with acts and diligence what other tools are for you guys using for the security analysis on the one hand and then the second question is around the crypto economic security analysis do you actually do that as well because you have behavior and contract that for example if you have likeness assumptions around staking mechanisms do you and then if it's not fully specified or there's something that's missing then you can't know that it's actually something that there's a plug because they don't check for example for block number I'll say that it's also part of like the conversation with the client not the tooling part, I love that plug yes me that's this great your tools are great there's a lot of great tools in the space though I agree that EVM is also getting open source this Thursday that's an example of a very good at the end yeah sorry not the EVM yet but I don't know how to pronounce it yeah there's a lot of great tools or tools are good but yeah answering the other question about the great economics sense about it I think that comes with a lot with the client normally you'd ask you can see from their specification what parts they want you to focus on but that also comes with that we actually ask the client what do you want us to focus on like what parts maybe it's just a code, maybe it has nothing to do with the economic part maybe they just want us to model the economics behind their smart contracts I think it varies a lot basically so one interesting aspect about crypto economic security which I think is largely overlooked is the fact that transactions float around on the network for a while where they're actually included and people can make bets against what they see is going to happen and investigate the transaction to know what's going to happen and of course do inject transactions before certain events until Diane is giving a talk later today about other bad things that can happen like for example it can be profitable to reorganize part of the chain and move around with how events play out in a way that even though it costs you a lot from the mining perspective the actual value you can cash out from that attack makes it feasible I'm curious to hear about that I think that plays into the fact that we're now building all these protocols and there's all these externalities that in classic security audit probably doesn't need to look at you being one of the main people at EF security who knows a lot of these things on how the protocol works do you find yourself getting more questions from people building things on top of it or do you think you'd like more interaction with people building things on top of the EVM and on top of Ethereum in general to come and ask about these externalities to learn about all the things that can go wrong outside of just the isolated smart contract and yes I definitely think that I mean as I said earlier a lot of the security folks who came into this place to study smart contracts have gradually turned their eyes towards the rest of the infrastructure and that's really great and I think it would be a good thing if more people in that developers and people who wrote smart contracts know more about the infrastructure as I hope Yes I think to that as a plug to anyone working on smart contracts and stuff there's this great telegram channel with a bunch of us in it a bunch of security guys a bunch of adaptable developers called Ed Security and it's easy to join and you can always get opinions from multiple people ranging from not just Martin but trailer bits is in there we're all in there so see if something looks a little weird you can usually go in there and ask and someone will be happy to help and explain from top to bottom what could go wrong so even before and on you can probably go in there and check that out Just a question about that is it a good idea to have anyone be able to join obviously there are some sensitive stuff that are being disclosed there say if like a black hacker joins it's all the stuff that's being talked there I'm not sure what the best approach is to sell that I think it's already above 200 people in it so it's all out there so interestingly with that entire black hat philosophy trailer bits is usually very sensitive on how much of the audit they release because they're from a very traditional security standpoint where they believe it shouldn't directly be open sourced whereas when I was auditing I had the philosophy of if the client doesn't want me to open source an audit I won't even work on it I think in this space because everyone can look at your stuff already if there's a bug someone will have probably already found it especially as we start growing I believe that if there's a bug someone will have found it already they may have just not exploited it yet so if it gets out there's things on like on protocol level as well where there was the Robston hard fork where we had to very slowly it was Robston or Rigby I don't remember Robston where Parity had enabled a feature that wasn't supposed to be enabled yet and there was a network split there was also that problem of how do you start to slowly communicate to all the actors in the network to start updating their nodes on live because it was something that could have been on live and I don't think there's any way around that that person could have easily done it on Robston and then done it on Mainnet and we didn't know who the attacker was to make that transaction and then we actually found out but there was some researchers who were trying out some Casper contract things it was totally innocent but it could have been something completely different you try to keep it silent but I think the main point is acting as fast as possible and I remember being on that and I was quite hectic we acted as fast as possible from contacting my crypto first to update their nodes to slowly get into miners and then get into exchanges and making sure that you keep things vague enough while you can but just ensure that people are actually doing their stuff that it's not like yeah I'll update my node tomorrow I think all the cases were just starting now are cases where the thing is live so I think it makes a lot of difference if it's live a protocol or smart contracts or if it's something to be released and in that case maybe it's much easier to publish the report like open it up for the community because there's no actual risk yet You guys heard what the Joker did in 1989 Batman movie this plot was about not teaming any one chemical but making a series he was a chemical genius and he had a bunch of different products that all essentially were saved by themselves but as they came together sent you into a neurological death spiral so I think about composability with this model that we're talking about now audits that happen on an atomic level they can pass but then people are bringing these money legos and these blocks in such a way that there are vectors that may not have been considered because use cases weren't anticipated and right now it seems like there's not obviously anybody that's looking at it you know when people are putting these things together in an open source way and I'm wondering if this is something that concerns you you see people building on top of audited products and assuming that because they're safe or they're audited in their atomic way that perhaps they're safe in a money legos tower yeah and I think that's a good question and it actually comes very much back to trust assumptions which have been mentioned a couple of times so I totally agree with what Martin said before they have to be clearly mentioned in the audit report and if you see trust assumptions in the audit report that's been luckily brought this up this year which they've come up with and made a really good point there which I think really has a lot to our whole community and so you see the trust assumptions so I think I should kind of answer this question on two levels I think on the smart contract level first of all if you assume that it could interact with anything else then of course compatibility will not always be there but should be there and if you clearly document the trust assumptions then people can see how it should compose and how it shouldn't and on the other side there's also there's also the law level protocol which where I also find this interesting I mean we now have half of coming up with six EIPs and then another one with let's say six or eight I guess entirely clear but there I find it even more interesting because you clearly have this compatibility problem because now we have cases where one EIP influences the other and by themselves they're fine but then you put them all together and they're not so fine anymore so I think because these are closely connected they're composable piece not so clear for smart contracts generally find it a bit easier to think about composability you get that in the traditional info sex space as well a lot of the time you find a tax on like servers and find backdoors it's not actually anyone's software that's a problem it's like a configuration fuck up between two things that leads to that problem so I think we all need to get better at investigating use cases and testing against those various use cases to try and find those problems and that's probably one place where tools can definitely help out where you it's just there's a big world out there and we can't look at everything yeah I just want to draw that this happens at all levels like for instance in our open seven contracts we have like a system of inheritance where you build the things out of smaller building blocks and you have like a very explosion of possibilities and of course you cannot audit all these possibilities and that's like for instance if you do multiple inheritance you have a linearization problem and if you do it wrong you might have a problem and so this is not only like assistance could work on legs but already in the building blocks one has to pay attention I'll just say something really little big we've seen that happen we've seen that happen with ERC777 for example just recently we see that that gets like some re-enterancy problems and we do that every day we do that every day with protocol using the protocol is right and again to that earlier point about 0x the static call we assumed like our trust model assumed that that was okay and it was not like really overseeing the thing that was happening below in the stack even not only about smart contracts but about like the way the protocol plays with those smart contracts there's a lot of problems there that's a hard part about our job I kind of want to bring up the financial side as well if you're a product building on other products or interacting with other smart contracts and other projects that build if you're extremely paranoid about security then you might hate to get all the other things all your dependencies automated as well but that's probably not going to happen most projects do want to kind of control costs as well so again there is kind of this financial limitation as well I think in some sort of way and that's where all the the trust assumptions come in I can't conduct an audit but there is part of the code based on the trust assumptions between other projects and other contracts Hi two things first how valuable are quick informal audits in your estimation if you're in an upgradeable world where you're going to be pushing out a lot of changes and whatnot some are valuable at least to have someone looking over your code just basically giving it a quick read through where you think that most of the value comes from being able to do a deep dive and then bonus question what would change about the the end of the world throw it away to that quick audit thing every code review I've gotten from Nick Jomson or like Martin Hulspand has been more valuable than any audit report I've ever received or anything I've built no offense to any of the guys here because they're all super confident but a lot of the times informal I find informally they also don't care about language and how they talk to me when you're in the business of auditing you have to keep your client happy otherwise they won't come back to you for a later audit when I ask someone like Nick to review something you'll break it in every way possible because it's fun which you can't do in an audit there's a certain limit on what you can do and how would you be to continue having a client I was going to say to ask the guys to do this for you we just did that again quick audit for them was it useful for you? yeah I think it's great but I would say it's the same thing it's getting a second pair of eyes you've made assumptions on and here's your question look at it, you get value I think an important thing is to be here on the expectations you get a quick look if you're here with that I think it can be very helpful I think it also differs on the stage that your product is at you might get a lot from a week in their audit but at some point you should get a real audit so not one pair of eyes like a lot on your code at some point yeah just to quickly add I think it can really help at an early stage especially with the architecture because you can design smart products and how they interact in many different ways and before you visit all the details if you design them in a not optimal way there's a lot of time that we have had some cases like this where you say early why don't you do it like this say to yourself a lot of time and I'll ask a lot of time because it's also going to be a lot easier to audit so regarding what would change on the EVM just minor things so right now when we do a call and we get return value call fails that is zero error is zero and success is one and we know from the Unix world that's just wrong I mean zero that's when it works and then you have an infinite number of ways that can go wrong and now we can change that we can't have it went out of gas or no it didn't go out of gas it actually reverted or some new interesting error happened because of that I need to say something long ago and we should also have a call with value so maybe to you especially because we're I think at F2.0 is like the big buzz with things like that with how the EVM designed do you find yourself talking more closely to the teams working on ETH 2.0 to ensure that we don't fall into these problems like the one thing which I know annoyed me at times is the way storage works in Ethereum is that it's very mathematically sound and every computer theory book will tell you to build it the way the EVM did it but in practice it's never actually done like that and I know Nick Johnson hates this as well it's the word size and that's like something I would change do you find yourself working more closely with teams to ensure that the protocol follows more of these each old design philosophies and is more sound in that design no actually would you like to yeah to some degree but so at this point ETH 2.0 has these different execution environments and I think the vision is that one of them will be EWASM based which is designed already by some other people which is smart and I assume it's good and rolling in ETH 1.0 means we can't modify ETH 1.0 too much and I don't think there's an intention to have like a new kind of ETH 1.0 virtual machine in ETH 2.0 where we just yeah refactor first and I don't think there's a push for that it would be nice if these things weren't so permanent so going back into protocol EIP 1283 delayed Constantinople I think you guys found that issue a bunch of us were on the call discussing that on whether we should delay it or not and EIPs is kind of this like rough thing everyone can kind of submit it and no one seems to review it properly and no one seems to audit it how do we improve that how to ensure that like we don't delay it hard for it's last minute every time because last minute is then when it's supposed to be interesting for everyone to look at how do we ensure that we get into this earlier and don't even get so far as to having to be on the call at 2 o'clock in the morning deciding whether to delay Constantinople or not so Hudson mentioned this briefly in his poll that we should do EEP Central Forks so that there is we change a bit how the process for EEPs could look previously we had someone submitted an EEP and a couple of people looked at it and said like yeah this is great this is going into whatever next hard work is and we said yeah we'll include these five EEPs into that work work which is going to happen on June 1st and then all the work started and problems and problems and as a post for work but instead of doing that the idea is to have the old quarter says yeah these five EEPs look great go ahead with these don't set a date don't set a fork none or nothing and then the work starts they actually build things and they get it accepted into clients once it's been accepted into clients in the form that it can be enabled by testing or by a genesis switches that makes it possible to actually run test cases for it as well so then they go ahead and implement test cases and they do security coverage and look into it from a security perspective and when they tick all the boxes yes it's imperative yeah Python test cases with a happy parent test cases for the work edge cases we've done a security review and yeah from another work edge cases it goes as well then they get back to the work order and say hey we have checked all the boxes when can we roll this out and the work order says yeah how long two months from now let's set fork number blah blah and that's when it's activated so it's not like we have a big heart for coming it's more like this EEP is now ready to just be enabled the work is done more than that incremental staffing I want to give a big shout out to Martin in the back because he was the one asking for security considerations in the IPs so that's a big big big step towards not letting these IPs fork up the network and it's baffling how this is not done before right it's not like it's not really passing a lot of the first like if everybody's just reviewing these things after as opposed to the one proposing them being aware of like how these things can affect the network at least they can give some focus to the security guys right if you are the one proposing it you should at least at least advise kind of trust model say oh this is how this could affect the network so yeah so I think 12 years security considerations are mentioned it's not merged no okay so we should work with that well with 1283 it was interesting because it's like a it's a behavior from the gas stipend which isn't easily found you actually have to know the protocol quite well to understand what the side effect what the security consideration is there which an EIP editor might not even know like the idea is fun but up until you know that nice behavior of how the gas stipend works you can't provide any useful information so is there an effort or should there be an effort where the protocol is documented better with all these like edge cases that got merged in previous EIPs that might affect other things so that when someone wants to make an upgrade to the protocol they can go there and see what they might interact with and if not is that something which we should ensure for ETH 2.0 that like really everything is properly documented because google Ethereum gas stipend and you won't find anything yeah just to follow up on the last part of the question so when I first came into Ethereum one of the things I liked the most and I think a lot of people would actually disagree with me here it was the yellow paper people hate it typically but coming out of the Bitcoin world where code was the documentation this was fantastic like this was way better than was there before and I find that a bit sad well again I understand it has been like I don't intend to blame anyone but it has fallen a little bit behind right so like not even the constant denormal codes are part of the current yellow paper so like you don't have all the current op codes in the yellow paper so that's of course not ideal I mean I guess I could also say another word in another way saying like a big shout out to like Yoji and so on who did a lot of the great work back then on the yellow paper so unfortunately we haven't been able to keep this up and I think there we have to do a bit more again because it was to me a lot of people like to flame but like in the end there's one place where you can go and you can check that has always been super helpful to me so I think that would be great and so and to quickly to a previous question about the EFPS on it's actually it was mentioned before it's a composability problem right like so because like you have these low level problems of like gas costs and so on and you have high level things like solidity brings things a certain way and so on and transfers and so on so when these come together then there's these assumptions they don't really mention anymore and it's also yeah what we saw now with this new hard work that there were some things being broken where people made assumptions about gas costs being fixed right and now cost of gas load is going up which is a good thing but like still it's breaking some things that people were not expected to be broken ever and so on so it's becoming more and more clear that actually there should be somehow more connection between low level and high level at least there should be more discussion about it it should still be clearly separated because the developers don't want to think about it and they shouldn't have too much but there has to be some more interaction between these two worlds I think so composability is the problem because that means Polkadot is going to have a lot of fun I think we're getting really close to an end here there's a question yeah sure over there I have two already questions so given that nobody can guarantee there is a smart contract so the first question is do you feel that in bitcoin there's a time lock type of mechanism allowing withdrawal hold in a certain period of time so you can call back you think that's a good idea to handle withdrawal the first question since that we are ongoing and we try to anticipate some zero day or maybe responsible disclosure of any issues that you haven't seen it so do you think it's good idea to put some kind of like pure switch either freeze the spam or have a quickly move the funds out with some higher privilege like what we see it's a kind of good best practice in terms of handle this kind of zero day disclosure or kind of like you know ongoing auditing finding that we don't anticipate in the open sibling contracts both the time lock contract and the also functionality chart too so yeah I think both are good idea what we used to suggest is have those in there up until you can prove that your code is mature enough to then feel safe to turn it off I think we're still so early in this space that we can't consider that a backdoor so for instance what we did with compound so what we were talking before is suggesting not only us but the community in general suggesting a time lock where you so if they want to upgrade the system or they want to make a major change they will need to wait for some period they introduce the change and then we have to wait for a period for the change to be implemented so people can if they don't trust the change being implemented with the other money so that's a solution and then of course there's governance and there's many governance solutions so they're from like a basic multi-seq to a DAO or whatever so I think this is like a huge copy but it's a really valid concern another question what advice would you give to someone who would one day love to do what you guys did for security auditing don't it's not that fun yeah a lot of responsibility and none of the fame just kept playing yeah just kept playing for shit like I hear that question a lot there's surprisingly there's a lack of resources basically I think the answer is that there's a lack of resources for security in the Ethereum world still even in the blockchain world but specifically for Ethereum we maintain the smart contract best practices repo which is like it's basically a community effort by now like we still maintain it but it's mostly crowdsourced people just like we just merge changes and keep it updated but it's the best one that I know of and that's kind of we have the Ethernet a game where different levels where you pack your way up launch that like two headphones and now it's kind of by the community and it's in the same world it's super interesting, super fun to play so that's a piece of side of everything learn the protocol, learn the EVM learn how Solidity works, how the compiler works and then start giving your friends free audits or code reviews and slowly at one point they'll offer to pay you that's how at least it works for me there's a couple more CTFs that there's oh man I forgot the name of the Captured Ether, yeah, Captured Ether by Steve Marks which is really really good and really fun and there's a couple more that I like we've also been maintaining one life from at least CC that I'm happy to share after this couple of CTFs, yeah just a photo, to what extent are some audits public to look at audits or previous audits as an educational tool is that a thing? yeah, I was going to recommend the Opens I think audits yeah, I think everyone will know a lot from those any other questions? I have a question about the processes like as what's all the saying for 0x there is when you deliver the report there is a fix that's coming up and you have to deal with it as a new audit and you already have new clients and new audits so how would you juggle between those new audits and the previous, how would you juggle between those new audits and the new mediation, new code and also when is when are you done with the code when you say it's a good code don't take on too many clients so you're doing a lot of fixes to code we already audit or new rounds for all clients so when you the initial report they come back with fixes and they look at the code again and this goes back and forth and then so for that part it's part of the service that clients will come with implementing the fixes we recommend and then who will look so it's part of the fault so it's a different thing and then for the new code of all clients it's maybe we try to prioritize them if we work with them before it's very complex from like classical security we would typically I mean in the follow up you verify that the findings are fixed and if they also re-did the whole thing it doesn't get a new audit and they would have to buy a new audit but this is like only a couple of take the checkboxes just these things that we found were fixed but it's not like that to do audit on the new code base just to follow up so if there is something found in that code is that you're calling to respond? So it's very important that you as an auditor to cover your own ass write down exactly we wanted this hash of the code in the repository the follow up was made on this hash and it specifically only covered checking that the previous findings were fixed and you should put these kinds of text in to specifically say what you did what it is and if you don't do that just you can get blamed later that these guys didn't see this whereas if you have it you can say yeah but that was after this metash and follow up The directory is a fun story with a classic security company on SSH they found a critical buy on SSH and they reported it it got fixed and then when they suggested a fix something else was implemented and they looked at it and in the end there was a vulnerability introduced by the fix that was not the one they recommended and they got blamed for years for introducing a backdoor in SSH so yeah it has to be very careful Partly it was a litigation so the ball around was reporting some memory errors and they took out the zero thing which let you weak But it might be a good tip to look at some classical security reports standard security reports and templates for that and you will see that there is like five or six pages of just boilerplate text saying like what the security what it has covers what it is and what it is not and maybe two pages there were actual contender findings and the rest is just very much get a really good lawyer for the company are there any other questions otherwise we are going to wrap this up because we have pretty much run out of time Thanks guys


As she mentioned, I'm Yagnik. I work at Shopify. For those who haven't heard about it, Shopify is not Spotify. There's a big difference. We don't do anything with music streaming. We're a commerce platform and we power nearly 165,000 stores at this point of time. So if you come to me and tell me, hey, you work at Spotify, how do I do this? I will punch you in the face. I'm kidding, not punch you, but I will be really mad. So yeah, Shopify right now is roughly stats, which is around 300 million unique visits every day, 8 billion in GMB, 10,000 plus in checkouts every minute, which really means for me, around 50 terabyte of data coming in every day and around 80 plus servers in our Hadoop cluster handling that data load. Why am I giving you these numbers? Really just to tell you that I'm handling decent amount of data. Would you call it big data or just data? It doesn't matter, but it's the amount of data that I have to play with on a daily basis and handle. So by processing all this data and taking care of it, we ended up going through what I like to call the path to data enlightenment. The path to data enlightenment in my view is something that pretty much every startup goes from the beginning right to the, I won't call it the end, but I'd say Google does it right now, so Shopify also does it. But point being, you start with something like querying your production slave. Usually it's your CEO or CTO who's directly going on to your production servers making queries and randomly getting data. And then when they do that, it's all fine and good because you're really small, you're a few people and it doesn't really affect your experience on the side or on the load or anything of that sort.  People have called it a data lake. I'm not sure if I should still call it a lake or a dump. But anyway, the data dump is essentially all your data going into one place. Usually for startups, it's like MySQL or any other database where you put in data from all the other different databases. So for instance, we have Shopify Core, then we have other microservices such as billing or authentication, and all the data will collaborate together into one MySQL database where you can join all the datasets, which wasn't possible before because all these databases were separate. But sooner or later, you come and hit a dead end where you can't really query and do a lot of stuff with this data. And that's where you end up going or that's where Shopify ended up going to Vertica, where we ended up doing the exact same what we did with MySQL but ended up putting it in Vertica to handle the kind of load that all our datasets would give us. So the sole reason for that was like that's our data warehouse. It's still a data dump because it's still all our raw data going into one place, but it's on an OLAP database which can handle that kind of querying and load. But a year back, Shopify learned that that wasn't enough. As we IPO'd, we realized that we actually cannot trust our data at all. The problem became even worse when you have to give GMV numbers out in public and you have to make sure that they don't change in history. So you can't just go back in time and make them different. You have to be the same. And that's where one or one and a half years back, Shopify decided to model its datasets, which is we started using Star Scheme if anyone's familiar with that or dimensionally modeling based on Kimball methodologies. And the idea was that we need to be able to trust our data. When someone comes to me and asks me, hey, what's the number for GMV yesterday? I should be able to confidently tell them this is it. If they come and tell me how many users we lost, say two weeks back, it shouldn't change when they ask me the same question two years later. And that's where the idea of building a data startup inside Shopify came in. And that's where we are. I do want to point out though is I put this on a graph which is complexity versus confidence. And that was the key point of this. The data sets initially, when you're querying it yourself, your confidence level is kind of, if because you're working on operational data, which is constantly changing, it's never doesn't store its history. It doesn't know anything about anything else. All it knows is its current state of time. And as you go through a model warehouse, what you're looking for is historical data. You're looking for changes in data. And more so, you want to be sure that the data is exactly how you expect it to be. Obviously, the complexity also increases as you go. Quering draw data is super easy, but building models and actually thinking about your data is a little harder than that. Took us a good one year to figure it out and build it out for ourselves. So it does have higher complexity. During this process, there are a lot of things we learned. And the biggest being because you've gone through these different phases of building data warehouses and building frameworks on top of that, reporting modules, yada, yada, yada. All these things has taught us a lot of a great deal of things that are required when you're building any kind of data framework. The point being, these data frameworks all have the core idea of processing data, but they all need certain aspects which make them powerful enough to do things, which make them production ready, or as we call it in our SaaS business, that any business should have a production ready app. You're something like uptime and monitoring. So taking all those concepts and applying them on data framework stack, and we ended up coming with a few of these. I'm going to talk about each one of them individually, but I wanted to list them down so you know what's coming up. I'll be talking about metadata, and it's hopefully not going to be just meta and go figure it out yourself kind of deal. Talk about instrumentation, storage, registration, data development, and build process. A couple things I will mention in the middle which are work in progress, which is these are my ideas or these are Shopify's ideas, but they're not hard and they're being battle tested right now. Whether or not they're good enough, I can't really say that because we're still testing these things, and I'm pretty open to discussing those ideas and get your viewpoints on that. So if you have any questions, feel free to ask me on those. Going ahead, metadata. I promise I won't make it very meta, so I'm going to talk about very specific things and give you exactly what we did and why we did it and what was the reasoning behind it. So the first thing was data schema. When we started out we had JSON and we started dropping all these data sets into what people call data leaks, and the idea is that when you put it in a data lake, you just drop the data and don't worry about the schema. What became really painful was the fact that people started dropping all kinds of fields into the JSON blob. Things that didn't actually even make sense just because they could store it somewhere, and then we started enforcing schemas. So every time you drop any kind of data set, then into HDFS or any storage that you use, drop the schema with it, and what that schema has is all the fields at that current point of time, and then the types of those fields. So essentially whether it's an integer, decimal, daytime, money object, yada, yada, yada. And you would store that with the schema. I'm going to get into storage of this schema a little later, but that's what the first part of the metadata is to store the schema. Along with that schema set, we also started storing retention and privacy policies. Being a public company, we had to follow a lot of rules and regulations around what all we can store and what all we cannot. And if someone asks us to come and delete their data, how do we handle that? So every schema set tells you whether this data could change in time because of some delete request or how long this data is going to stay in HDFS before it's going to be perished permanently. Really depends on the data set of what kind of data you're using. So something like logs are probably going to remain forever, whereas something like passwords are never going to come in, and they'll be retracted under the privacy policy. Next up is operational metadata. I added these as fields so that these are the things that we started storing, but the thing I want to focus on is the reasoning behind it. What came up quite often in our past data systems up until the model data warehouses, when people were looking at reports, they would often ask, is it actually up to date? How many records did you load last time? Or how are you doing these things? They were also started asking questions like, where are you getting your data from? I trust this data. And then when our engineers and analysts came around, they were like, but I want to know which code actually generated this data set, and how do I know about that? And then finally, because we have multiple data products, which data product is actually dropping this data set? So the metadata for operational data sets was exactly those questions being answered by the data and by the system. The idea was you'd give every detail of which source system was putting in, and when it was built, how many input-output records you would put in, and then at which point of code you could go back and rebuild that exact same data set, knowing that it will actually build up and give you the exact same results. Next up would be business metadata, which is definitions, questions, source, and owners. This is purely for business users. It had nothing to do with engineers and analysts. It was for all reports that we had that you could just go in and ask questions on the portal, and it will tell you which reports match your specific questions. Trying to be smart about, instead of a new manager coming in and asking for, do I look for GMB, or do I look for products? Do I look for shops? Instead, they could go to our portal and just ask in natural language. I want to see how much that shop sold in the last 10 years, and it will get converted to the exact thing or the exact report that they need. So that's the kind of other metadata we started storing. What all this metadata actually was meant for was my happiness in the end. I'd say Shopify is pretty strong on developer happiness. That's what I'm saying. They care about my happiness, and my happiness was this one chart. This is what we call overseer. The point of its existence is to help me tell what all datasets are there, what all their dependencies are, what are they dependent on, whether they passed or failed in a little lost run, whether they actually loaded any dataset, and how late or behind the data could be. This one thing tells me everything I need to know about the system at any given point of time, and all this metadata actually serves into overseer, which is this graph. So as you can see, for example, I pointed out on one of the green dots, which is that job pass, it's channel transition facts, and it tells me a bunch of metadata that I need to know at that point of time, but it could be expanded into a lot more in terms of doing this, doing any kind of analysis. Taking it a step further, though, we also use it for impact analysis, where if you make any changes in schema or in any kind of metadata, you know exactly which report it will end up impacting, and in our case, we break CI and don't let the developer merge in or analyst merge in any changes that would actually cause a breaking change into something else. So that's the point of metadata. Next up, I got a single slide for instrumentation. I think that was as an engineer, it's critical for me to know how are my systems running. And as data developers, we're pretty used to hard drives and network failures, a bunch of issues on the hardware level. So what we do at Shopify is we instrument literally every little detail of hardware that's running. So we know how much network IO is happening, disk or memory IO happening, whether something is dead. Along with that, we also have service instrumentation. When I say service instrumentation, it essentially means how a service is reacting. So in our case, we use Redshift, which is for loading our data sets. So is Redshift behaving properly? How does its metrics look? Whether it's getting too much, too hammered? We don't have access to DirectBox, so we have to work with APIs. But the point being, every single service in your system needs to be instrumented. And finally, job instrumentation. A job instrumentation could also be seen as the metadata just seen in real time. It's not dropped at a time of load, but it's being shown to you while it's happening. So something like this is what one of our graphs looks like, where this is one of the operational or system instrumentation. Job instrumentation would tell you more of what all jobs are currently running, how many records they have loaded yet, and then how many records are remaining, and how they're performing overall. So that would be instrumentation. Next up would be storage. So this was a huge debate at Shopify. Initially, we started at Vertica and putting everything in SQL, and suddenly we're moving to JSON blobs. But when we moved to JSON blobs and started using it with Spark, it was pretty good. As a developer, I could see my data. But it was really painful for the servers itself, mostly because we were loading all that data set, sparsing it again and again, and it made it really heavy. So what I'm going to talk about storage is what should be, what you'll be thinking about when you're choosing your storage format. In my case, I'd recommend Avro or Parquet, and I'm going to tell you why in either case. So first up is you want something that's shareable across applications. What that means is different types of data applications should be able to load that data and then also drop that data. Avro being a standard format, Parquet is a little less standard than Avro, but the differences that are pretty major to consider. Next thing is fast to parse. So the idea is JSON is super slow to parse when you're loading from disk every single time or putting it on a network band, taking over your network bandwidth when you're moving it around. Whereas Avro is binary encoded, which is fairly nice because it's super tiny. And when compressed, it could easily take care of your stuff. Next thing is the wrapper should also worry about schema. So I talked about schema earlier, but how do you actually store it? My recommendation would go storing the schema inside the wrapper, which Avro does. And what it does is for every single file, it will store the records and it will also store the schema that all those records conform to. Next up would be schema evolution. Sorry, I am getting a little thirsty, but the point with schema evolution is your data is going to change. Every single time you load any new data, every single time you have any migration, every single time the developer decides to change any event or your data is going to look different. It might be something as trivial as adding a field, or it might be something a little more complex like changing the data type. And your library or your schema library should be able to handle that kind of evolution and be able to go from one to the other without making all the downstream jobs break. So schema evolution is something that Avro right now supports. A lot of other libraries like Trift and Protobuf also supports that. But in Hadoopland, Avro is probably the best choice. So what are your options? Avro, Trift and Protobuf are all row-based tools. And then Parquet and Arc are more of columnar. The idea behind making a columnar is it's much easier to actually run a query on it because when you're running a query on it, it won't load the whole data set. Whereas it will only load a partial data set and therefore be a little nice to your machines and your memory. What I would recommend is do not please for the love of God. Don't use JSON, CSV or text files because initially it might be good. But as you scale your systems they don't scale with it. And you need to be able to, you need something that scales as your data grows. Next up is orchestration. By orchestration I mostly mean running workflows. Uzi is probably the standard right now. But Shopify had multiple experiences. We went from Luigi to Askban and now we're going to Uzi. And the biggest main point was be able to schedule stuff when data changes. Which something that Askban and I believe Luigi supports it in an iffy manner. But Askban doesn't and Uzi does. When you're picking up anything that you're orchestrating, you should be thinking about whether it's event or time-based scheduling. The idea behind event is whether that event could be a data is dropped or an event, something that's coming from some kind of message bus. Or in our case sometimes it's time-based for example our CFO looks for his reports at 7 in the morning every single day. So every single day there's a job at 12 which makes sure that all the data from yesterday is ready for him to look at at 7 in the morning. So support for both of them is extremely important. Next up would be notifications. For notifications what you're looking at is essentially whenever something goes wrong or something's going haywire, it should tell the person responsible to fix it or to look at it. Most workflows now have some kind of notification or failure emails but if you could have something smarter like I'm not loading enough number of records then that would be something even more awesome. I don't believe anything out there that tells you something like that but for failure notifications pretty much all of them do that. And lastly metrics and logs. When we were initially looking at things no one really cared about metrics and logs that much. They were like we'll worry about it when we need it but logs are critical when you're building data sets and data products. Knowing how much data you're loading, knowing what all is happening in your current data product or current application is critical to any data engineer and you should think about when you're picking up an orchestration system you should think about both metrics and logs and whether it supports it or not. This is probably probably my favorite slide because this is what I care most deeply about. It's small it's just one but matters the most to me. When you're building a data product don't just care about your data, care about who's going to work with it, who's going to be building on it both analysts and engineers and how are they going to work with it. So for in our case and what we made called Starscream which is our framework, Starscream has a bunch of options in it which allows happy developers or I hope they're happy. Some of the things are impact analysis which is what I showed you earlier. Overseer is an impact analysis tool and what it does is read all the metadata and knows about when things are going to break and notifies you before you break them as opposed to most data products where you will find out after you broke something that you broke it. So have something which allows you impact analysis and tells you on CI whether that's possible. Next up is CI. I'm giving an example of Shopify and Starscream. We run on Spark and every single transformation that we do on our data is actually tested in continuous integration. So we are testing on how whether that data is going to pass or not, whether it's allowed or not, whether it meets our standards or not and should be part of your data product. Next one is probably the hardest thing that came out of our experience. Every time a data engineer was building something they would either work with a very small subset of data which is not representative of the actual data profile locally or they would be working on the whole Hadoop cluster and taking over resources because they want to run some job over every single page view that happened on Shopify. What we realized was that if we could somehow deterministically bring down a set of data and then let developers use that, then they could just set a percentage and then we would call the data. So the idea is you give it a percentage and then what you get is the actual profile of the data being pulled down to your local machine which is represented of what's on the production servers. So for example, if I have pageview fact or pageviews of Shopify and they're highly biased towards certain pages, then the culled data should also look highly biased to that data set and how we do this is we run a job which basically creates ratios of culled data present for you and you could just download those. So what it would do is go over the whole data set, get its profile and how every field looks like and how often that field is seen and then take certain ratios so one, ten, fifty percentage and get those ready in HDFS every day so that you could just download that data set and build with it. A step further recalling is you'll never work with a single data set you're always joining different data sets. So what you could do, what you could do a step ahead is if you give it a job it knows all its dependencies and gets all the data sets in a culled manner which will join how you expect it. So that's the next thing that we did where we could you give it say for instance a fact building job about admin pageviews which requires pageviews, it requires user details, it requires shop details and what culling will do is go into every single data set and look at how they join and pull it down for your local machine, for your developer to develop. After that comes reconciliation. Most people at least in my experience at Shopify, every time we had data products we would just change things and change the data with it but once we became, once we started modeling our data and we went public we couldn't really just change our data hoping that nothing else would change. We needed to be sure that any time you make any code change it will not cause change in the data sets that's not expected. So someone in the finance team joked about and called it an RDD which is an abstract layer of spark but what it from our perspective it means is reconciliation develop driven development. The idea is every time you make a code change and when you push it to GitHub a bot will go and run your job and run the same job on master and see if the data changed where you expected it to. So in our case what the command really looks like is reconciled job one and I expect only column one to change and what it will do is run both the data sets, compare completely and then make sure that only one column changed and finally error reporting. At Shopify we decided to differentiate between filters and rejects. I'm going to go a little bit into it. This might be, I'd say this might be a little dicey of a topic because there were stronger arguments there and I'm guessing there'll be strong arguments here but every time you in our data sets you could not just filter your data you could also reject your data which is things that I know that are crap and I'm going to get rid of it early on which is whereas filter is just I'm going to get rid of it but there's no good reason. Rejects are actually stored separately with the job knowing this is the reason that I don't like the job, don't like the data set I'm going to get rid of it. Some people have argued that one or the other is not required. We can talk more about it if you feel strongly about it but that's what we do right now. Next up is a build process. This is work in progress. We're still trying to figure it out but I was mentioning to someone when I was talking about presenting here that incremental and full builds is something that's very much in talks in the data ecosystem right now. When I talk about incremental what I mean is you only look at the data that has changed from the last time you looked at the data so it's only deflogs or only change sets whereas full build is you can assume that every time you process your data you will get the whole data set and you're going to look at it completely. One thing that we definitely learned is don't worry about your aggregations as part of your incremental jobs. If you're only looking at deflogs don't try to be smart and go like I'll just add it or I'll just take the mean and do whatever of those sorts of things where when you're aggregating your data or taking sums. It's pretty hard problem and if you have a mistake you need to figure out you need to have a rollback strategy which is not straightforward in incremental jobs. I would suggest having a separate job that would think about just aggregating. So you have one job that does your incremental part which is cleaning and conforming it and then you have your aggregation part which just goes over the whole data sets and aggregates it which is really straightforward. And then finally MOTM which is what we call men of the maxes. Whenever you're working with a bunch of data sources then you should be worried about how far ahead each of those data sources are compared to each other. So for instance at Shopify we have orders and each order has many line items but they both could come into Hadoop at different point of time so you could theoretically have an order which does not have all its line items in it. It's more of a consistency problem but what we do with that is you make sure that you only look at men of the maxes so you only look at the timestamps which is the minimum of all the max values of all the data sets. So say for instance orders is present till yesterday and line items are only present till day before. Then you only process for that incremental job till day before because the rest of the data set is actually missing. This is still work in progress so I don't have a lot to add to the build process part but that's still building up I guess. And finally these are things that I haven't really talked about and we're still trying to figure it out but we're not sure. Something about data movement, how data moves from one place to the other and how it should be stored. We're using HDFS right now but it might be a better idea and it might be a better idea to think about another way of storing these data sets or moving them from our source data which is scoop or in our case custom build solution. Choose something else and people have talked about Kafka and streaming but data movement is something I believe right now is highly dependent on the company you're in, what's your SLA and how much you're looking to or how early you want that data. Second is data quality. Personally I feel that building multiple frameworks for data quality that would evolve with the company. We haven't come up to a solution which is which would be ideal for building a data quality framework which could also evolve as your data products evolve. I'd love to hear your ideas if you have any. And finally partitioning, that's again something that's left to the company and how you query your data. You could partition on date times but I've seen in our case we also partition on shops. That's again very specific to the kind of queries you're running. Someone talk about that. That's yet at this point of time I'll open up the floor for questions, anything I could answer. You talked about storage means how you guys are storing the data. So you talked about OZ or not OZ, you talked about ORVO, Thrift or other protocols. Sorry I am sorry I didn't catch that last part. Hello? Yeah. So you talked about storage like you are saying you you guys are using ORVO or other storage protocols to to store the data. So you are talking in context of SDFS or you are talking in terms of some no sequels. HDFS mostly. So you mean you guys are storing the data which comes under files in form of ORVO. So ORVO is actually marshalling or marshalling stuff for you. Yeah. Okay. So you guys are not using any no SQL to query the data. It's more towards you always have a batch processing towards your data to to give some meaningful information to you guys, right? Only recently we started using Cassandra to store our data which is only needed for incremental jobs. So something like if you're doing a large join we look it up in Cassandra but all that data is also present in HDFS in case you're doing a batch processing job. So we do use no SQL in our stack but it's not primary, it's secondary to HDFS. Okay and how the performance differs. Like as you told as you suggested no never to store the data like in CSV or in JSON format which is readable sometime means you can like at some point of time we need to get the data means you need to just look up the data but if you are using all these protocols like which are encoding or decoding the data to make it more to make it smaller to process with it better or so is that a challenge means you always need to processing on the data to look into the data even? That was exactly the reason we selected JSON initially which was developer happiness we need to look into the data itself but what happened what we realized was that a hue offers that you could look at Avro data and secondly the performance implications of it are extremely high. I don't have to graph up but I can show you the memory difference between a spark job that's loads into JSON versus Avro is significantly different and amount of time it takes to load move it from one box to the other over network is also significantly different. That's a significant improvement you saw and in terms of orchestration you you you talked about OZ so it's more towards jobs combinations means if you if you have a big task to do so it's more towards a job orchestration right? Yeah. Okay thank you. Any other questions? So you talked about representative data so can you explain more? Sorry, representing data. Yeah I mean see what you're talking about that representative set from the actual set if you have a very big data set you want a representative set from that right? Right colleague so okay let me put in the example of page view so say for instance you have a hundred million page views happening in an hour and when you have those hundred million page views you can't or in our case it's it's a lot more but in general page use is a pretty high big data set and you can't expect your analyst to always production machines and run their jobs on the whole data set to see if it's working. So what we realized was that what we needed to give an analyst or an engineer is the ability to get that same data set locally but it can't be the whole data set I can't ask them to download a terabyte every day what I want them to do is get a gigabyte but that gigabyte should be very much representative of the data and when I say representative the profile should match so the number of null values how often a page is seen those should still relatively be similar or the probability of them being different is very low. So what we wanted the idea with Culling was that you deterministically figure out how the data looks and then also give a percentage so like I as an analyst I could say I'm gonna work with 10% of this data so I give it 10% and call it and it will go look at the whole page views look at its profile and then select rows that match the criteria and then download it on the local machine. So we run this job on a daily basis for our critical data set so something like GMV or orders which are pretty high and then we will store it on a daily basis for 1% of the data 10, 25, 50% and then an analyst can start with just working with 1% build their job and then go go from that to testing it on 25% making sure they caught all the assumptions testing it on 50% and then we let them run it on production to make sure that they caught every edge case that possibly existed in that data set and all their assumptions are still valid. So that's the idea behind Culling. Is this offline processing you do mean is it how much time it takes to do the Culling? It depends on the data set so something like page views would probably take a few hours whereas something like shops would probably happen in like 15-20 minutes. Okay thanks. Hey you are talking about continuous integration just wanted to know what kind of tools are you using and what exactly happens underneath like since we are talking about big data and how do you exactly you know measure the code quality for example for the commit or so how it does it break and things like that. So at Shopify we build something called Starscream which is a framework that I cannot talk about because it's not open source but the idea is the Starscream is something use something to build dimensionally modeled data or Starschema and what we did was it's based on Spark so we took all the spark primitives made a mock out of it and then every single transformation you do on your data set you actually have a corresponding test for it so for example in your you have your fixtures which is your test data and then you apply those transformations to make sure that the data is actually correct. The CI tool does that runs it and mock and also runs it on an actual spark cluster with test data to make sure that the transformations at the assumptions are correct and they still hold true when someone changes the code and then finally what it does is whenever you're it does the impact analysis because it over so every time you deploy to see up every time you send a branch to GitHub and you enable CI it will send a request to overseer which is the impact analysis tool which knows about all the data sets all the jobs and all the schemas and it will see if anywhere in the DAG you're breaking any assumption that someone else made and then tell you that this is broken you cannot merge it in and that assumption goes all the way until Tableau which is our reporting framework so we know that if I made a change in the name of the field that's being used in Tableau that will not get merged in because it will be read on CI. Those are the two primary things that CI does for us which is impact analysis and then code coverage. Does that answer your question? Yeah thank you. Excuse me so here here so you mentioned about the culling right so basically you have let's say terabytes of data and you want to give only GBs of data to your analyst people to explore it more better way okay so how you are giving this freedom to analyst people like you are giving it through redshift or what basically you're using is it a day job or weekly job? We do a daily job right now for most of our critical data sets our analysts are allowed to run a job for a specific data set which is not already present for them and so that's that's what we do right now so you could run it yourself or you could pick one of the data sets that's present on HDFS and then it's stored as a file on HDFS you have to download it but it's significantly smaller. So for storage what you are using like which technology? Sorry for downloading. For storage yeah for storage. Just Hadoop commands like Hadoop FS. So command line arguments. Yeah hi I have a question. I'm sorry where are you? Yeah so I have a question like how do we handle inconsistent data so as we store the data format that we follow is and blocks or some other kind so the data is continuously changing so how do we deal with the inconsistency of that data? So something like that happened with our Kafka data source where the data was constantly changing with event streams so say for instance you have an event for admin admin being all of Shopify admin and then all the events coming in had different profile attached to it and what we realized was that people that was just having an open firehose was a bad idea and we moved to curated data sets so every single event stream is a single type of event and the schema only changes as much as your database would change in its schema so only once in a while your schema changes and we managed that with Avro schema evolution so Avro allows you to change the schema and be able to tell you whether it's backward or forward compatible in which case in Kafka's case for example you have your Avro schema and then if you make a change to that schema if it's not forward compatible it will actually break things and the CI will break and won't let you merge that change in because you broke something but if it's compatible it'll just let you have it so adding a field is always a compatible change whereas changing the data type is an incompatible change in most cases so that's how we did it thanks we have time only for one more question hey hi so I have basically two very quick questions specifically from a monitoring point of view to see if everything is working fine and things like that do you have an in-house custom solution which you have built for monitoring so do I have an in-house what for monitoring in-house solution we use a data dog which is graphite based okay for monitoring so it's for real-time monitoring that is right along with that we use Splunk for our log monitoring okay so that's it like our code base sends events to those things okay and for visualizations again data dog is like they present a visualization library thanks a lot last question two more minutes hello hi this is Praveen I want to understand like did you ever get a problem in a phase problem in compliance where you store the PI information at different geographical location and use Hadoop Federation sorry can you repeat I stood PI PI information where so the concept of Hadoop Federation did you use or any other methodology I used we have been looking at it but we haven't implement anything for Hadoop Federation we just handle it at the time of loading so we have a custom instead of scoop we have poor man's scoop which what we call longboat and we built it in-house so we have PII data have like PII control happening at that layer itself okay but right now we don't use anything on Hadoop side okay but how do you manage to use that data to show it to the whole geography so we have a central service for PII which tells us what all what all rules are applicable for private data and whether that data has been removed or not so we already are aware of deletes in our data set things we need to care about is something like an email or sorry something like a phone number which needs to be retracted so you're not allowed to store the whole phone number you're only allowed to store the area code so we have certain rules in our we have a separate service which manages different rules and knows about different kinds of PII data so we hit that service and we ask it whether this field needs to be PII compliant and if yes then what kind of rule are retracting logic do we need to apply to it and you give it the field it'll apply to retracting logic and then you store it into HDF into HDFS yeah okay does that make sense yeah I got it I I feel really sorry but I'm gonna have to cut you and please take this offline