ic in Bristol. And I've had the great pleasure of working with both the data hub and data helix project teams over the last year on creating kind of like synergies between two projects which I'm pretty sure both Paul and Andrew are going to talk through today so Paul it's over to you. Great thanks James. Can you hear me okay just check the microphones working. You are good. Fantastic. Thank you. Great. So yes. So hi I'm Paul groves. I work over at city group as an architect in the client and boarding space and thrown over to Andrew say hello. Hello, so I'm Andrew car I'm head of consultancy for the Bristol Scott logic office. What's the goal. So, um, let me get that working right so data hub. That was a contribution that city group we made as our first contribution to Finos earlier in the year. Very briefly, data hub it's a synthetic data generator. We support two modes of operating which is you can kind of handcraft your own rules, or you can use production, you can use like existing data sets, data can analyze it, produce like a statistical model out of it and then it can use that for synthetic generation to try and preserve privacy. So kind of support these kind of two things. Now data hub it's largely what we've done there is we brought lots together lots of different kind of open source existing libraries together and try to make a very, you know, seamless consistent interface across them so there's lots and lots of tooling out there, but the problem was there's a lot of time you had spent to try and bootstrap these projects up and so we decided to give it all a bit of consistent interface and then we've done a bit of our own work as well on top of that as well. So that's kind of data hub what it is and I guess Andrew data helix. Yeah, so data helix I guess came from a slightly different point of view we tried to build a tool to start with to mostly be used by our testers to be able to generate test data really quickly so the aim was hopefully without coding they could write a config file, which described the rules of the data, and then run the tool to to generate some data. So similar field to data hub but actually a different use case which I'll get on to a bit later so essentially what we ended up doing is having a system that described this own language which is what you're seeing on the screen right there, which is a data helix profile. And this is a playground that you can you can find on on Phinos. And if you modify the the profile data profile on the left, and then you press run, then it will show you some sample generated data. And it really is the idea is to satisfy the use case of generating large volumes of data quickly for doing low testing, or stuff like that, where it would just simply take you a decent bit of coding to write your own data. So next slide please, Paul. Right. So synthetic data. I guess, you know, what is it has been a lot of talk about it recently. It's all a bit mysterious, I think. So, really simply put synthetic data is anything that you can algorithmically generate. And there's lots and lots of different ways that can be done. Lots of different kind of applications for synthetic data. So you can go back to things like the oil gas industry, the medical industry, you know, government statistics they all heavily use synthetic data approaches. And then I think probably a bit more one day also from finance, we tend to look towards synthetic data for data privacy.  So largely with synthetic data, there's two broad approaches that's that kind of defining your own rules.  Yes, there's a second one where we do the data analysis. We try and do use a bunch of statistical analysis. There's lots of different ways that we've done, particularly GANs are a big area of research and a lot of commercial products are now using GANs to analyze production data, you know, work out distributions and then use that kind of statistical model to then forward generate the data out of that. So it's not particularly a new thing. It's something that I think there's been just an increasing amount of talk about it as well. So a bit of history, I think, you know, if we looked at what we think of synthetic data now it actually probably started back in the 90s with the US Census Bureau. And they had a problem where they wanted to get the, they had kind of a couple of problems. One is not all the census data is complete, you know, people might partially incomplete the forms. There might be a lot of missed households wouldn't even respond. So they wanted a way to statistically populate the missing data with kind of what they would thought would be good data. And that's what we kind of think we often now use things like machine learning for. So they came up with the statistical approaches to do that. Now, the other thing they want to do was they wanted to give out the data sets in a safe way. So redaction had a lot of problems where you could essentially re-identify people pretty quickly. So they want to take their big US Census data. They want to then transform that into a, into a same data set that was publicly shareable and you had no risk of re-identification. So that was kind of their idea. And there was a whole load of techniques that they came up with in the 90s to do that. And so that's where a lot of the work comes from now. More interestingly, or more fun, if anybody is in their 40s and used to have a BBC micro spectrum, you probably recognize this elite. So this was the whole procedural generation route. And what they were trying to do with the Abley was had very limited resources, tiny amounts of memory. But they wanted to create an entire like galaxy of 256 planets, eight galaxies that you could travel around. They wanted to make it seem very, very realistic and really kind of give you this immersive experience. So they went down this approach of actually doing procedural generation and they had a very couple of very small data sets. It was like a couple of like string lookup tables. The whole thing would have taken like a couple of bytes of memory, the actual data they needed. But what they could do was consistently then generate this entire galaxy on demand as it was required from that very, very small input set. And that would create this kind of little galaxy of planets that will be of all their unique properties. And that's really what procedural generations about. And again, elite back in 1984. While it, you know, lots of things had been using procedural generation for them. It's probably the first time we really saw somebody do procedural generation and make this kind of real lifelike immersive experience. So that's kind of, you know, a little bit of history about where we come from. So yeah, since I just nothing particularly new, but there has been it has been coming up quite a lot. There's been a lot of new vendors coming up around in this space as well. And I think that's for two reasons. One is we've had all the GDPR and data protection. And I think a lot of us, we've all gone down the same route of trying to do the data redaction data anonymization. And that's quite an expensive process. And often I think the results have been a bit unsatisfactory like particularly once you know redaction, we've all had problems with re identifying data. And if you're anonymization quite often you end up with this, just this massive ball of redacted fuzz that has no use in your test systems at all. Also, there's been quite a bit of interest from machine learning and AI. So how do you create realistic events, generate realistic events for your, not for your training of the model, but for your kind of your back end engineering data pipeline. So how do you get extremely realistic data into your, into your kind of AI ML kind of pipelines that you can be doing your development with. Of course, you would still always use the real data to train your model, but you know that kind of bit before it. I think some good examples are, say you're doing your AI process or your ML processes needs some new data tagging, and that tagging is not being done yet. But your modelers need a bunch of data to start working with. Well, other changes have been happening to try and do that tagging process. Now as you kind of separate that out a little bit as well, you generate some data upfront that you might not actually have access to. So yes, redaction, anonymization and synthesis. When it comes to data privacy in your development systems is your free approaches. Redaction to remove sensitive information.  Anonymization is a fuller process where you're not just redacting, but you're going to make sure that there's no way to re-identify out of that redacted data set back to a real person. And that's actually quite a complicated process. Or we go down to synthesis. And as we said, there's two synthesis ways. You do a completely procedural waste handcrafted or we use the existing data sets. So when we talk about redaction and re-identification risk, this is a very simple example.  So if you just went through and just remove people's names and maybe just took a few other fields out and then you put that into your test system. You, it wouldn't take a genius to prickly pretty much quickly work back out who the people were from, you know, other information that's internally available like your address book, particularly because you've got something that's like there. The department they're in and they're even just simply the department they're in and their corporate title. Quite often you might only have one person one corporate title in a department. So that's a really easy thing to just go and re-identify all that information again. That's a very, very simple example. There's been a lot of case studies into re-identification and you need this. You only need like a couple of attributes often to be able to go back and re-identify. And that's where we get into the area called differential privacy. And the point of differential, we could probably spend hours talking about this, but the basic idea of differential privacy is it should be mathematically impossible to take a statistical set of data and then backwards to work out back down to original individuals again that should just, it should be impossible. So actually differential privacy, it's kind of quite firmly rooted in cryptography. And so we might just try and stay away from this subject a bit today because we'll just go off in a tangent a little bit. But differential privacy, if you're doing data analysis to then use the synthetic generation, it's kind of quite a key thing in there. It's very important. So how are we doing for time? Yeah, so if you look at like a typical kind of synthetic data flow, you'll start with your production. No, you've got kind of two bits. You've got the stuff that you need to do in your production environment and then the stuff that happens in your non-production environment. So you might start with, you know, in your production environment, you need to get your data out. So you query your data, you kind of do a basic bit of redaction, make sure the PI attributes are kind of removed. And then you'll do a bunch of data analysis on that and then produce some kind of statistical model like your differential privacy file.  So now you've got that file and it should be cleanable PII. You shouldn't be able to go back with that file and re-identify anybody. It's just a big statistical model of that data. So what we do is we take that, then we can do synthetic processing over the top of it to then generate back at, generate back out all the row level information. And then we can add back in PII-like attributes. So not the real PII attributes, like somebody's real name, but we can then generate fake name, fake addresses and fill in all those blanks that we've, those bits we redacted out as well. And then we can write that to a prod database. So hopefully what we've got then is in our non-production database, we should have a whole load of row level data that looks like the real data, but it's no way to ever, it feels like the real data. It feels realistic, it feels, you know, immersive and whatnot. And it should be decent quality. And it should obey all the rules that were observed in the production data. So that's kind of that's the, if you've got an existing data set available. Or we have the opposite route, which is there's no data available, or you kind of don't want to use it and use code for straight procedure generation, which is fill up a handscraft a bunch of rules. You produce a whole load of data. You put it in your data store. So it's quite simple. But obviously there's more time consuming to do this. Your developer, your person who's watering those rules is going to have to have a much better understanding of the data, the constraints, the rules of the data, and we'll have to have much more innate knowledge about it to be able to do that. So cool. I think that's a hopefully a very brief view of sensitive data. I guess over to you, Andrew. Cool. Yeah, so I'm going to talk to you a little bit about the typical use cases for synthetic data and I guess how they impact procedural generation. So as Paul mentioned, there are a few different approaches, obviously taking normal data and then the modernizing and redacting it has its challenges. I'm not sure, but I suspect Paul's probably going to chat about that later. Certainly one of the issues with doing that is you can end up with data that's in no way useful. You readact it and anonymize it so much to the point where it's not actually useful in terms of fitness for purpose. And I do think when I guess Paul and I have found certainly being involved with this fitness data generation, we've ended up having lots of conversations with people about their use cases that they're using synthetic data for. And the more you talk, you begin to realize that actually the solutions for synthetic data, it really depends upon the use case that you're trying to solve. So for example, if you're looking for kind of lower volume, highly accurate data to test your functionality, you're probably going to go for a very different solution than different. The other use cases, which are more like your kind of high volume reasonably shaped data and reasonable accuracy, which you might be just doing something to do a like a low test, or if you're actually using some high volume data that you want it to be statistically accurately shaped, because you're going to be using it for modeling, machine learning, analytics, etc, etc. Paul, can you go to the next slide please. So I guess depending on which case you're considering, I'm going to go through a simple case and demonstrate why the rules based approach is really good if you're going for I guess the kind of load based testing where you want the data to look reasonable. It doesn't have to be that accurate. But you want to generate the data really quickly. And then, you know, at all like the data helix and there's there's a number of them out there, where you can just use a conflict file to modify the rules can be a good approach for that. But I guess it's not such a good approach. When the data you want needs to be highly accurate, and you need it either for, I guess, functional testing, or you need it to be, you know, statistically valid shape where you're probably not going to want to go down different route. Cool, if you can hit the next slide. So let's let's consider a really simple use case. So like I said, the kind of low volume, highly accurate data, let's see what we can do. So imagine you define the test data as a really simple and this is just five columns. This is a massively simplified example, but it's just really illustrate how complex rules can can get really quickly, even to get data that on first pass looks reasonable. So if we have some sort of trade ID, a stock ID, whether that's a RIC code or QCIP or ISIN or whatever, a stock name, some sort of price. And again, I'm simplifying it so we're not not including currency and all the things that you do typically do actually need to include to be useful and the trade date time. So if we hit the next slide. So if we if we just literally defined, we needed the fact that we needed like an ID, another ID, a name, a price, and a trade date and we just let a generator go off using even you know some really basic modeling would clearly get nonsense data. And so from a, you know, I suspect from a low point of view, this wouldn't even be that useful because the stock ID is probably the wrong size. It probably doesn't contain the typical characters that stock ID contains the price clearly contain the wrong number of decimal places. If you're doing a low test with this, I suspect while this might have some use it would probably have negligible use. So it's not really, it's definitely not usable for functional testing. It's probably not usable for volume testing, and it's definitely not suitable for machine learning or statistical analysis of anything. We go to the next slide.  We can certainly go for enumerations, for example in the stock ID, and then go for enumerations in the stock name, and then we define a little bit more the range that we want the price. Again, these could all be rules language we use is, I guess in material. And if we put a little bit more rules around the date time as again, we say it's I don't know greater than a week ago or less than today. And even with these sets of rules, if we head to the next slide, we can see that while the data looks on first pass a little bit more sensible, it doesn't take us more than a second to quickly realize that the stock ID and the stock name don't match, which which quickly brings you to the conclusion that you really need to do dependencies between the columns. So if you have any row of data, if one of the columns is one value you probably want to narrow down the field of values of the other columns. And in particular in this case we're all aware that the stock ID and the stock name clearly need to match, or we're just generating data that doesn't really make much sense. Now for volume testing that might be okay, depending on the situation, but possibly not. But for functional testing it's clearly not going to be useful. And it's definitely again not going to be useful for machine learning or any form of statistical analysis. So if we go to the next slide and try and tighten rules up a little bit again. We can do dependencies between the columns and say okay the stock name has to line up with a stock ID. Either you could have some sort of air for you could have some sort of enum where they, they say they line up or however you wish to do it. But if you had a bit further say the float price needs to be two decimal places. And then if you go to the next slide and try and generate again. Okay so it's starting to look a little bit more sensible in the stock ID and the stock names now line up, but clearly the price is swinging all over the place. So this might start to get to the point where it's useful for volume testing where it kind of looks okay enough. But for functional testing and machine learning analysis, it's not enough. And this has been a really simple situation where we're just getting started and getting the data anywhere close enough to be useful for volume testing. And I guess for for projects I've been on just involved with recently, we were looking at a situation where we had a table of trades, where the trades had 140 columns, and terms of the rules based approach, you needed about 3000 rules to get the data to look accurate enough to do volume testing on. And even then it wasn't accurate enough, I guess to do functional testing. So, I guess I hope I've demonstrated even in a small way how quickly these rules can build up. But it definitely gets to the point very, very quickly where you need a very large volume of rules to get accurate and and going forward if we head to the next slide. If we if we start to look for a more complex situation and trade so we've gone for the really simple situation if you consider something like a bank account generating data for that. You can clearly see that you're going to need start to need dependencies between the rows as well, and also some sort of state, because you're going to typically expect salary to come out of the same day every month. You're going to expect the same big outcoings every month rent you're not going to expect that to be some variable that massively varies with over the place. You're going to also hope that the outgoings are less than the incomings. And also you're going to want the amounts to be realistic such as if people get coffee at Pret on weekdays when we used to go to offices and stuff. And also you kind of want the number of realistic events to be realistic so you know you're not going to want someone to go to prep 300 times in one on Monday, and then once on Tuesday. So I guess what I'm trying to get across here is just the more complex and the more accurate the data has to get and certainly when you're leaning towards functional testing of statistical analysis, you really need to start looking at either a different approach which isn't the rules based approach, or you need to just add so many rules as to the point of it's almost going to be as complex as the business logic of the application itself. Okay, next slide. So I guess this is just a summary so if you're looking for volume testing and the data has to look sensible rules based approach approach can work. If you want super accurate data for functionality testing and validation. You're going to have to probably get involved in doing your own coding and development because the complexity of the rules are probably going to be such that it would need to be as complex as the application or close to it to be able to generate the data properly. But if you're looking for data which is the right shape. As Paul mentioned you can use one of those techniques that you've used machine learning or something else to look at the data, extract the shape and then generate data which is a very similar shape that you've seen. Next slide, Paul. This is my slide actually. Sorry. So, kind of examples of some of the things that we've been using synthetic force so particularly in client onboarding, how do we generate kind of realistic onboarding events. There's, you know, it's very heavily workflow based. So there's a lot of complexity in those workflows, you know, only certain steps in workflows occur with, you know, particular client types and the rest so that's one big thing that we kind of you started this problem on. Things like generating risk in PNL with realistic sets of values. So that's just kind of big volume data. The constraints of the data is relatively simple. Then we also have things like since actually generating portfolios of trade. So, you know, just creating portfolios of slots for example, credit card payments between motion card holders and one of the main reasons we've all seen this is an even easier relationships with vendors and cloud providers when we're trying to have that exploratory first part before the NDAs get signed and all that complicated stuff.  So that's kind of the kind of some of the examples that we've been kind of looking at and using. Moving on to the next slide.  So data hub relatively it's a Python library. Now that's to say, we're heavily based on things like that pandas data frames. So what we do with data hub is we just return back data frames back to you. What we do with that data frame if you want to go and populate a database put into a CSV file. That's kind of outside the scope of data hub bears. There's plenty of other Python libraries out there that will handle, you know, populating databases with data frames for you. So that's why we kind of sat around with pandas and got we didn't really go down the whole data syncing area. But yeah, just you type pip install data hub core and after a couple of minutes of all the dependencies going down, you should be get going. So we'll do that now because it will take a couple of minutes. We'll use a system thing I've just set up. So first of all, just going to quickly show you how to boost your project up and handcraft a couple of rules generate say some client data. So if I tap out and over to here. So take that out first. So that way. So what's going on here. We maximize that up. So we simply call it we've just imported data hub here. It's all Python based and the way we script the rules again, it's all kind of in Python. So it's very much in the Python ecosystem. So we call like a data hub generates and we give it some properties like some descriptions of the attributes that we want to generate. So though we're just saying create me a field called region and pick one of these values and use these weightings and give me a hundred of them and we've got a fixed seed number here so use a fixed seed number. Every time you generate it, you'll get the same data back each time. So if we were to remove that completely, it would be completely random each and every time. So we also allow for consistent generation as well. So if we just save that for example, we comment that out as well just going to be useful later. And we you run that you can see we basically get a list of 100 regions and they're just the burying. So it should all match the weightings that we've seen.  Now let's say we want a country field. So we've got a we've got a client and it's in a region and now we want to put that client in a particular country so we can do something called gen dot country codes. There's an in built actually regions are in built as well but so that was just to illustrate to generate some country codes. Now, what's going to happen let me change that to 20 actually, and then we run it. Right, so you can see again we've got it here and now we've got this very ugly object in there, a country object which doesn't look very nice for you. So let's go back and we get a bit of post processing to just get the alpha free code of that country. So that's what that line of codes doing. We go back and run that. And now we've got a we've got the country alpha free code. Very quickly, you're going to go this is rubbish. You know we've got country Poland and we said it's in North America, we've got India in North America, India in a package is clearly not very good. So what we're doing data is we kind of understand things like regions and countries and currency codes. So what we can do here is we can give it a straight. We can go region filled region and this generation function will understand that constraints that we're putting into your country. Now, if we run that again. It should look a lot better now so not now USA North America Australia and eighties and eight pack Ukraine is in a mere and so and so it was all good. We've got a few more bits and pieces so let's pick an industry. And so you don't watch me type stuff out I'm just going to go and steal from a, a mother. So let's go back. So we're on like an industry type and a camera in there and a sick code for this client we're making up. There's just been a post processing because these are actually objects and there's lots of different elements you can in that object that we can do so to keep the rendering nice in my screen. Let's just add that post processing back. Have you render that now. We need to fix that. That's why it's sick code. Sorry, run that again. We've got a little bit more so we're saying we've got a region the country now we've got industry so mining, and then a particular kind of element of the mining so this is just using all the public standard industry code information is out there so again data hub So we're going to generate a legal name for that company and it's going to be specific to the industry and the country that's in so we're going to give it these extra parameters say look there's this that's where you can find the industry code for it and this is the country it's in. So again we can name things accurately, depending on the country and the industry. Again, let's go in and run that one more time. It's. Here we go. So, now we've got region country industry. We've got the sick code and then we've got a name. So, you know, here we've got company in finance, which is more to bank is a loan correspondence. We've called it like hold the loans group for example. There's a construction here called that's a bit of a horrible name. You see we kind of try we kind of generate appropriate names for for that kind of industry and country. If we go back let's go and get some numeric numbers and there's all sorts of ways we can handcraft how we want these numbers to look so with at the moment we're just going to do it very simply just pick me around a number between these two points. But there are distributions that we do we can say look use a normal distribution particularly around a particular with a particular standard deviation. So we've got all those kind of normal distributions it will find. So assets under management and estimated value. Let's just do that example and run it again. And now you can see we've got the AUM and the UV mode done over here just random randomly generated. So that's kind of how we can quickly handcraft rules together and get very quickly a reasonable looking data set. So now if we head back over to the slides. The other one was like well how do we do analysis of data so I'm not going to run this live because it can be a little bit slow and probably got better things at the time and watching a now clock hourglass tick by. So what we can do here is this very first line that you're seeing up here called analyze run so we're going to give an input data set or CSV file and we're going to ask you to output like a model file is done. Let's look at it a little bit about the data set the bits of it that we're interested in so essentially what are the discrete values and what are the continuous values. So we say the discrete values are region countries that code industry and so on. And then we say the continuous values in our AUM and EV. And there's a bunch of other bits of pieces that we do. There's different kinds of analysis modules so there's one which is called the fast bucketing model which runs very very quickly and generates data I would say a reasonable quality. We can go into that baby later and go into how that works if people are interested. But there's also things like we integrate CT GAN. There's another one called STV. And there's a couple of models as well. So there's another one that uses it uses like linear regression, for example. So there's different kind of models that we can plug into this to do the analysis and then they all basically output this a file. So what we then can do is we can then write another function which is then generate the data. So we're going to call the gen.generate from model. We're going to set it what the model is so we call this we're going to use this fast bucket model. And we are going to supply this model file.json in there. And then what we can do is remember if we go back to that slide earlier where we said about adding back in like PII attributes like things like names and addresses. What we can do here is in this properties area we can add back in like a name and we can tell it about where it will find the particular bits it needs to generate that name in the data set itself that it's generating. And then we can basically redecorate the data with these PII attributes that should have been stripped out. So if you notice over in this analyze.run we're not we don't put any kind of we only put the elements in that we actually want to do the analysis on what we want to fit to the distributions. We don't put things like people's names in their social security numbers, bank account numbers, all of that you add back in afterwards. So if we go back in that earlier picture, this generate model function, that's what we'll be running on against our production data. And then this second function generate that's what will be running in our development environment to actually produce the data and then do something with it. And I said you end up with a pandas data frame so you can easily just call data frame dot to CSV. And there's plenty of like things like for populating Oracle databases with data frames in there as well. So that's pretty much what data hub does. It's pretty plugable model. We're constantly trying to generate more and more models and refine these models. Now I went through differential privacy thing at the moment. It's not as tough yet as some of these models has been able to do for differential privacy, getting kind of close but there's a lot of legwork that we need to still do just be able to actually make that claim that it's done. So that's something that we're trying to really aim to get done. Also, data hubs incredibly easy to extend. It's just Python. So if we wanted, if we wanted our own generation function that was unique to us that you quickly wasn't part of data. So you should just write these couple of lines of code here. You need two functions. One's a partial function. So this is this is a extension we're doing, which is going to just say hello, a name or something. So again, we can see we put the region in put the country in do then we do the name. So we're going to generate a person's name. We're going to make this extra actual message. And we're going to call this little extension function we just wrote here. So what we're going to do here is the data set should be saying hello, Paul, hello, Bill, hello, Jennifer, and so on, whatever that name is that will, we'll see it in that message. So that's so that's how it's quite easy to extend the functions there. I guess what's next with data hub and data ethics and Andrew chip in here, if you want to. So we're going to start merging these two together. So we're particularly interested in the spec language that data ethics has. So how can we use that in data hub as well. So it's much easier to probably just write a quick JSON file than is to get down into Python. So if anyone's at first writing Python, we can actually still use these data profiles that we've got in data helix. Also, we wanted to have a big investigation into the legend language that I think was presented much earlier on. And so can we take legend data type specifications and then can data hub understand that and then start since it's been generating data to that specification and to the constraint language that's in legend. So that's that's a kind of a big area of research we're looking at right now. And so we've got data version two coming quite soon. At the moment, we only support when we come to analysis. We need discrete data as well as continuous values. We can't handle data sets of continuous values only. So that's a bit of enhancements coming along. We'll take the multi table support as well. So actually you can say you've got the classic kind of, you know, customer supplier database, you can export those as CSV files or whatever. And we'll be able to and you can supply those as like a multi table with understand it will understand the relationships between the different tables, you know, the form key and primary keys. So multi table supports coming and we'll finish actually with the CT GAN integration as well. So CT GAN is an open source. Dan model that generates synthetic data. And so we're going to bring that into data hub as well kind of wrap it over and give it that same consistent way of working as you would with all the other models. So hopefully that's going to be in the next month we'll be making that big PR and it'll be available. Then in we go to the version three features. So a big question is people always ask you as well how big a data set can you generate at the moment we're reliant on pandas data frame. So the question is always well how much memory you've got. And that's how big the data set can be that you can generate. So our techniques we can do to break it up into smaller parts, but really that's that's your limit. So we're looking for spark integration. So we can actually run these as like spark jobs on a deep cluster, for example, or and then generate really truly huge data sets over in Spark. So the integration, that's what we're looking for. We've got this other element. So I think earlier dates type predictors. So I think earlier when we saw here, when you do this analysis, you've got to pass in the discrete continuous values. So what we'll look for is there's a another bit of work that we've got going on where you give it this input set, and then it will go and predict what all the types are, and which should discrete which are continuous which are PII. And then it'll automatically kind of generate these these parameters here for you or and present or present them to you in a way so you just, you know, double check what it first of all. And that data predictor as well we're looking for it to understand things like is it QCIP is an eyes in your standard industry codes. And so not just that's a string, but that's a country code and that's an eyes in your back. And yes, that's the next bit so there's a few more financial types that we need to build into this so understanding intrinsically, we already understand things like industry codes, any eyes, and so on. You know, do come we start understanding things like QCIP buys in curves, tenors and all those other current general financial data types as well. And again, it'll be really interesting to start seeing how we can integrate that with with the with the project. And lastly, this is probably going well into next year but we're going to finally bite the bullet with agent based modeling. So we can actually see if you start using data hub for more simulation based things you know around agents. Andrew, do you've got anything you want to add I'll throw the mic over to you. No I think you did a brilliant job there Paul. Yes sorry lots of talking have to drink a lot of water now. So I'm Andrew and poor hope you don't mind me cussing in now, but I think it might be quite nice to throw the Q&A open to our attendees in the audience. So if anybody's got any questions to ask them both Paul and Andrew, feel free to use your Q&A button that I believe is right in front of you to ask your questions. But in the meantime so each of these questions going to be to both of you, because I know data hub and data helix, you know they're similar but they are also different. So Andrew first, who is data helix actually geared towards so who your customers who should be using data helix. It's an excellent question I mean we developed firstly when we were doing a project for a company that ingests large, basically flat files of trades, and then sends those trades on to the regulator, and they basically enrich those trades and do loads of validation on those trades, and really isn't regulatory reporting technology. And what we needed we needed to generate lots of fake or test data to test their data in QA environments, because a lot of their QA and dev environments were in the cloud. But obviously their QA and test environments weren't secured in the same way that their production environments were, but also they didn't want to use any of their production data in their test environment so we really designed it for testers to be able to get some synthetic data up and running super quickly. Also for that particular project, they built or they had their own spec that their engine did to validate the trades. So we built a UI that created those rules, and then we thought well once we've got the rules why don't we create a synthetic data generator that uses those rules and generates data that fits them. So it's really aimed at testers or developers who don't have much time and want to generate some large volumes of test data super quickly, but don't want to end up writing a whole program to do so. That's amazing. Thank you. And Paul, I don't know if you're able to answer the same question. Who are your customers? Because I know that this is a city project that you developed and so it would be good to know who you actually developed it for. Yeah, sure. So this started off Life City at something called E4, which is our Distinguished Engineering Program, which that's where I kind of offered it. So originally we were aiming at the area I work in, which is client onboarding. So we had some very complicated client onboarding events. So it was a struggle to work with vendors a lot in this space, particularly when it's coming down to APIs, how do we work around a particular data set we want to do? So that was probably our original use case that we had. And then slowly internally it grew to our big areas, so creating portfolios of trades for various asset classes. I think there was some other elements where we're looking at kind of payments and credit cards for some of our clients as well. So working with quite a few clients where actually some of our clients ourselves wanted to generate synthetic data, so we started to engage with them, kind of helping them out. Yeah, and then after that, it's kind of, yeah, I guess we're kind of looking at more and more use cases we go. So we have a working group that James has been running for us and I think now we're going to be. So, yeah, if there are use cases out there, I'd love to hear from them and we can always try and look at the roadmap. That's amazing and Paul, as I asked Andrew, the next question, it'll be awesome if you could share your screen and actually show people the data hub and data helix repositories on GitHub. So we can actually show people how to get involved. But whilst you're doing that, Andrew, I'd like to ask you, are there any potential synergies across other Phinos projects for data hub and data helix? Yeah, I think Paul and I are very interested in chatting to, I guess, the more fear developers and also the legend developers, because from our point of view, if there's areas of Phinos where they're looking in the modelling space where you model things, obviously that that kind of spills over into once you have a model of the data, can you generate synthetic data from the model. So it turns out you probably need a little bit more information than just the model because often the model would describe the type of the data. So it's an integer and it might have a meaning, it might say actually this is a RIT code. But it doesn't give you rules between fields and often complex rules that you need if you're going to generate data that looks like the data you want to generate. There's definite synergies and we're very keen to talk to them. And then we're keen to work out actually what above that modelling do you need, such as do you need a kind of a shape of the data overlaid on the model to give you enough information to be able to generate realistic synthetic data. But it's definitely something that we're very keen on having those discussions. And I think it's a good first port of call to understand how that good sync together. That's amazing. Thank you very much, Andrew. And Paul, I can see in the background you've been given us a bit of a virtual tour of the data hub repository on GitHub. Would you mind just giving us a bit of a voiceover and maybe taking us through the types of issues that you know are within the backlog and how people can get involved with the team. So, I guess, reach out to any of us. So, Andrew Carr, myself, James, there's also Ben Fielding as well from Jensen. So we're all part of kind of guess this working group under Finos. Now, I probably say always the best way to evolve is just get on and raise an issue and to start a discussion. And we're really happy to get a discussion going on there. You know, and then we read what we are particularly looking for is anybody from AIML data science type backgrounds that want to, not even if you're contributing code, but we'd love to hear your ideas, just to help us with creative creative thinking. So, you know, people want to get in and start actually helping contributing codes. Fantastic. That'd be great as well. So yeah, I'd say you always best way to get involved is raise an issue and start discussion. That's amazing. And what type of contributors are you actually looking for within Data Hub and also Data Helix? So maybe I'll ask that question to Andrew first. What type of person would you like to get involved in the project? For me, both, I guess, architects in the working out the synergy with the other components, developers because we want people to use it and find it useful, but also testers so that they can kind of keep talking about how easy they find it. Now, I know certainly from our point of view, again, you know, if there's a way of seeing how these synergies work, but also playing around, I guess, with the Data Helix playgrounds, you can go online, you can change your profile, you can press Generate and actually see, you know, what the rules can generate. So feedback on those rules as well. And that can be from anyone. So Paul, I know that you're controlling the screen at the moment. So if you can remove issues from the URL so we can see the entry point for Data Hub. And then in a new tab, go to Data Helix, because I just want people to know that I'm Data Hub and Data Helix are two separate projects. And so if you'd like to find both projects on the final sort of the first is Data Hub, which is on Paul's first tab, and then the second is Data Helix, which is on Paul's second. And so Tosha, I believe you also have a question. I do both Andrew and Paul, feel free to ask them. I do. Thank you, James. So I worked in a previous part of my career in electronic foreign exchange, and obviously there's lots of big data sets and, and I'm wondering if how easy it would, it would be if I was still there to generate a data set of realistic, essentially tick data for particular currency pairs to run it through a back tester and evaluate if our, you know, algorithms were, were working the way they were supposed to is that is that a use case that could be met through through either of the data. So I'll just talk about Data Helix first of all, and then hand over to Paul for Data Hub, not with Data Helix easily, if I'm honest. I think the rules are such that one of the products which look at the kind of shape of the data and then try mimic it would give you much more realistic data. Unfortunately, because Data Helix uses rules, it doesn't have rules between the rows. So if you did, for example, you know, something that was ticking, it would jump all over the place. It would be 35 one second and then 102 the next second so it wouldn't be realistic enough. Now, certainly, Paul and I have discussed kind of, you know, how you get something to do the correct shape of the data. And I think the tool you would need to do that would need to mimic the correct shape of the data, such that the data wouldn't just jump all over the place.  So for the moment, Data Helix would be a no, but you know, looking forward, I think with Data Helix and Data Hub looking to come together, I think you could end up with enough components, including Data Helix to be able to do that. But yeah, I think Paul has to talk about the data, Data Hub. Yeah, so for us, when you, I think I showed earlier how you can write your own extensions. So I don't do it quite now. But it's actually quite easy to write your own extension functions. So really, I guess if you're looking at time to his data is the last couple of values you've generated are an input to the next value you're going to generate. If you do it very same, we basically pick or have a number pick a random number between one or 2% from that number, maybe from the previous number, if that makes sense, maybe you would that maybe you want to generate something like that. So you can do we can do that with your own extension function. I don't handle the analysis yet. There is a library. There is some Python work that's been done by some people out of Cornell University that does do analysis of time series data, and then again through GANs again. So I am actually kind of looking at how that could be integrated in Data Hub as well. So, and if they're things mature enough yet to use it. So the answer is kind of a kind of you can. Fair enough. Fair enough. Thank you. I have a another one back from my FX days. There were, we, you know, we had electronic trading happening in multiple different areas across the firm, and some of that information you couldn't. So actually on trade data, not real time, you know, ticking data but trades that were executed. I don't know about them, but because of data privacy laws, you can't pass that between that information between jurisdictions. So, is that a use case where I could essentially, as you were talking about earlier and, and this is not an area of expertise but the privacy plays into it. Is that aware of a use case where I could pass the data over from one region to another, knowing that I have safely not passed any client information or specific regional information that wasn't allowed to be shared. Got it. So I think that's more a problem that would be lend towards redaction and anonymization, because what you what you don't want to do. And with that you want to, you want to straight remove it. Yeah. Otherwise we'll be making up the trades and it'll be a bit scary. While theoretically the shape of all the trip you did like generated 10,000 trades hopefully they'll be assisted the same but can quite guarantee that. And then Andrew I saw Stephen go bump from more fur actually came on cam to ask you a question say Stephen are you there. Yeah, yeah. So one of the things that we've been interested in more fur is the idea of in domain modeling that you want to sit down with your business owners, and kind of come up with a model in real time together. And that would be generating sample test data so as you're coding, you know, imagine like a decision table. It's got a finite set of values. And it would be great that as you're coding and adding that finite set of values in an enum or something like that that you can generate the sample data on the side, and that can run and your users can see that in real time and see that. The rules that we're coding right now actually pass those tests. Is that something that you would see would be doable. I mean you can definitely do that with the data helix now with the playgrounds online playgrounds. I mean it's depending on how complex the rules are would determine how quickly it responds, but you can do that you can you can. You can do a live demo just because I'm scared of doing live demos and things always go wrong when you try them live but you can just edit the rules in the web browser press run, and it will regenerate the data. And I totally agree with you Stephen I think, ultimately certainly with users when they see the data in reality that's when they go, oh no that's not right that can't be that because of that and that's when you get really useful information that they just happened to have forgotten to tell you. When you go and code it and come back and they go, no, no, not like that that yeah if it's interactive you totally get that response. Yeah, cool. Thanks. So that's great and so before I ask the next question I just want to remind people in the audience that you can actually ask your own questions by using the Q&A facility that's in front of you on screen. Thanks for your time so Andrew I understand data helix is actually written in Java and Paul, we know that data hub is actually written in Python. As you've been exploring the synergies between both projects, can you kind of talk about the ideas that you've been having to bring in both projects closer together in terms of functionality. I guess for me, I think Paul and I have been talking about three things in terms of synergy one is architecture, two is API points and three is language. I think the data helix was originally written in Java, I guess for a couple of reasons one because of the project it was involved with to start with two because the resources are useful Java developers I had them available. It kind of made sense. And three at the time, quite a few of the organizations we thought would use it. We know were heavy Java users. Looking back now though, I probably would have done it in Python. So I think the move forward is to sync up with the data hub and really move any data helix components we want to keep into the Python so that yeah so that the data helix and data hub become components in this, you know the thin-off synthetic data components I guess. And Paul, is that something that you agree with? Okay. I think he agrees with it because we're moving all the components over. Yeah, sorry, I muted myself and actually I muted myself to talk, I didn't realize I was on mute already. I'm muted already. So yeah, it's obviously something I completely agree with. And I think one of the benefits of Python in this space is just the weight of that data science community that's behind it. It's by far clear the winning language. And I think also within most financial firms now, most of us are using Python to glue quant libraries and all sorts of things together. So it's just a, I think in that space is just a clear winner. Mainly because I think also with Python it's if you could particularly found in the synthetic space every time I go, I'm going to code this thing together. I'm a quick Google search fan so that somebody's already built it and I can just install it and then expose it, you know, make a few modifications around the public and expose it through Data Hub. That's brilliant. And I'd like to remind everybody, you know, who's actually watching us speak that data hub and data helix are open source projects and so they're here for people to utilize now. I don't need too much on the spot.  So, okay, so the most simple way is starting Data Hub is. Let's just do this quickly. And it's going to generate a Python virtual variant, right. So we can see Chrome at the moment. Hopefully, interesting that you can see Chrome. So apologies, let me stop that share. And start the share again. Share the screen. That's it. I can see there we go. Excellent. So, activate that. So, getting some data is simply as simple as install data. I can't type today. And you can see it's bringing it down.  So you want to start with it. That's what you can do. And then all you need to do is go into one of our test folder folders test, pick a test. Cut and paste that in. Put it into a Python file and then just run it and it will work. So that's if you want to run your own. Otherwise, again, the easiest way to normally start with stuff is, let me maybe start another window up. Where am I? Data Hub. So if you were to clone this again, is again, start with your virtual environment and just run PyTest. And that will run all the tests and show you that it's all working. So just explore all those test examples that are there. So that's actually my death machine. So that's a broken. So of course it's not going to work because I know. Yeah, it's lucky I didn't start from there. But there's a good example of the tests are broken. But that's fine. And so both the Data Hub and Data Helix community is very welcoming for people to join the repositories, bring them down locally, install the virtual environments and run tests to get it up and running. And then also if they find issues to feed that back into your, into the GitHub issues for the project and get involved with the community. Exactly. So, and that's brilliant. And so as we lead into the final two minutes of our session this afternoon. So Paul and Andrew, do you have any kind of final asks, you know, for people who are actually watching us this afternoon? Is there anything that you would like people to get involved with where you need help? Just so you know, we've got to know an entry point for anybody who wants to get involved. Yes, Andrew, do you want to take that one? If not, I can. I shall take that then. So I think some good areas that we're really interested in. So there's certainly that, that link up with Legend. Morphe also sounds really interesting. I know we've got a couple of people on from there. So again, you know, what synergies are there for that area. I'd really like to know kind of any missing functionality. So real, real primitives that might be missing for somebody, you know, if there's anything there. Another thing that's going on is creating a set of examples based on the Eastern CDM specs. So that was the thing that also we engaged with in early in the year. So could there just be a whole load of examples of like this is how you make a this type of is the contract or that type of is the contract.  You can start since actually generating that that is the object. So again, that's that's a really nice easy area to get involved as well. So anybody with is their understanding. Fantastic. Yeah. I'm trying to think of some of the other areas. Andrew, have you got some days? No, I think you've covered it all for actually, I think, yeah, just feedback. If anyone's got any use cases like Paul says, the more use cases we do the more we kind of get to have experience. On, you know, what would work well with the projects and what we know how the projects need to change and evolve to hit new use cases. Yeah. That's, I was just going to say that it's been an amazing past hour of being escorted through data hub and data here. Thank you very much Andrew Carr and Paul Groves for taking us through both of your projects and with that I'd like to hand the group back over to Tosha Ellison. He's going to introduce the next section for us. Thank you. Thank you very much for that. So in our final session, I'm, I'm, I'm pleased to have a presentation on perspective which was one of the earlier contributions that came from Jake Morgan into the foundation and so we've really had an opportunity to develop and see the community grow, which is, which is always nice and it's certainly something that is an area that I'm interested just from my personal background and, you know, which has broad applicability across industries is how you present complex fast moving in ways that's easy to interpret and how do you give access to your, you know, your users to be able to to make sense of that data too and of course that ties into some of the conversations in the previous one so with that, I'm pleased to show this presentation from Jun Tan in the perspective team. I will stop my video. Hello everyone. Thank you for virtually attending. My name is Jimmy and Tan I go by June. I'm a software engineer at JP Morgan and for the past two and a half years I've been part of the team working on perspective, an open source streaming data visualization engine that's built for large streaming data sets. And being a final subconference I figured that there are people in the audience who've already used perspective. You might have heard of it you might have used it a little bit you might have even contributed to it for example. And so for this presentation I want to talk about something that's pretty new in terms of our project we've been working on it for a little over a year now and it's taking perspective out of the WebAssembly context and porting it to Python and more specifically using the using perspective Python was in JupyterLab to do streaming data visualization in a whole new domain in a whole new way. So I wanted to start off by talking about the sort of two predominant ways that you can do streaming data visualizations and interactive visualizations which is a Web dashboard and in JupyterLab or using Python. So Web dashboards are great they are interactive they're easy to use you can have really visually beautiful visualizations arbitrarily complex UIs, but at the same time not everybody wants to write JavaScript and not everybody can. Python and data science is really the de facto language the de facto standard, and it's difficult to have somebody, you know have to learn Python and do JavaScript and sort of the UI part and the data part at the same time. Look at JupyterLab for example it's extremely powerful it's Python is a de facto standard JupyterLab is the standard, but at the same time it has to sort of drawbacks that Web graphics and sort of Web dashboards really excel at which is that it's more difficult to write UIs in IPI widgets and as a data scientist as a developer you might not want to spend all of your time doing custom UIs for each individual notebook or even, you know, kind of hand rolling your own UI library to do these sort of visualizations and transformations. So perspective in JupyterLab really kind of takes the strengths of those platforms and provides something that's not only easy to use not only interactive not only, you know, intuitive and has really a lot of power just in doing the sort of analysis and visualization, but it allows you to do it on top of the power of the compute power the threading the parallelism of Python, as well as the extensibility of Python, as long as you give perspective some data that you know it can visualize it it can pivot it can transform it, and you can use Python to create arbitrarily complex data sets arbitrarily complex data sources and as long as you give perspective some data it'll work and you get the UI part you get the dashboard part almost for free in in JupyterLab. So very quickly, for people who aren't familiar perspective is a interactive visualization engine for large streaming data sets. It was built at JP Morgan for the trading business it's still used at JP Morgan and by developers institutions around the world it's open source it was open sourced in at the end of 2017 and now it exists as part of a fitness project and allows you to not only transform data in the browser and in the engine so it allows you to apply pivots filters aggregate sorts computations using a really high performance C plus with core engine that's compiled to WebAssembly. It also allows you to visualize data we have a custom data grid and we have a D3FC based data visualization package which D3FC is built by Scott logic who I know someone from Scott logic speaking at this conference I'm sure there are Scott logic attendees and it's a really great visualization product that we're, you know, extremely happy to use and very happy to have people from Scott logic contribute to perspective. And it runs in the browser initially but now we've ported it to Python as a standalone runtime and in specifically in JupyterLab so to kind of look at this how to do streaming visualization in JupyterLab with perspective, we're going to take a very common scenario we're going to take a fictional portfolio stocks, a basket and we're going to visualize and we're going to analyze it and we want to do two things we want to do real time analysis we want to have streaming data that comes in and we want to do we want to do pivots we're going to do aggregations etc in real time. But we also want larger data sets of historical data and we want to look at that and you know do it in the same sort of context do it within JupyterLab. We don't want to have to write different data sources we don't have to do different fetch API for example in a web dashboard or connected to different data sources. Using a web framework we just want to do it always in JupyterLab and most importantly, we want to do as minimal transformation in code as possible. So there is data cleaning there is sort of getting the data into the right format for perspective but we don't want to do more calculations in the data we don't want to have custom calculations on a data frame for example or a JSON object we actually just want to take the data pass the perspective and have perspective do all the heavy lifting and you'll see that it's actually really quite easy and very intuitive to have that sort of framework. So I'm going to start off by doing a very quick rundown of the perspective basics the API. If you look at the browser versions if you've used the browser version the UI is exactly the same. The API in Python is pretty much the same as the browser version and we've designed it so that the two are completely cross compatible. So I'll start by just importing and creating a table so obviously this being Python and Jupyter you can now give pandas data frames to perspective we handle that very quickly. And you can either create a table the sort of base container for data from a data set so from this data frame here or you can create it from a schema which is just a mapping of column names to column types. And with the schema we're going to create a table that has an index and this is important later on because we're going to be creating with various data sources we're going to be creating a unidirectional data flow where the data comes from one source the IEX cloud API that we're going to be using to get our streaming and static data. And it flows through a series of perspective tables almost like a graph if you if you will to actually create to store data to manipulate data to allow you to do all sorts of different analysis kind of all based on this one day to flow. So if we get started we create this table and we create a view which is a query on the view on on the table excuse me so in a sequel analogy it's kind of like a table is a sequel table and a view is a continuous query where you can update the underlying table as many times as you want but you never have to recreate the view the query can stand on its own and it gets notified of new data coming from the table. And you can always recerealize the view out into whatever format you'd like and it will always be up to date. So we'll create again a unindex table and index table which has a primary key and we're going to update it so if we update the unindex table and we create a view and we query it we'll see that it's sorted here from 2500 all the way down. And if we update it again but we don't recreate the view we just re query it you'll see that we actually get the most up to date data so you've got 2500 and 1500 and created a couple of seconds later. And it appends on a unindex table but an index table it will actually update based on primary key so if I do an update here you'll see that it overwrote some of the rows and appended some rows and it left some untouched. And this is again really important because we're going to be using primary keys and indexes to store some of our data in certain ways where we only want to store one row with a given primary key. But in another table we want to store every single row that comes in regardless of primary key so for example in this scenario we want one table that has the latest price for every single stock. We want one table with every single price change for every single stock and that's where the index really comes in. And we're going to do this sort of linking like I've talked about using on update which is just a callback that runs every time the table updates. And this is kind of where the sort of joining comes in, the joining data sources. We have a table that has an on update callback, we set it on the view and the callback runs and it takes the rows that are updated and it passes it to a different table. So it passes it to the index table. So now if I update the unindex table and I query the index table you'll see that it actually updated the index table with all this new data. But it also looked at the primary keys in order to apply it properly. So if I were to actually look at table.view2df you'll see that I've got all this data right and because it's unindexed it appended all of it at the end. But within the index table we have it updating based on the primary key. So this is really important we'll get to it in a second. You can serialize your data from a view so you can do to data frame to a CSV to to JSON and that's all great. But the most important thing is actually you can serialize your data to arrow so perspective integrates is Apache arrow extremely extremely well. We load Apache arrows with almost a zero cost load by copying the underlying memory into perspective. So loading Apache arrows is 50 to 100 times faster than loading a regular data frame. But we also allow you to write data out to arrow and we allow you to pass these arrows back and forth. And again it's a really easy way to work with Apache arrows which is you know a super high high performance binary data format. But being a binary data format means that you really have you have to use their library to to parse it to do to write it to do different things. And perspective offers a visualization a transformation layer that's extremely intuitive on top of Apache arrow. So we save the tarot we can save it to the file system. If I load it you'll see I've got example dot arrow. And if I re if I if I load it from the file system you'll see that this is all the data that we just had. And we can sort it we can do whatever you know anything we'd like on it we can group it we can filter it for example we can do as do as much analysis as we like. And this is kind of the the the meat of what we're going to be talking about which is using perspective widget and using the perspective UI but within JupyterLab. So you'll see that here we're going to create a perspective widget it's the exact same UI that you're used to on the browser. It has all of the same features as the UI on the browser it's completely cross compatible. It allows you to not only not only not only interact with it on a UI level so I can take away a pivot I can add a pivot I can add a filter maybe I want it to be 2.5. Or maybe I wanted to sort it by B or get rid of the sort or add another pivot or I guess D is no in this case I guess we can pivot here but it also allows you to interact with it from the Jupyter kernel. So if I typed in widget row pivots you'll see that I can I guess C and if I typed in widget row pivots equals none it unpivoted everything. If I type in widget reset it'll reset the entire widget except I guess not that one for some reason. But you can transform the state from from Jupyter but it also means that you actually can save the state within a within a library. So you could very easily, for example, instead of just initializing the widget you could you could have these options you could set them as you wish. Contributes right or you can do it later on the fact so again it's a it's a whole different way to interact with perspective instead of having to sort of save your. Interactions or the state of your actions for example in in a in a way that might be cumbersome in the browser doing it in Jupyter doing it in Python is extremely easy. And we can also do streaming data in in perspective in Jupyter so obviously streaming data is a big point here. But the issue with doing it in Python or doing in Jupyter is that we can't block the main thread we can't let the notebook be blocked while we're writing this new data while we're rendering this new data. So we're going to use hardware threads on Python and we're just going to create a new thread we're going to run perspective in a separate thread and we're going to have it update the UI as it goes. So if we create this widget right now it's completely empty and if I start this thread you'll see that it's actually running on the separate thread as we go along. And I can do any sort of now site likes I can sort it I can pivot it again I can do all these different things with it. And I can also while this is running I can use the kernel I can still continue to use the kernel as much as I like you know I can check the size I can create a new view. Let's see if I did row pivots equals what's a good column name. And I did to Jason and 100. So it's querying it and because it's multi threaded it's it's you know dispatching those was in a way that's not going to block the main browser thread and not the main browser thread to make the main kernel thread, which is really important. And so from that sort of overview or perspective now we can look at very quickly the data sources that I wrote, which use the IX cloud API to give a streaming market data and also static market data. And it's all dummy data it's really easy to get a sandbox URL and to get a sandbox key. So we're going to be using that but again the most important thing was these data sources is that they don't block the main thread. And again we're going to use multi processing we're going to use threading and we're just going to hook them up together. All of this code is available online all of it's open source so definitely feel free to take a look at it. You know play with it play around with it if you want. And so we're going to get on to the sort of main part of this presentation and we're going to start by doing our imports again and importing our schemas and our data sources and creating a PYX client and PYX is the IEX wrapper library. In Python and it's really easy to use really simple to set up and gets us our data really quickly. So we're going to use that it's great it's great library and we're going to create a portfolio stocks just some random stocks here tech stocks. Really no particular reason random randomly generated it's all dummy data and we're going to start by setting up our data sources. So like I said earlier we're going to create a few tables and we're going to link them up properly to create this data flow. So we're going to start by creating a table around what stocks we have right now a holdings table. And we're going to want to index it because we just want to know right as new prices come in. If the prices all append we're going to have to do some pivots but if we didn't we don't care about that information really in this table. So we're just going to store it with an index which means that we're always going to get the last most recent price and most recent value for our portfolio. And so we're going to update here and then we're going to create an unindex table which is going to store all of the price changes for all of the stocks as an our portfolio. Which excuse me is going to be important because we obviously want to know how it changes over time. And so we're going to create that and we're going to hook it up with an update like I talked about earlier. When the index table updates it'll take the row and it'll push it to the unindex table which is basically just a dump for all of the data that's come in so far. And I have also a very interesting architecture diagram that I cooked earlier and we can look at it here. So the architecture diagram is basically we have the IX cloud API and it's all one way. So the data stream comes into quotes table which we're going to be creating in a second. When the quotes table updates we're going to update the holdings table. When the holdings table updates again that's the index table we're going to update the unindex table. And from the unindex table we now have all of the data that we've been getting and we can dump it to disk using Apache arrow which allows us to stream it, allows us to store it, allows us to pass it over a network boundary, to pass it to different notebook, to send it to somebody using perspective, to host a server using perspective on that data. Really there's a lot of possibilities. So if we create these tables we can now create a widget. And so we configure this widget and most importantly we configure this widget with a computed column called value. Right which is a column on the table on the widget that actually calculates the value of the portfolio by multiplying quantity and price. So this kind of goes back to the requirement we had earlier where we don't want to have to touch code as much as possible. We're not doing this calculation in Python or at least in user facing Python. The user doesn't have to write a code that transforms the data before it goes into perspective. They can do it completely within perspective. In fact they can do it from within perspective using the expressions API which again is actually something new that we've built this here which allows you to write these arbitrarily complex expressions that will update as your table updates and it will resolve itself properly. So if I were to do, see if I deleted that and I did the square root of value for example and I push it out. It will actually give you all of these columns and you'll see when we start updating the table these columns will update at the same time, in real time basically. So now we can create another widget for our total table. This is going to be configured as a line chart that's just going to look at all the prices as they take it. So again looking at the sort of data flow what we've set up so far is we've set up these two tables and these are going to be our holdings. And now we can set up a quotes table which is the table that's actually going to be taking data from the IEX API. So again using an update when the quotes table updates we're going to send the data to the holdings table. And when the holdings table updates we're going to send the data to the total table and so on. So we can create two more widgets here. One of them is just going to be a grid, sorry, just showing us the latest prices. And the other one is going to be a chart showing us the latest prices coming in and we can start to data source. So we can do a little bit of cleaning. The sandbox data doesn't have the right date time set and so we want to have it sort of look live in this case. They're randomly generated date time so here we're going to actually just have date time now as a sort of proxy and we're going to start this data source. So now you'll see this data is coming in live right now, right? It's updating here. It's updating on our quotes table. These two actually being fed from the same table. It's updating the value of our portfolio over here. So this is the value of our portfolio right now over time as it, you know, increases or decreases. Because it's dummy data will probably stay pretty still but you'll get the point. And here is our columns. I think I need to change this to not know and now it'll update. Zoom is at zero for some reason. I think that might be a data issue. But you'll see that as our prices update on this index table, all of the computed columns are updating at the same time just in lock step. Right? And again, going back to one of our original requirements, the main browser, the main kernel thread, my bad, is never blocked. I can do as much, you know, queries on this as I'd like. I can do quotes table size, for example. Ah, quotes, excuse me. And I can see it continue to tick. I can again do the same thing. I can get another view out of it. Let's do two dict and row or start row 100 and row 150. And that's 50 rows in a dictionary out. And I'm going to clear that output just so we don't have to look at it. And we can also do other things like we can take this, we can take this output, we can put it in a separate view, we can have it over here. So now we can look at our data as it is ticking live, that might again be sort of dummy fluctuation. And we can also do, we can also create more computer columns. So for example, if we didn't care about the data ticking in every second or every sub-second, we can have it actually look at it per minute, for example. So if we bucket by time, and we change the pivots here, we can see that now everything's being aggregated by the total value over time. And I think this might be wrong. There we go. Yeah. So it was because it was doing a minute, it was adding together the value at every second. So we change that to last, which is the basically the last value in the table for that minute. And this will give us the correct value for our portfolio. So you can see, and we can also get rid of the split. So now this is just the, I think this might be a sum, and we might just have to repivot it. But you can kind of see the point of it allows you to do as much analysis as you'd like on in JupyterLab. And you have the UI, it's given to you for free. You have the ability to do complex visualizations. And you also have the ability to do static visualizations at the same time. It's just as powerful. We can back test using historical prices. So a little bit more cleaning here, and we're going to get the data for the past five years. And we can create another widget here, and we can render everything. And now we can see this is, excuse me, the open high low close for the S&P 500 index ETF. And again, if I didn't want to look at it every single day, I could maybe look at it by month. Bucket, actually, I think. Yeah, month bucket date. And if I were to bucket it here, you can see the actual open high low close for each month. And I can take that, I can pivot it out by symbol. Now I can see it on all symbols. I can also, again, filter it down to something more specific. So if we can look at SPY, we can also look at Snap in this case. Again, completely different charts. And then if we wanted to look at it on a more granular level, again, we can just take off that pivot. We can add this pivot back in. And now we've got an extremely granular view of per day on the open high low close prices, right? But again, all of this is extremely easy to do using perspective using the UI that's already been provided. It allows you to hook up a complex data set from Python from Python kernel and allows you to do it with very minimal transformations within the code by the end user. So imagine an end user who's looking at a notebook or a voila app, or if you're preparing a report using a notebook, for example, you can give them the notebook and instead of kind of looking at pre-generated charts or having to dig into the notebook to regenerate your charts, you just show them perspective and they can do all of the transformations, do everything interactively within the UI. And one of the other really cool features that I wanted to talk about was this sort of integration with Apache Arrow. If we create a view here that just has the value, we don't care about the price, but we're calculating the value and we save it on another thread. We've just saved everything to the file system and we can open it up. And again, there's quite a lot of power to this, right? We can do exactly the same analysis that we did earlier. So, well, maybe not like that, but we can do it here and split it by time. And now we suddenly have all of the prices that, you know, we've been looking at, except in a static form, obviously. It's not updating live, but you can see the sort of power in this where if you wanted to, for example, build a notebook that ran live analytics over a day. And at the end of the day, you saved it to disk and you ran it for 20 days and you wanted to look at every single day of analysis. You don't have to go to the web API or whatever data source you're fetching it from and try to reconcile everything or try to maintain a separate database. You just store 20 arrows and you load them in a row and you can put them in the same widget. You can put them in the same perspective widget and there you go. You can host a web server. You can run a remote perspective using perspective Python and perspective in the browser. There are a lot of possibilities. And so one of the things you can do here really simply is we're going to restart it just to make sure. Open it in a separate notebook. So now, again, we have all the same data again. You can do all the same pivots. You can do everything in the same way. Or you can actually open a perspective workspace that's running on a perspective service. It's getting data from a perspective server. And actually this whole time I've been running a server that is in the background that's using perspective Python. And it's actually going to have, I think the render thread might be blocked, but it will be. There we go. If I update all the points, you'll see that it's been running for about 40 minutes now. And it's still getting a fault because I can tell it should still be getting live data. And if we were to, for example, bucket this by minute, you'll see that we have each minute here. And so this really brings together, again, the sort of paradigms of a web dashboard and a Python back end. But it's more powerful than what you can do maybe in Jupyter lab. But there is a little more setup with a web server and you have to deploy it or you have to run it somewhere. But it allows you to create even more interactive dashboards that are fed by Python, all the status synchronized. It's extremely powerful in that you can, you know, if I used last, there we go. It's extremely powerful. You can, you know, have all of the power of perspective workspace. You can have this, for example, be a global filter. You can click on it to filter on the rest of the dashboard. You can package a dashboard. You can deploy it at an endpoint. You can send it to somebody. You can dump the data out of here into a CSV, etc. All of this. It's all, you know, there's a lot of possibilities there. And so I think this really brings my presentation to a close. I wanted to thank you all for attending. I wanted to thank the organizers, both at the Linux Foundation and at Finno's for putting this event together and for inviting me. Everything that I've talked about is open source. Every single part of the code that you saw is open source. Perspective is fully open source. The notebooks, everything on this presentation is online. So feel free to take a look at it. Feel free to reach out to me or on the perspective repo or with any questions or any ideas. And thank you guys for attending. Thank you so much, June. That was, that was fantastic. You've already answered. One question online, maybe in case anybody missed that, do you want to just touch on touch on that question and answer? Hi, I don't know if you can see me. Okay, awesome. Yeah, perfect. Thank you. Yeah, I so the question from let me read it out again from Adam Jones was about whether we can create recordings that can be rerun or view it at a later date. And I'm not exactly sure what recording means in this sort of context. In Jupiter lab, like the notebook state is always persistent. So as long as the kernel was alive, it'll be there with the perspective viewer with the widget actually the way we've implemented it is that the viewer state will be persistent even after the kernel was dead. So the Python kernel could be, you know, crashed or stopped or whatever but because perspective is a wasm based, you know, engine and we were actually able to run the exact same data set in wasm at the same time as the Python kernel. So you're able to basically see your state you're able to do any more transformations whatever you'd like within the UI. You know, long after the kernel is dead. And you can also serialize your data out to different formats you can save the config of your widget. And it's kind of part of that's enabled by the platform of Jupiter lab and this sort of notebook architecture and part of that is the way that we've implemented perspective widget and I, I don't know if that fully answers the question but I was my take on what kind of recording meant in this in this case. Thank you. And Adam, we can always put you in touch with June afterwards if you'd like to carry on the conversation but he says yes that does so thank you. If there are there, I know we're just we're a little bit over time so are there any other questions before we wrap up. Great, June and everyone else thank you so much for for joining us and as everybody has reiterated this is all open go check out the repose figure out how to get involved. You've you've heard from our, you know, project leaders that everybody is friendly and, you know, and really welcomes contribution so please do take the opportunity to to join our community. And with that, I will thank you all very much for attending.


Hi, I'm Brooke Chatterton, Intellectual Property Lawyer at Protocol Labs. Was it a big firm before this, before Monica saved me from the legal coal mines? Excited to be here. Matt? Yeah, my name's Matt Sumner. I'm a Lawyer at Protocol Labs as well. Before this, I was at SCAD and doing intellectual property transactions mostly. I have always had an interest in science and technology, and so I'm happy to be here with you all. So let's get started. We're happy to kick off this workshop by discussing what patents are, why they are what they are, and how they're not necessarily living up to what we'd love them to be. So welcome to the imperfect bridge across the innovation chasm. So I'll hand it over to Brooke to talk about why we have patents. Awesome. So just starting at the top to promote the progress of science and useful arts, which is our base in the Constitution for having our intellectual property system, which is super cool to put at the top of a talk because the framers were thinking about these ideas when they were writing our Constitution, which is very cool. And if you're a legal nerd like I am and many of the lawyers are here, we like to talk about those things. But why do we have them? We need some mechanism to bridge the gap in between academic progress to get us to a functional technology. And we use the patent system, which primarily relies on economic incentives to do that. To take a little flow through how patents are supposed to work, the inventor has a great idea. They know that there is a product that will fit a need in the markets, and they are so excited about introducing this idea, but they're scared. They want to invest their own capital, they're going to invest their own time, but they're worried that if they introduce this idea into the marketplace, someone else will snap it up and all of their time and efforts and ideas will be for naught. So that's where patents step in. They grant a limited term monopoly to buy time for the product development, and also to increase the chances of the inventor getting a good return on investment. And that stimulates the good flow of capital because the inventor is reassured that they can put their capital in time in and it attracts external capital because venture capitalist companies are see the patent and they're like, okay, this is a good signal for me to invest into this inventor and his new idea. A corollary to this is these new inventions are shared quickly and in depth by the public patent filing. So that's, you know, it's an instruction manual of how we do a thing that goes into the public realm. Hopefully other researchers and inventors are reading those patents, and then they are able to further develop technology because they're building on those innovations that are published patents. Next, we're going to zoom out a little bit, talk a little bit about more incentives. So over to you, Matt. Thank you. So as Brooke said, let's zoom out. The goal of the patent system, as you mentioned, to promote science and the useful arts. Are we doing that? How do we do that now? We're currently using a proxy, right, of widgets, essentially. We're saying, let's push out these widgets into the market in order to push humanity forward. And that is a proxy. It's not an exact corollary to what we're really trying to do here. And so we do that through, as Brooke mentioned, economic incentive. What else could we use to, as incentives, to get what we want? It doesn't necessarily have to be widget sales. It could lead to widget sales. But what else do people use as incentives elsewhere in the system, not necessarily just with patents. Let's take a look. So one thing that we've talked about in recent days is reputation, recognition. In academia, this is very important. People publish or perish. They want to be seen as experts in their field. A lot of people publishing are tenured. So they're secure economically. And they still want to, they still are incentivized to build on knowledge because they want to be seen as experts. It either makes you feel good or whatever it is psychologically. You want that. There's also religious or political motivation. So the US government doesn't wait around for the private sector to be motivated by the patent system to sell them weapons. So what do they do? They fund things like DARPA. They fund other projects, pour billions of dollars into them to make technology move forward. Everybody can have their opinion whether that's good or bad, but that is something that can be done. One third way that we'll talk about is altruism or at least perceived altruism. So if you feel like you want to donate to cancer funding, for instance, you can donate to funding for rare diseases. Pharmaceutical companies otherwise may not be incentivized through the patent system or otherwise to develop these drugs and have them save a thousand lives a year because they'd have to charge a million dollars a pill or whatever. So let's zoom back in though because cash is king, right? So let's talk about how we currently work in the patent system using economic incentives and some ways that we're implementing it, perhaps not as well as we can. Awesome. So problem one, transaction costs. So fundamentally, the patent system does introduce some frictions into the marketplace. There's time frictions. There's money frictions. So if you're an inventor, it costs a lot to get a patent and it takes a while, which is just kind of the name of the game. What's worse is the litigation costs associated with like the only real winners here are the lawyers, which is great for the lawyers in the room and it's probably terrible for society to be honest. But there are costs to defend if you own a patent or if there's a vague patent out there to challenge that patent is, you know, infinite dollars essentially. That's why I've included more dollar signs and I've seen on any restaurant in this slide. And the example here is the smartphone wars.  And there are like three losers in that the consumers lost because your phone is probably slightly more expensive because those costs are internalized and passed on to the consumer. The shareholders lost out of all those public companies, you're supposed to distribute your profits to your shareholders, there's less profits to go around. And kind of worst of all is society is worse off society. Like if we're pushing humanity forward with technology, no one cares like what shape your phone is, like we're spending time arguing about these little things and not using that R&D funding to advance society. The corollary to that is we're writing these patents to write good patents and not to advance our technology as far as we can. There's just a lot of just frictions that are brought up in this system that are not ideal. Second of all is duration of patent terms. So in the US for utility patents, you're looking at a 20 year standard. There is some exceptions to that in the pharma space, but for the most part, it is not industry specific. So what that means is you're treating a pharma product the same as like a bare bones solution. So it makes sense when you have a lot of R&D expense, for example, in pharma to have that really long term monopoly for 20 years, just because there's a significant amount of R&D, and there needs to be a significant tail in order to capture a significant, like a fair return. If you have like a SaaS solution, I know some of the engineers in the room could probably mock up a product in a couple of days that they could get a patent on. And that's being treated the same way as like a whole team of like very intense scientists with the clinical trials. And that just doesn't seem like the right way to allocate capital. So at the end of the day, capital is not going to be allocated efficiently, because we're treating everything the same. Back to Matt for the next problem. Thank you. Yeah, so problem hard to tell who owns what, right? So there are approximately a zillion patents in existence, right? What does that mean? It means that you don't know where your patent lies and where you're restricted in relation to others. Or it's very hard to know. It's very costly going back to transaction costs. Not only is it costly to figure out, you are worried about being sued. So going back to what Brooke was saying, if you're worried about being sued, you're maybe not going to pursue what the patent grants you the ability to do. Or you're going to do it in a way that avoids litigation. That is not ideal. That is not optimizing for moving technology forward, right? Oh, yeah. So one element to this, right? There are very vague patents out there. When patents are vague, this is like the worst of all worlds, right? Because you don't know what you have and don't have. You're constantly under the threat of litigation. That's not great. So one reason why we have these vague patents and like a million zillion patents out there, there's a bit of a principal agent problem here, right? So the USPTO and the government may have all the best intentions in terms of what patents they're granting and when and to whom. That doesn't mean that we're getting that. The patent examiners are incentivized to be granting these patents. They're incentivized because oftentimes they're not paid maybe what they should be to not fall victim to these incentives. So therein lies a bit of an internal issue there, but it's actually leads to many bad outcomes. It leads to a bunch of these issues. So our last problem, everybody's favorite patent trolls. And thanks to the EFF for this image of a beautiful troll. So patent trolls, if you're not familiar, are NPEs or non-practicing entities. They're typically shell companies. They acquire patents and then they extract value. They're rent seekers essentially. They extract value through the threat of infringement lawsuits or the lawsuits themselves. So this is not good, right? Like you're going back, a lot of this goes back to transaction costs, right?   They may have a case against you. You may not know because the patent might be vague.  This is really bad. This is bad for attracting VC investment, right? Because those investors aren't going to want to send their money away to pay to a patent troll once you get successful enough. And patent litigation is notoriously expensive, just putting that out there. So that is the conclusion of our problems, of our talk about what the patent system is, why it is that way, and some issues with how it's implemented. So we'll open it up to questions and thank you.


Naval history and heritage commands underwater archaeology branch received more than 30 artifacts from CSS Alabama, June 23rd. These historic treasures came from the Warren Lash Conservation Center in Charleston, South Carolina. The artifacts included two wooden boxes, a partial gun stock, a copper alloy belt, and pieces of wood from cannons. The two boxes contained two pieces of inactive ordinance. CSS Alabama was commissioned in 1862. In 1864, she engaged in battle with Sloup of War USS Kiersarch off the coast of France and sank shortly after. It was rediscovered by the French Navy 120 years later. For 21 years, American and French archaeologists have studied the conditions of the sunken Confederate ship. For more information, visit the Naval History and Heritage Command website. From the Defense Media Activity, I'm Petty Officer Jen Blake.