Hello and welcome my name is Shannon Kemp and I'm the Chief Digital Manager of Data Diversity. We would like to thank you for attending Database Now Online, the first occurrence of this online conference produced by Data Diversity. We are very excited to kick off the event and have a great lineup of sessions for you today. And of course a special thanks to all of our sponsors today to help make it happen. Just a couple of points to get us started. Due to the large number of people that attend these sessions you will be muted during the event. For questions we will have a short Q&A at the end of each presentation today and we will be collecting questions by the Q&A in the bottom right-hand corner of your screen. Or if you'd like to tweet we encourage you to share highlights or questions via Twitter using hashtag DB Now. If you'd like to chat with us and with each other we certainly encourage you to do so. Just click the chat icon in the top right-hand corner for that feature. And for this event we will send a follow-up email next Monday to all registrants containing your unique login to access the recording and the slides from today's presentations. Before that may introduce to you our very first speaker for today, Dan McCreary, who will be discussing data lakes and data hubs, which one is right for you. Dan is an enterprise data architect with a focus on NoSQL database architectures. His interests include topics such as distributed computing, programming languages, databases and XML standards. He's the co-founder of the NoSQL Now conference and is a co-author of the book Making Sense of NoSQL by Manning Publications. Dan was also a principal consultant for Mark Project. Dan is currently an independent consultant with a focus on AI and natural language query front ends to data hubs. And with that I will give the floor to Dan to get today's conference started. Hello and welcome. Thank you, Shannon. I'm looking forward to presenting and this is a great topic and I think you have a great audience for us today. So let me just give a brief overview of what we'll be discussing here. The first thing we're going to be talking about is kind of an introduction to the terminology of what is a data lake, what are definitions. We're going to really emphasize why some of these are superior architectures than trying to use relational databases for certain business problems. I'm assuming that most of the audience here is familiar with relational databases and their strengths and weaknesses. But what we're going to do is talk about data lakes and data hubs and their common attributes and then talk a little bit about how they're different and then we'll go into a little bit of a case study of an infrastructure for feeding data into these AI and deep learning systems that have become very popular right now. We'll wrap up with a proposed objective selection process to help you try to minimize the bias in the selection process. So when I start out these presentations I often start out with how many people have actually heard of the data lake and data hub sermon. I think most people have heard of data lakes and a growing number of people now have also heard of data hubs too. And I think in general about a third of the people that I talked to are using them or maybe a subset of the technologies. But I'd say at least two thirds are actually either building or considering building a data lake and a hub. So it's starting to grow in popularity. And just a quick background for some of the concepts that I introduced today. They are discussed in much more detail in our book, Making Sense of No SQL. And this is really a guide for solution architects. So we're going to take what's called the solution architect approach today that is given a business problem. What's the best objective way to come up with the right architecture based on your business crimes, not necessarily on your legacy systems or what your IT has done in the last 10 years, but it's really an objective matching of a business problem. So let's review what these patterns are. Relational databases we have in the upper left-hand corner where we store data in what's called a row store. We add one row at a time and the constraints are that all the columns have to be in a consistent structure. For example you might have dates in a certain column, all the things in there have to be dates. The second one is the analytical where we have these star schemas with a large fact table in the center that could be billions of rows long. And then outside of that you have dimensions for dimensional analysis. Then we these are the first two traditional ones and then the four after that are considered the not only SQL patterns. And we're going to briefly talk about them and then talk about how we use various ones for data analytics. Key value store is the most basic and everybody uses a UNIX file system or Hadoop distributed file system. You have a key which is the path to the file and the value. But the value is not indexed. There's no way to search for data that's inside. You can't say select all data where the value is something like that and that's the most straightforward and easy to set up system. We're also going to talk about column family stores, graph stores, and document stores. And so we'll go into a little bit more detail in each of those as we need to. So just to make sure everybody knows a little bit of the trends, I mentioned that data lakes and data hubs are both popular terms and growing in popularity. They've had peaks at various times. But I think both of them have been around for a while and continue to grow with people's awareness. And the biggest thing that we want to understand about these two architectures is what's called horizontal scalability. Horizontal scalability means that as I add more data or more documents or more information about the systems I want to analyze, we simply add new nodes to a shared nothing cluster. And as we add more nodes, we get increased computing power. And this is different than many other of the systems on those six, specifically relational databases have a characteristics where they try to do joins. And there's nothing wrong with the SQL language. It's really the joins between those structures. And what it means is that joins need to move data around. And the more data you have, the harder they are to scale. So we're going to talk a lot about what we call that scalability factor. So let's talk first about the data lake definition. So what do we mean by a data lake? So I'm going to use the definition from tech target, which I think is very small and brief and concise. It's a storage repository that holds a vast amount of raw data in its native format until it is needed. So I'm going to focus on three words. First of all, vast. Vast is of course a relative term. And every year, the definition of what means by vast seems to go up. But in general, I'd say it's usually data lakes have usually more than 10 terabytes of data. Remember, you can buy a 10 terabyte drive now for $300. So that doesn't mean you're spending a lot of money. It just means that you have a lot of data. We also want to focus on the term raw. That is, it's data that's not processed. And it's in its native format. So if you dump it from a relational database into CSV values and you have a lot of numeric codes, those numeric codes are going to be stored untouched, unchanged and untransformed. So let's talk a little bit about some assumptions whenever we talk about a data lake architecture. This is what they have in common. First of all, they all have the ability to scale out, but they're not designed to do joins between systems. They're designed to be consistent for consistent performance. So if I'm reading data, regardless of how much data or writing data, even under a heavy load, you should be able to get a consistent performance over these clusters. We also want high availability. So if one node in a cluster of 100 nodes goes down, the entire system is unavailable. There's replication. By default most of the system have what's called replication factor 3, although that's very easy to change. That means when you store a document, you're storing it in actually three different places. The other thing is that there are no secondary indexes. By indexes, just like we have a back of the book index or we index words in a search engine, data lakes don't, by default, index your data. And they're also very low cost as measured by terabytes per year. So this morning I went and looked at the Amazon S3 pricing and they're running at 23 cents per gigabyte per month. So if you multiply it by 12, that's 27 cents per gigabyte per year or that's $276 per terabyte per year. So that's kind of the baseline pricing. Everybody kind of measures that. If your organization isn't doing charge backs, you may not know what it costs your organization to store a terabyte of data for a year. But it's something that people that are building cost effective data lakes really are concerned about because storing your data in a relational database tends to be several orders of magnitude higher than this. And then a couple things about Amazon S3 as a baseline is that they're designed for five nines availability. You can always read your data at any point in time and 11 nines durability. That is the probability your data will get lost. And I haven't ever talked to anybody that's ever lost data in Amazon S3 because they replicated on solid state drives very frequently. And an interesting thing. Amazon pricing has never gone up. It's only gone down as they use more efficient drives and they've passed those savings on to customers. So how are they implemented? In general you can implement data lake in any technology you want but the most common is probably Hadoop. Hadoop has what's called a distributed file system often abbreviated as HDFS. You can also implement this on any object store. And there's a growing number of these distributed file systems. So you're not really locked into just HDFS. And they are getting more popular and better documentation, easy to set up. And a lot of them actually adhere to the APIs that Amazon S3 has established. And there's a very good Wikipedia page that has a list of these distributed file systems. So that's kind of a quick overview of how we define a data lake. Now let's define what we mean by a data hub. So the definition from Wikipedia is a collection of data from multiple sources organized for distribution, sharing, and subsetting. So this is a very different definition. This doesn't really talk about storing raw data. This talks about making sure that it's useful for data subscribers. That you can subset it often means you want to do it quickly. You want to generate web pages in real time. And this is all about indexing the data, all about caching the results of those queries. But still can we benefit from this distributed architecture. And that's really the goal of the data hub. So it's also very different because we try to look and make data consistent. The homogenization process of building what's called canonical things. And that's not a trivial thing to do, especially if you have a large organization that can't agree politically on the definition of data. But that's really what we're trying to do when we build data hubs. So let's just compare the data flows here. So in the data lake, we take applications and we're going to either get CSV dumps or if you're doing Internet of Things and you have log files of all these sensors, you're going to take all these things and you're going to start to dump the raw unfiltered, unprocessed data in a distributed file system and then it's ready for batch analysis. This is not a real time process. The focus there is analyzing data that is relatively what we call immutable. It doesn't change a lot. It's not ongoing twinkling transactions. Whereas the data hub is a much more ambitious architecture. We have these applications all injecting data into this data hub where we're going to try to make it meaningful, convert it to consistent format, and then provide a series of high availability sub millisecond response to web pages. And that you can also do things like search. You can manage your transactions and you can do analytics all from the same data set. So very different use cases. So let's just take a little bit about how we actually go about and do some of these things. And by the way I should mention data hubs and data lakes can work together. You can standardize your data and then dump it back into the data lake. So these really, it's not really, you have to have just one or another. They can work together and they frequently do in large companies. So let's just talk a little bit about how these data hubs actually do their data. They also take the raw data and they put it in what's called a staging area. Staging area is also very common to anybody building a data warehouse. And the most common pattern there is every time you have a row in a spreadsheet, in a database, or in a log file, you create one document per row in these data hub architectures. And then you transform those into what's called one document per object. So what do we mean by an object? An object is an enterprise entity. It might be a person, an organization, a product, a sales transaction, an event, a claim if you're an insurance. And it's all the data that's a social thing. So if we think of relational databases having lots of one to many and having all the do those joins, we eliminate the need to do joins by collecting all the information for enterprise objects and putting it together. We can then do things like validate the data against standard document schemas and then we can look it up against metadata and reference data. And then we can create these extracts of that on the index data and create output from that. All right, so that's a little bit about what these architectures start to look like. So what I've been doing is specifically focusing on AI driven strategies and if you think of AI driven, we mean a lot of these things that are doing deep learning where we're taking a lot of data but we need to format it in a correct structure. We call this feature scaling and means normalization. It means that instead of having data that has huge ranges every all the data sets might range between maybe for example negative one and one and then everything is scaled on that. We can then take training sets out of that, train our predictive models and bring them back and then we can use those models to do the predictions and build AI applications. So one of the things I always ask people is, so if you have the question, what is the answer to life, the universe and everything, the data lake would answer that 42 just like in the book, Hitchhiker's Guide to the Galaxy. And the problem here is that what does 42 mean and that is the meaning should make you think about semantics. Semantics is the process of assigning meaning to this low level numeric or binary data. So I'm going to propose seven levels of semantics here from stage one where we get a raw, commas value and the column name may not mean anything to us. For example, this one actually means the individual gender code in 3D, but the numeric code once again doesn't mean a lot. So we can change the names and have more human readable names on stage two. We can then associate those attributes with an enterprise object like person. And then in stage four we can go from short numerics to maybe short letters but maybe we actually really want to have searchable strings. So when people type in, find all female physicians within 10 miles of my house, the word female can be used in a search string. Then we can start to associate each of those elements with a formal namespace for your company. And then the last stage is the ultimate in semantics is what we call the RDF model. So let me show you what that RDF model means. It means that for every element in our data hub we have a subject which is a full URI that points to the person in our metadata registry. It has the gender code which is also a pointer called an IRA which is the predicate and then the object. Sometimes objects are literal strings and sometimes they're also pointers. So RDF is the ultimate semantics but it is very verbose and most organizations just don't have the people and time and patients to go to that full RDF system. So what we talk about now is let's talk about the spectrum of semantics. Not everybody is going to be able to move their data to full RDF but what we are starting to say is that data lakes focus on cost. They focus on getting things up and running quickly so that we can have lots of data to run our analytics on. Whereas data hubs have a much more ambitious goal of doing incremental harmonization of that data and then optimizing it for queries and indexing it. So it's a second set of processes around that. All right so one of the things about these data hubs also is they help you build integrated views of your customers or maybe integrated view of all your hardware devices or whatever and the key about that is if you have 10 different ways to represent a person or a claim or something like that and you put it in your data hub it's going to be very difficult query because your query is going to have to have get this data, get this data, get this data and then put in the if then and else rules for all of them. If you really wanted integrated views of your customers we need to start to talk about these data hubs and how they work. A couple things about implementation here. People ask wait a minute we have this wonderful storage area network we've spent millions of dollars on this. Can I build data hubs and data lakes on this internal shared storage area network lands and sands? And the answer is really no. It is these data hubs and data lakes are designed to scale on shared nothing on the far right hand side in the green box. Every single node in your cluster you may have 100 nodes and each of them have their own desk, their own RAM and their own processor. The other architectures where you have shared memory and disk may be good for certain things maybe your sand does automatic compression of your data sets and things like that but what you find is that these data hubs and data lakes need to have their own hardware. They need to be in independent and highly specialized things and especially in data hubs where we have lots and lots of rights to indexes you have to have very fast right throughput using these storage devices that have lots of very fast solid state drives. So a couple things that we talked about the 360 and integration. I just want to really emphasize the fact that many of the studies that show that people try to do these integration tasks on relational systems have huge costs over on some huge delays. IT business edge said they did a survey of people who tried to use relational databases for integration hubs they had 100% failure rate on their survey and the reason is variability. If you start to gather one set of data and it has some small variations you can model it in your ER model but what about the second data set? Well you try to standardize it and they have separate variations so you have to go back and remodel the data and then you get the third data set and you get this constant things where people have to constantly change that enterprise relational model. So the key thing about this is when you talk about variability make sure you know that you can't predict future variability so you should strongly think about an architecture documents and graphs that support that variability. So just to summarize what we've been talking about both data lakes and data hubs have that consistent horizontal scalability. They both have this agility and flexibility. We call that schema agnostic. They don't care what data, what the structure of the schema comes in. They just will absorb that data and store it. Both of them leverage the low cost of shared numbing racks of computers commodity maybe not cheap but certainly commodity so that you can use generic components and they both know how to reliably distribute the data and distribute the computing loads. So if you're doing a transform of your data and you have 100 nodes the CPU load on all of those 100 nodes should be evenly distributed and both of them help organization understand the challenges of distributed computing. So what do I mean by that last data? One of my favorite articles is the Peter Deutsch article called the Eight Fallacies of Distributed Computing. The network is reliable, latency is zero, bandwidth is infinite, network is secure, topology doesn't change. There's one database administrator transportation of data around the network is zero and the network is homogeneous and in general you don't see these things. These things are all challenges that people that have never done distributed clusters and replication between data centers they simply get, they underestimate the transition. So one of the things I'm saying is if you don't really have a business need to do distributed computing try not to do it because it is complicated and expensive but if you are really trying to build a 360 view of the customer then it is a very good thing. So let's just talk a little bit more about the difference in philosophies and let me zero in on how data hubs and philosophies are different. So we're going to say data hubs we ingest everything, we index everything, we analyze everything from those indexes, we run data profiles that is we do statistical analysis of data variability, then we can track data quality and we can incrementally harmonize. Remember several people who set up these things think oh I have to translate every single element of every database into a consistent representation and that's not the case. We use these things called the wrapper patterns, we put the data that's been harmonized in a header and we can put the source data in the body of the document but we don't have to harmonize everything from the get go. We also want to promote strong data governance and stewardship. We also want to make it easy to do transaction search and analytics on these harmonized data. So let me just talk now about the challenges of using both graphs and documents. So these are the last two patterns and I think one of the things that I have found is that if you use graph and documents in a mix of these things they're really the ideal structure for loading data in and specifically to help you get data out faster. We'll talk a little bit about why not having to do joins really does make your performance go up and make it much easier for your users to access this data. So just to introduce some terminology here. If you've never heard of this term enterprise canonical model it's important that you understand that what this is, this is a standard pattern you can go to the enterprise integration patterns book and you can look up canonical data model and what you see is that this is the way that we minimize dependencies when we're integrating applications that use different formats and what we do is we effectively have this canonical data model and then we translate it into that standard structure and if people need different structures on the way out we can transform it. So this is an important pattern that everybody understands it is not necessarily an inexpensive pattern and you need to know when to apply it. So one of the things about this is understanding your cost models and remember if you look at a traditional we call three different component system where you have transactions you have your data warehouse and you have your search through separate computers and separate people managing and separate security models your applications will store their data in the transactional database but then you have to make copies of it every night into these other systems and then you'll run your BI and analytics off your data warehouse and your search indexes. So the cost of these systems you have to add them up it's the cost of storing relational storing the data warehouse data storing the search data as well as the two ETL processes and then I ask the question so where do you store your metadata is it consistent what are your charge backs and do you have consistent security models and if you're dealing with highly audited organizations healthcare data is a very common one HIPAA rule says that anybody that touches any of these databases has to be audited and you have to have logs to do it so it gets to be pretty complicated. The big problem with this is the ETL ETL can become the monster of complexity it is very brittle it's not suited for dealing with unstructured data it doesn't respond easily and quickly when you add new columns and it also doesn't scale it doesn't it's very hard to distribute ETL because ETL often uses joins and those joins suffer from the same problems so it is the really in my experience it has been the biggest linchpin that has made the cost of the traditional data warehouses go through the roof and caused huge amount of people and time and overhead so let's talk about the deal no sequel system so if you have a cluster of computers and each node in that cluster has the ability to handle transactions and search and analytics then you don't have to move data around I work with a CIO that frequently pounds his fist on the table and says every time I move data around it costs me money how do you minimize moving this data around and the data hub architecture is a way to approach those cost savings by scoring it in a single structure that supports all three of these things transactions analytics and search we can do a lot more and I remember I mentioned that example earlier finding all female doctors within 30 miles of your house if you don't convert the numeric codes into those strings you can't do search and you can't come up with canonical models that support all three of those things and that requires some relatively senior people to build those models all right so document stores let's just go into them a little bit ideal when we have any highly variable datasets the reason is every document can grow branches on trees as the variability comes up comes in there's no rule that says you have to store things in certain structures and you can add optional elements at any time and we don't have to have the same problems with joins so moving to document stores has been the principal thing that helps people lower the cost of building data homes they're very different because they handle data in a flexible way one of the most important case studies around is the healthcare.gov site where they tried to do it in almost three years with a relational database they had 37 different states they had 37 variations and different business rules but they could easily do it in a document store because those stored that variability and there's a lot of diverse examples and having a document store that has good security where you can audit personal healthcare information where it has good role-based access and we have that volume scale out and those are the really key things about using document stores so summarize there an operational data hub is really a single system that does three things does transactions it does the search and it does the analytics and the idea is if you can do it in one box then you start to minimize the cost so when people start to go through cost of these things they say wait a minute this data isn't that going to cost a lot of money so one of the things I try to mention is that data storage costs are going down and they continue to go down so I went to ebay this morning and I noted that you can now buy a 10 terabyte higher drive for under $300 so it's not really the cost anymore this is becoming so cheap so why not index everything and why not index the data as it comes in so that you can do analysis as well as index it as after you've transformed it let's talk about this thing called normalization and normalization is when we put the data in specific tables and then in order to get the data out of that we have to do joins document stores store things in what's called the denormalized where they effectively unfold all the joins they do them all and they put it in one document model so instead of doing 16 levels of joins on 17 different tables that's actually a real case study from a data warehouse that I worked on it's one line of code and because the data has all been pre aggregated together about each person those business entities your performance goes up there's no joins you can distribute your documents and it's so much easier for people to write code that accesses this data all right so let's just talk a little bit about data quality and data governance so data quality is really when we can start to apply a schema against valid canonical data now remember I said that data hubs and data warehouses are schema agnostic they don't care about the structure they load but that doesn't mean that we can't come up with business rules and store them in documents that validate documents for consistency what are the required elements what are the option elements what are the data types what's the order and all those things can be stored in what's called an XML schema document and that means that we can validate those documents and then we can assign a numeric score for data quality once every document has a numeric score and then you can do things like only allow data out in search where the data score is above maybe a score of 60 out of 0 to 100 and anything less than 40 we have to go through a data quality process and continue to enrich to make that data better so just to go through this we talked about these different sections just wanted to make sure that that everybody has clear definitions of each of these sections so staging is where we put the raw data canonical is where we start to denormalize and enrich it start to make it semantically clear and then egress is the area that we start to pull the data out and that's where we make sure that we have the right indexes and the people that are accessing it have SLAs that are going to meet they're both their read access times as well as their high availability I should mention that many of the projects I've worked in have huge requirements for reference data I'm not going to spend too much time on that but I want to make sure everybody knows that reference data is one of the most difficult things and many projects once they get underway half your budgets can be sent on getting your reference data up and running and consistent for reporting. So how do we do this we take muddy data in our data lake and we try to clear it up we give it continuous enrichments so this pipeline in the bottom says we're going to validate it we're going to enrich and we're going to run that cycle over and over until we get a score that's reasonable from the EIP book there's a very good section that talks about the enrichment pattern where we take resources and one of the best examples is you want to search by how far people are away from you find all physicians within 10 miles well they may have an address you have to enrich that with your longitude and latitude and index that so you can do distance calculations those are all examples of enrichments. Every time you have numeric codes you can think of converting numeric codes into these high precision things with very clear URIs associated with both the data element names and their values as that continuous enrichment process. So what we then talk about is this data opaqueness in the muddy systems you don't have the definitions it's expensive to understand them there's no data stewardship and change control and once we have a lot of these process in place we start to clear up our data make it easier and more transparent for people to access we have great APIs that are well documented and all of our processes are designed to make things easier. I should just mention that there are a lot of very good tools for doing data profiling that's really essential when you load your data you index it even in staging so that you can run analytics on your data and you can find out what is the variability of the way that people represent a gender code and if it's 10 different ways you know you have a lot of work if it's all just to one of two values then you know your transformation process is going to be much easier. What I want to do is make sure people get out of the trap of doing data archaeology of data in your data lake and I've certainly seen projects where they have numeric codes the numeric codes changed every release of the software they don't have the back versions of that software and they'll literally spend tens of thousands of dollars to find out that they can't figure out what this numeric code meant that can be very very expensive and so what we want to do is make sure that your downstream systems archive any reference data have time stamps if they change those so you can know the time set ranges of the meaning of changes and the semantic drift over time. In general what we find is if you don't have strong meaning the value of data has a half life typically it's 18 months so data that's 18 months old is half of what it is that data that just got ported in because the software that created it is much more up to date. I also want to mention that when we use the word semantics there's two different meanings. There's the uppercase S meaning semantic and the semantic web stack versus lowercase which is having the data governance and registry so it's different things. I'm going to just quickly go through making sure people understand that search is very complicated.  Use an existing search engine because they really do do a lot of the math of how to do good search relevancy and word density ranking. That's not something you want to do on your own. We want to mention that you want to make sure that you understand this concept of coupling between an application and a database and you have to decouple application code from the data that it lives on. The biggest thing I want to mention about this as we wrap up is that when you start to build data hubs you're not going to see a lot of value as you start to spend it up. The value of any network in Metcoss Law says that the more data is in it, the more highly these things are connected, the easier it is to provide value. So data hubs are low in value when they start but as more data is in there and it's structured the value of it it goes up. What's also true is that the incremental cost of adding a new dataset tends to go down because people often know the structures, know the codes, know the reference data, have all these things. So it does take a while to do those things. So as I wrap up I just want to mention that if your job is to be a solution architect, to give an unbiased view of which is the right architecture, it really means that you need to know more than just one pattern. You need to know more than just relational and maybe a little analytical.  If you don't have people that have built these systems in production in your selection process you're risking not providing value to your solution. So I encourage you to go out, reach out to other people that have built these systems. I do have a process I recommend and this is called the ATAM process. It's a standard process out of Carnegie Mellon and what it means is that when you consider your architectural alternatives, that box in the middle that has the picture of the six models, that you consider all of them. And you also in parallel you build what's called a quality attribute system. So what is a quality attribute system? If you think of a tree, a utility tree, that utility tree describes how important a feature is to your customer and maybe just a high, medium, low setting. And then how easy is it to implement that system on a certain architecture, maybe high, medium, low. These utility trees will show you the fitness of an architecture. So if you think of a globe fitting a hand, it's really how well does the architecture fit the business problem. So in summary, what are data lakes and data hubs having common? They both are great at distributed computing. They both have horizontal scalability. They both leveraged low cost storage systems. Make sure you're measuring it in terabytes per year. They don't work well with storage area networks. They like to have their own hard drives that they manage the systems. They don't require upfront data modeling, although you may design the structure of folders in certain data lakes. And they're very different in terms of semantics, indexing, caching and security models. So my recommendations is be very cautious about doing integration in a relational database. Make sure that you understand the role of data variability before you do these projects. Not just variability today, but variability in the distant future. Look at data lakes for lowering the cost of analytics. Use documents and graphs for building integration data hubs. Use data hubs when you care about the search and analytics and eliminating the need for expensive ETL. So with that, I've said pretty much everything I want. And Shannon, would you like to open it up for questions here? Dan, thank you so much for kicking it off in such a great way. What a great presentation. So I will jump in. Just a reminder to answer the most commonly asked questions. I will be sending a follow-up email on Monday for this event with a unique login for everybody to access the slides and the recordings. So let me jump into the Q&A because we don't have much time here for it. Is a data sandbox more akin to a data hub than a data lake? Boy, I would call it a data lake. Sandbox has that very informal nature to it and doesn't really have anything to say about making your data consistent or indexing it. So I'd say it's much more like a data lake. Alrighty. And if you have questions, feel free to smith them in the bottom right hand corner in the Q&A section. So is data modeling not required at all for data lakes and data hubs? I would say in general to get started with a data lake, there's very little modeling. And the only thing is remember that's a key value store and the key has can have a folder and you may design folders. And for example, you may want to put this year's claims in one folder and prior claims in another folder and then be able to narrow your things. So you might design some folders in a data lake. Data hubs do require you to do modeling at the document level. So there is a lot of data modeling process, but sometimes your organization may actually have a canonical model. So you might use that too. So in general, data lakes have relatively little modeling and then data hubs much more document and sometimes graph modeling too. Sure. And what is the difference between traditional staging area in a database warehouse and data lake or data hub more specific to data hub? Yeah, that's a great question. Both of those systems use staging areas and there are separate copy of the data that's kind of blocked off from user access, but there are usually replicated copies of the data that are dumped from the relational system. So I'd say they're very similar. In a data hub though, as we move towards document stores for that flexibility, we use this pattern called the one document per row. So every time you have your relational databases, you're going to convert each row of every table into one document. And you may think that that's going to be millions of millions of documents. That's true. But remember, if you have 100 servers in your cluster and each of them, and if you have a million total documents, one one hundredth of them are going to be stored on each system. So in general, we don't worry about the scalability of lots of small documents in the staging area. So this summary is they have very similar function, but the implementations are slightly different. Sure. So where, how does data hub reduce data anomalies and create, update, and delete since it is denormalized? Let's see. I'm not sure I understand the question. So when you talk about data anomalies, so let me just assume that what there's users saying is you have data that has unusual data elements in it, and that you haven't seen those data elements before. And data hubs do store and index that, and it's always immediately searchable, but it may not validate against your data. So, and it may not be included in some of your reports. So that's what the data profiling tool really does is it detect anomalies and that's what schemas do. If one of the favorite jokes I hear about is data hubs allow us to ingest data that we actually get, not what people said they all sell us, send us, right? So a lot of times people say, well, we're going to give you this extract from our database and here's the fields going to be, but somebody adds a new column to it some week and then you get that column. And you've never seen it before. So your validation can quickly say, hey, I got a new element. I don't understand it. We're going to put the data quality score down to 10. We're going to look at that. We're going to say, oh, that makes sense. We'll add it to our schema. We'll validate it. We'll boost the score up to 70 and then it's searchable. So there's two data profiling for the first set of data, data validation for continuous things that will help us detect anomalies to data sets that are coming in. And I hope I answered that. And if I didn't, send me email and I'll try to answer the other meanings of the question. Love it. And so where integration is heavily needed, what are the general challenges of moving relational data to NoSQL? Great question. It's something that I have been obsessing over for almost two years. How do we get data out of relational and into these data hubs? And in general, there is a very important pattern called the load as is pattern where we just move the data directly into staging area unchanged. It's very important to use that load as is pattern because we can do data profiling analytics in it. So load as is really a five step process of getting data ready to be transformed into standard formats. Here's the problem. Once we do know the transformation, that transformation is somewhat a manual process. But wait, there's this new technology called artificial intelligence and machine learning. And what we're starting to realize is that we can use AI and machine learning to partially automate the conversion of data. Partially because it's a huge complex problem. I've heard of several research projects that have doubled or tripled or maybe sometimes a tenfold increase in the speed of converting relational into other formats by doing machine learning on it. And effectively what we're doing is we're doing a matching. We're matching. Here's person's last name and the relational, here's person last name and in our canonical, we have a vast amount of statistics. What's the probability distribution that the first letter is going to be an A in the last name? We can store those statistics. When we sample our new data, we can put that statistics through machine learning and it can give us a probabilistic estimation is this in fact a person last name? And it can do that based on lexical, what's the column name, semantic, what's the definition that we got for the source as well as statistics which is from the data profiles. So the number one thing is that right now moving data from relational into document stores and data lakes that's meaningful is an expensive and slow process. It takes about 70% of the budget as the data transformation process. And in the next five years I suspect we'll see at least one and maybe two orders of magnitude lower costs in automating data conversion because of machine learning. And that's certainly one of the most interesting topics that I'm interested in. So anybody else that is interested in using machine learning to do data mapping, let me know. I've been working on that for quite a while. Dan, thank you so much. And there's so many great questions coming in, but I'm afraid that's all we have time for in the Q&A. Dan, thank you so much for kicking off the event today with such a great presentation. So thanks to our sponsors and thanks to all the attendees who have joined us so far. We now have a 10-minute break where we encourage you to network with each other as you hear us set up the next speaker. The next session will begin at 12 p.m. eastern where we will hear Donna Burbank discuss the latest in database and metadata relationships. Dan, thank you so much and thanks to all of our attendees so far. Thank you very much, Dan.


A few weeks ago Tyler and I did a podcast about firewalls and how they are important for people if they want to stay safe while using their computers. And one of the things I asked on that podcast was if anybody was interested in me doing a brief tutorial on actually how to use a firewall on Linux. And several people indicated that they were in fact interested in such a video. So you ask and you receive, that's how these things work. So today I'm going to be talking about one of the most simple things you can do on Linux to keep yourself safe. Now the first thing we should talk about is what actually is a firewall because at its heart a firewall is actually fairly simple at least in idea like the concept of a firewall is very simple to understand what it does is it acts as a breaker point between you and the outside world so that any traffic that you are targeted with so things from the outside trying to come in things like trying to get through SSH port or something like that can be stopped if it's not authorized. That's why they call it a firewall. It's meant to be between you and the outside world technologically speaking firewalls are a little bit more advanced than that and a little bit beyond me. I'm sure that the development of such a thing is which were complicated than the actual concept of it. So what we're going to be talking about today is a specific firewall called UFW or the uncomplicated firewall. And what UFW is is what it says on the tent is a firewall that is very, very uncomplicated. There's not anyone out there that couldn't find themselves able to implement this on their system. No matter what system they're using as long as they're using Linux this will work. Now there are two ways of setting this up. You can use something called GUFW which is a GUI front end for UFW or you can use the terminal version of it which is what we're going to do today. Now the reason why I'm not showing the GUFW program is simply because I've never used it. I've seen it used and it looks very simple. It's just I don't do stuff in the GUI. So if that's something you're interested in I highly recommend checking out district tube's video on this. He did a very brief exploration of GUFW at the end of his video so if you want to see that you can do that. I will leave a link to that in the video description. So here is how you use UFW. Let's go ahead and jump in. Okay, so here we are on a very standard version of Arch Linux. This was installed with the Arch Linux GUI installer so there are some things installed by default but for the most part this is just vanilla arch.  Sometimes it's UFW, sometimes it's something different so you may encounter some conflicts if you're on a different distribution. Chances are the distribution that you're using either has UFW installed or it's not installed at all and you can install it from the repo. So in the case of Arch Linux you can open up a terminal and we'll install UFW. So I don't know why the console always opens up like that in this particular version of Arch. It doesn't really matter. The point is that it's easy to install. So sudo pacman-sufw. Now if you are on like I said Ubuntu UFW should already be installed. If it's not you can install it like this apt install UFW. Now once you have installed so I'm going to go ahead and install this here just like this and enter a password. Once it's installed we need to enable the service. So if you are on a system that's running system D this is very easy. If you're using a different in-it system it's going to be different. You'll have to look up that syntax for enabling services in that particular in-it system. So if you're using OpenRC or run-it or something like that you'd have to look up the syntax for enabling services and how to use UFW. The chances are there is a guide out there somewhere on how to do this. But for system D what you want to do is run this command here. So sudo systemctl which is the command we use to enable and deal with all system D services. And then we want to do enable and then we want to do dash dash now flag which basically what this does is it creates a sim link where it needs to create a sim link so that every time you start up the computer this service will start. But we'll also with the dash dash now flag it will start the service now. That way you don't have to run a separate command saying start the service. So what we're going to do is type in UFW if you can type those letters and then dot service enter if you haven't already entered your password it will ask you for your password but if not this is what the output should look like and we can see that it's running if we do sudo systemctl status UFW dot service. And we should see here that it says active and it is so we can hit quick queue to quit that and we now have UFW enabled but still not quite running it. So the next thing we need to do is make sure that UFW is actually running. So we want to do sudo UFW enable. So that just tells you UFW to actually be on this is turning the on switch the service is running but if the service is running but UFW is not enabled it does nothing it's just running the background taking up resources but not actually doing its job it's taking a break or something I don't know. The point is once you've done this enable thing it will tell you firewall is active and enabled on your system startups so you can again check and make sure UFW is actually running correctly this time by doing sudo UFW status and it will tell you this the status is active so let's go ahead and clear this okay so theoretically right now you're done like there's absolutely nothing else you have to do you have a firewall that is now protecting you from traffic from outside of your network and you could get up and walk away here if that's what you want to do but there are a few more things that you probably should know. The next thing you want to do is check and see what options you have for allowing traffic in so you want to have full control over your firewall so that in case you come across some kind of application whether it's you know SSH or torrenting or something like that you want to have control so that you can allow those things to have access to the internet properly have access to the internet I should say in order to see what apps have the ability to be permitted that access you run this command here sudo UFW app list and what that will do is it will show you every application that you can allow to have access through the firewall so things like SSH things like IMAP ports things like torrenting clients so on and so forth because by default UFW blocks everything that comes in okay so anything like SSH and stuff will be blocked by default you have to same thing with Samba even if you're just using a Samba local share your Samba has to have permission to communicate between the two computers and if there's a firewall in between them it's going to block it so you'll have to make sure that you have allowed Samba in order for that to work so you can see through the list of stuff here that there's just stuff that you have option of allowing through your firewall in order to do that it's really easy so let's say we wanted to allow SSH so we do sudo UFW allow and then the name of the thing that we want to allow in this case we want to do capital SSH so SSH hit enter and it will tell you that it's added a rule it will also tell you that it's added a rule for v6 which is basically telling you that it's allowed created a rule for IPv6 which is the internet protocol stuff it's complicated and it's not necessarily all that important but it's creating it for both versions of IP the internet protocol so once you've done that we can actually test and see whether or not that rule was actually created we could do you sudo UFW status numbered and this will tell us that that we in fact do have complete SSH access now technically this is the wrong way of doing SSH because what this will do is it will allow SSH which is what you wanted to do but it will allow bad actors to have complete access to the SSH port without any limitation whatsoever so they could continually try to get into your computer without any limit to number of times they can attempt to do so so what we want to do actually is delete that rule so how would we go about doing that so the reason why we use this command here with the numbered option is because we want to delete both of these so we want to do sudo UFW delete and then one and then it'll ask us to confirm it yes and then if we run this number again we'll see that we only have the second rule and what we want to do then is just delete two one here as well and then hit yes and then we can actually run this team here and we'll see that we have no rules whatsoever so what do you want to do with SSH if you want to limit the amount of attempts a bad actor could potentially use that service to try to get into your computer well there's a way to do that you can do sudo UFW allow excuse me that's not allow it's supposed to be the limit so UFW limit SSH do that and now if we do this command here the sudo UFW status numbered enter again we'll see now that the action here instead of just saying allow in it says limit in that way that there is a certain standard for the amount of time someone can try to get into your computer in a certain amount of time that way if they're trying to get past it like a lot of those things are automated so that if you if you know they're trying to guess your password or something and if they try to do it so many times in like 30 seconds though that IP address will get blocked so that they can't try it again they'd have to try again with a different us you IP address and it makes you safer in that way so let's go ahead and allow one more application so if we do sudo UFW app list again we can see that there is a bit tort client here called transmission so let's just go ahead and allow that so sudo UFW allow transmission you want to make sure you capitalize it appropriately if it will allow you to create rules for things that are not on this list those rules just don't do anything so you want to make sure you actually spell and capitalize things properly so transmission just like that enter and now if we do sudo UFW status numbered we'll see that we have the SSH and transmission so really that's all that needs to be done for UFW in order for you to use it competently you need to know how to install it you need to know how to enable it you need to know how to create rules and find the apps that you can create rules for and you need to know how to delete things that's the five things that you need to know in order to use UFW there is a ton more stuff that you can do with UFW I highly recommend checking out the man page because there's a ton of documentation there that you can use to do more technological and advanced stuff that doesn't really need to be done but can be done if that's something that you want to do so that is UFW if you have questions you can leave those questions in the comment section below I'll try to answer them I'm not an expert at UFW I've never gone past exactly what I've shown you that's what I do every time on my systems and that's pretty much all you ever need to do so if your question goes beyond that I may or may not have an answer but you can still leave those in the comment section below if you have thoughts comment section below you you can follow me on Twitter at Linuxcast you can support me on Patreon at patreon.com slash Linuxcast before I go I'd like to take a moment to thank my current patrons Robert Sid Devon Patrick Fred Kramer make one Jackson I have some tools today based I break it Linux Eric Samuel Mitchell art center carbon data Jeremy Sean Odin Martin Eve Andy Ross Merrick camp Joshua Lee J-Dog Peter a crucible dark band six primes and PM thanks everybody for watching I'll see you next time


Stop! Kill animals! Would you care if someone killed you in a humane, quick way, and ate your body in a sandwich? No, no, of course he would. You wouldn't care? No, of course he would. Why not? Well, I'm dead. You're dead. So why would I care if I'm dead?   Uh, me? No? No, no, no? Oh yeah, I would hate that. Like, ate your body in a sandwich, a burger? Yeah, I would have liked that, yeah. Nah, nah, me neither. If you wouldn't like that, why do you eat other animals? I'm a vegetarian. Oh really? And you? Because they taste good. They taste good? Okay. What if I do humane slaughter? Yeah. Humane, and I get to eat you? Is that okay? No. Why? Tell me why.  You don't want to die? Yeah. You don't want to be eaten. You don't want to be eaten. I don't want to be eaten. What if I killed you really quickly and you didn't feel it? Is that okay to eat you? It's better alternative doing that than obviously putting it through pain. So if we have to eat some food. I'm talking about you. Me.  If it was legal, then you could do it. No, no, no, brother. No, not legal. I feel that's so cool. Only from ethics. Only from ethics. Ethics.  If the death is fast. So if I do what humane, it's okay? No. Okay. So if it's wrong to eat you because you want to live, why isn't it wrong to eat a cow if they also want to live? Because the cows generate, they generate their new cows.  No. Maybe I'll take DNA, make another one of you.  I don't know what I eat. It's basically what the carnival is. So the tooth, especially... Do you think this is a carnival tooth? No, no, no. Do you want to grind meat? Just on hand? No. It's pretty pathetic, bro. Come on.  Have you seen the lion's teeth? Yeah. So why is it wrong to humane-ly kill you? Humane-ly. Why is it wrong? It's not my... I'm not the one who decides whether it's wrong or not. No, no, no. They can both be wrong and they can both be right. I don't have a choice. I just like meat so I eat meat.  Is that humane? To kill me? If you want to live, yeah. Is it humane? You can do whatever you want. Are you religious? No, no, no. Yeah, we have different beliefs, bro. Are you Christian? Yeah. Have you heard this before? Do unto others. Yeah. As you would have them do unto you. You know the golden rule? No. The golden rule? Animal is not a human, bro. No, no, no. But they are an other. An animal is not a human. Bye, bye. As Muslims, we like to slaughter them quick and then it's a quicker thing. It's a better alternative. You sound very, like, English to me.  So you eat animals out of UK farms and slaughterhouses. Yeah. Yeah? And then you buy the meat that says halal. It has a halal label on it. Yeah, obviously there's not so much that we can do with it. Okay, tell me the question.  Is that humane? No, it's not humane. But the soul of an animal and the soul of a human is different. How do you know? You measured the souls, have you? No, but... Is there someone inside of an animal experiencing the world? Suffering, pain, well-being? Yeah, of course. Yeah, yeah, yeah. They are others. Experiencing the world too? Yeah, they experience like a dog. Like, dogs like humane and...  No. Yeah, why? Why is it different? Because a dog is a friend of... This is a friend of a human.  Yeah.  Halal chicken?  Obviously if we find out that they're torturing them instead of that we are not allowed to eat them, isn't it? Why are they serving it to you then? That's not... That's the ethical... Torturing animals is against our religion. I know. It's haram. I know it is. When these chickens are born, they're born into these Frankenstein bodies that grow too fast. Yeah, yeah, yeah. And they take those chickens from the factory farm, some of them die in the farm, and they are suffering from existing. Yeah, yeah, yeah. Then they take them to a halal slaughterhouse. And then they slaughter them. They say a prayer in there, and they slaughter them almost the same way, and then they chuck it in the supermarket with a halal stamp on it. Is that halal? Yeah, obviously that would... I want to become vegan. Would you want to be eaten? No, of course not. What if I killed you quickly? I want to be eaten. I want to die.  Okay, if we tell you... If we tell you I'll be vegan, would that give you peace of mind? No, no, no, no. I want you to... You're trying to convince... I want you to think for yourself. I don't want you to harm the animals. That's all. Because animals are harmed when you go and buy a chicken, and they slaughter the birds, and they replace it when you buy them. I feel like it's fine if there was a certain amount of animals being slaughtered. How many? How many is okay? If there's a dog you don't know... Yeah. Is it okay to kill that dog? No. I can never eat a dog. I can eat not a dog. Why not?  How can we eat a dog? Why can't you? It's a dog. I think it's wrong, but why do you think it's wrong to eat it? It's a dog. Yeah, it's a dog. But why? Tell me, explain. It's created by God, by Jesus. You get me.  Yeah. Didn't you just say that before? Oh, yes. No, no, no, no. It's good. It's good. No, no, no, no. Listen, in the Bible it doesn't say not to eat dogs. No, but what's it called? Does it? It doesn't say not to eat dogs in the Bible. We have different beliefs. Sometimes it's morally wrong in it that they're doing, and obviously there's one thing they're telling us all. Who's telling us? It just depends on what you've seen in it. I've not seen any of this. Of course not. Obviously when we live in our village... Your family's from Pakistan? Yeah. Whenever I go back there, I can see it is calm. It's naturally born in our village. We buy the chicken ourselves.  Okay, I'll catch you. I'll catch you. We can talk about that too because I personally don't think that you need to do that. What's your name, man? Joey. Joey, yeah. What's your YouTube channel? Joey Carpstrop. I love a look at it. I hope you get some good views on YouTube. Bless you. Take care. Take care, take care.  So when an animal has their head open, they're trying to breathe and they're terrified and they're pissing blood out of their neck, it's a horrible, horrible, brutal situation for the animal. I wouldn't want them to die in that way, that's for sure.  Let's keep going. What's the acceptable amount ethically to slaughter? What is the acceptable amount of animals? To be honest, how many are slaughtered? She made that statement. Let me answer. How many is allowed to be slaughtered in your eyes? Okay, how many are slaughtered right now? 70 billion land animals every year and between one and three... I'm turning vegan now. No. This guy is a person, right?  They communicate. They communicate. Yeah, they're not going to talk to me. But they don't want to die, do they? Yeah, but they're going to get eaten. They get scared? They're going to get eaten even way, bro.  No. Why not?   But why are you going to eat me, man? No, I'm just saying... It's a fetish. To eat people? I want to find a reason to eat you. I think it's a fetish. Come on. I'm just saying, like, if you don't want to be eaten, you could probably agree that a cow doesn't want to be killed and eaten. Yeah. They don't want to kill, but I like meat. Is it justified to kill an animal because they taste good to you? Through the animal's eyes? No. It depends on if I'm grieving it. No. Well, I'm just kidding. Yeah, but does the animal... Did you want to eat meat? Does the animal want to be eaten? Carnes are cruelty. Yeah. You know, if you're going to? You're going to? No, no. Why not? A lot of it meat. Why? Why? It's bad. Because carnes are meat.  Yeah, we know. Is this good? This is not good. It's a good one. It's sad. Why? You're not going to stop people from eating meat. Yeah, but that doesn't mean we should do it, just because we can't stop some people from doing it. There's a lot of things we can't stop people from doing it. I guess so. That's a belief. No, but brother, there's people that still get murdered and stuff. But we don't do it. I don't murder people. Exactly, you don't. So just because people still going to eat animals doesn't mean you should. Yeah, but... Because you think it's wrong, yeah? Yeah. You think it is wrong. What's wrong? What's wrong?  If you don't need to, if you don't need to for survival.  I'm going to talk about them. I don't eat them nearly 10 years. Because... Society does. You like the taste? I like the taste, yeah. You might taste nice.  Why do you think that? No, but like, let's just say, if you cooked me up with spices and this and that marinade, I might taste good too. It's not a good enough reason to kill me. How do dairy products, like drinking milk, eggs, just harm animals too? Yeah, so in the dairy industry, they forcibly impregnate the cow because she needs to be pregnant in order to produce milk, yeah? Yeah. It's really perverted, so excuse me for being free.  They masturbate the bull to get it out or they use a... It's like an electronic dildo that forces the bull to ejaculate into a cup.  Yeah? You don't even think that's good by itself? Yeah, no. I'm sorry because the animals kill her. I love animals. But you eat them. You can't love them? We love animals. I smell them. You eat them? But, um, eat them.  You can't love them.  I love animals.  No. This isn't love? No. That's not love. It's not a good idea. It's not love. That's bad. The car, the car, the car is so good. Meat. That doesn't matter for the animal, it's bad. Yeah. I questioned that it was wrong, so I stopped doing it. I wouldn't want to be treated like this. I didn't think it was right to treat animals like this and I didn't have a good enough reason to do it. Justification. Okay. You cannot eat the eggs because you...  But the eggs, they live. When they deliver the eggs, they live. Yeah, but they're laying egg after egg each day and makes them suffer. The boys, the little chicks that are boys, they go in the mincer. They get killed straight away. I don't know. I don't know. I don't know why they hate to eat. It's a good question to ask yourself. I feel like animals were put in this world to eat them. Obviously not in this amount. We're eating an extreme amount. If we cut this by half or even like... Wait a second. Wait a second. There was no rule only eat this amount.  I don't think so. I don't think so. No, no, listen to me. The reproduction process is... that's what's f***ing up.  So they don't get hurt and they take it... So they don't feel the pain for as long as they do. If you electrocute an animal they suffer for longer. The way we do it they suffer for the shortest time. They suffer for a minute. I was non-stun Halas water cows suffer for up to a minute. Yes, but it's the least suffering way for all animals. It's more suffering. The animal doesn't eat animals, but... You're hungry. You can eat vegan food. Vigano. Vigano. I've tried the vegan food. You've tried the vegan food.    You forgot to do this. I know your family's saying one thing but you have to think for yourself. Yeah. The people here have to think for themselves otherwise you listen to others what they've said. Because I was a trade like this in my family. My family they're not vegans. Mine weren't too. They are now. They are now. But they weren't before. Mine are now. They like me. Yeah. So did I. So did but now I know what's behind it. I've got a question this at night. You have to think about it. Let me tell you move it to watch called Dominion. Dominion. Dominion. Dominion. Dominion. Dominion over time. Dominion over time. Dominion. Do you you could hear me. Do you have any thoughts about it? Do you think this would stop it, make a difference? I think that you can make a difference in your life. After hearing you I would like to turn Meghan actually. Be vegan is hello. Are youiscited by You said they don't suffer, you've seen Halas a little? Yeah. You've seen an animal get packed in the neck and bleed out? Does it look like the animal wants out to happen to them? No. If God put them there for us to eat, why don't they want to be killed? If God put us here to live, why don't people want to die? Same thing. We're all gonna die at some point. Yeah, but murder is much different to natural diet. Death. We're all gonna die at some point. Yeah, but it doesn't make killing okay. Correct. Like animals don't willingly walk into a slaughterhouse. They go there in there against their will. And if they were put here by God, they would willingly go into a slaughterhouse and be killed.  They have to deceive them and make sure they don't see each other being killed, which is a Halal standard. Don't show them being killed.  I feel like baby steps could get me to being vegan. If you start convincing people to make their kids vegan, like I'll make my kids vegan. I think, yeah, I mean that's good. If you want to help me. I want you to be vegan first and then you can help me. It's like being addicted to cigarettes. But what about... Except every time you have a cigarette, an animal's head's cut off. Yeah, when you have a cigarette, you call yourself. It's a personal choice. Yeah, but it's the same thing. If everyone decides to make the next generation vegan, they won't... Everyone cares about themselves and they don't care about other people. So let's say for people that don't want to be vegan themselves, try to convince them to make their kids and their grandkids and everyone else vegan.  No, I'll do it. No, no, yeah, I mean, like... Other people that don't want to be... They should educate the next generation. Yeah, yeah, they don't want to be convinced by you, like for their own personal reasons. Ask them to... To educate others to be vegan if they don't want to make the personal shift. Yeah, yeah, yeah. I'm not a vegan. Okay. We'll be vegan for you. For the animals. Thank you. For the animals. I'm glad you're thinking.


Well, it's been a while since we've checked in on our good friend Dave Rubin and He doesn't seem to be doing that well So he is doing comedy again and some of you might be wondering again when was he ever doing comedy? Apparently Dave Rubin is a comedian. I'm not sure if he still identifies as a comedian I would probably push back against that claim, but he used to do stand-up and the old clips of his stand-up We've talked about it before We're not good. They were pretty cringe worthy, but he's doing comedy again And on his program for whatever reason he decided to do a parody of the view in full drag and just a forewarning This is one of the most cringe worthy things I've ever watched in my entire life and I get that the word cringe is overused But this is quintessential cringe. It's painful to watch So we'll watch his parody and then we'll talk about what he is trying To parody in the first place take a look.  What are you talking about? You want to arrest people who are different than you politically Tulsi Gabbard? She's a member of our military. She's a former congresswoman a Democrat You just don't like her so you want to arrest her you want to arrest Tucker Carlson the very dapper Tucker Carlson What are you saying? You you guys are nuts. You're you're just nuts I only agreed to do this show because I thought that whoopee whoopee was sane. She was good Sister act sister act to Anna's just nuts. I don't even know what she's doing here. The other ones are racist to the there's the drunk one I what is what are we all doing here?  It's so painful it literally sent shivers down my spine. It's so bad. Oh Dave. What were you thinking? Oh my god. Oh It hurts the level of secondhand embarrassment that I feel. Oh It honestly is almost Nauseating. Oh Dave Dave Dave stop. Oh my god. I I don't even know how to react to that It kind of threw me off a little bit. Oh Just Painfully cringe-worthy.  Is this really the person who I want to be? now The ironic thing about that clip is he's kind of making a point that isn't insane. He's just making Probably a more reasonable point comparatively speaking to what he usually says But he's doing it in the most cringe-worthy way imaginable so he's talking about the view discussing Tucker Carlson and and Tulsi Gabbard and how they may be Russian assets and perhaps we should have the DOJ investigate them Not necessarily something that I think that we should be advocating for openly as American citizens I get that during times of war people are up in arms, you know, they're emotional But we have to uphold civil liberties. We can't resort to some new wave of McCarthyism We have to respect the First Amendment and as Justin Barragona of the Daily Beast put it in response to the view Saying that we should investigate Carlson and Gabbard. We are really drifting into scary and dangerous territory here And I agree with him here. He is no fan of Tulsi Gabbard or Tucker Carlson. Neither am I but to Suggest that we should investigate Americans for Saying what they said I think that absolutely is a bridge too far We have the First Amendment in this country for a reason and as much as you Dislike what Tucker Carlson and Tulsi Gabbard says here that doesn't mean that legally they should be prosecuted or penalized or investigated We don't need McCarthy witch hunts In 2022 absolutely a horrible idea. So I understand the reason for Dave Rubin to Kind of have this visceral response to what the view said here But he didn't in the most idiotic way imaginable He actually could have made a cogent point had he chosen to to respond like an adult But instead he did a parody where he dressed in drag and tried to do comedy and it just it didn't work out very well for him But let's explore what the view said here because I do think this is an important conversation So Tucker Carlson and Tulsi Gabbard they have been basically parroting Disinformation from the Kremlin. They're using pro Putin talking points. Now, I'm not going to ascribe motive to them I'm just going to tell you that they're parroting what the Kremlin wants them to parrot now as Justin Barragona explains in recent days Both Gabbard and Carlson have been proponents of the theory that the United States is developing dangerous bio weapons in Ukraine a Centerpiece of a Kremlin disinformation campaign to justify the invasion of its neighbor and potentially blame a Russian chemical weapons attack on a false flag Operation clips of the ex-lawmaker on Carlson show discussing bio weapons in Ukraine have been showcased across Russian state media alongside the Fox News Host saying Russian disinformation is true Furthermore, it was reported this weekend that Moscow has pushed the state-controlled press to promote Carlson as much as possible Because he sharply criticizes the actions of the United States and NATO now the bio weapons claim is Russian state disinformation as the article explains there and it has been thoroughly debunked and either wittingly or unwittingly I Mean Tulsi Gabbard and Tucker Carlson are spreading talking points from the Kremlin now That's bad and I think that they should be condemned for doing that because they're aiding and abetting Putin here at least rhetorically and that's Disgusting and egregious but Having said that though what the view recommends here is also Disgusting and egregious two wrongs don't make a right and then them saying something is egregious But saying that they should be penalized legally for saying egregious things like that is also pretty gross So take a look at what the view said for me the question is what is in it for someone like Tucker Carlson, right? I mean is it money who's paying him that money? I'm not making any allegations, but it's just it doesn't make sense that he would Do this kind of thing. I mean he also says that the United States helped encourage the Russian invasion He said the United States engineered a coup in Ukraine in the name of democracy He as we just mentioned confirmed these Russian claims about bio weapons And he's characterizing this situation as a border dispute, right? And so what is in it for him? I know that he's visited Moscow and other places and you can tell me in my ear if I'm wrong about that But I just don't understand what the impetus is. I hope he's planning to move to Moscow Well, he's not gonna be welcome here for much longer. I think his viewers like it At all and look, I but I think that's an incredibly relevant question And I think DOJ in the same way that it is Setting up a task force to investigate oligarchs should look into people who are Russian Propagandists and shilling for Putin. That's being if you are a foreign asset To a dictator it should be investigated in fact I remember when Tulsi Gabbard and I even hate that we're discussing it because I think to myself Who is this one? She's a you know, she's no longer in Congress. She's a failed presidential candidate Yeah, she only practically exists on Twitter and the fact that we're giving her oxygen is what makes her relevant That we're talking about her on hot topics But on the other hand, how do you not call out something that is repeating falls Russian propaganda That has been brought down. They used to arrest people for doing stuff like this if they thought you were Colluding with a Russian agent if they thought you were putting out information or taking information and handing over to Russia They used to actually investigate stuff like this and I guess now, you know There seems to be no bars and people are not being told to hate Putin Putin doesn't need a reason to be hated. It's pretty much clear So I'm gonna say it again You can condemn Gabbard and Carlson as I do because these are terrible people with opportunistic nefarious motives But I don't think that we should just openly Support this idea of prosecuting people because of things that they say I think that even if they say things that are incredibly egregious Calling on the DOJ to investigate them on national television saying we used to arrest people for saying things like this as if that's a good thing That's that's not a good thing again during times of war I get that emotions are high and it's easy to you know, think wow these people are pushing This this propaganda that's so harmful. So maybe we should take away civil liberties I get people have the instinct to do that But you have to fight against that instinct because during times of war It's kind of like a stress test on our civil liberties and we have to uphold civil civil liberties and the First Amendment So I don't agree with them. I think that that's that's awful But I also don't agree with Tucker Carlson and Tulsi Gabbard what they're doing here for whatever reason is is sickening But they shouldn't be prosecuted for saying it again We don't need to turn this situation into a new wave of McCarthyist witch hunts I think that that would be horrible because we saw the way that it turned out Before during the Cold War and we don't want to reignite that hysteria, right? We can condemn people like Tucker Carlson and Tulsi Gabbard without calling on them to be penalized legally I think that that's the responsible thing to do, especially if you have a really large platform like the ladies on the do on the view Have now in terms of what the motives are of Tulsi Gabbard and Tucker Carlson I think that when it comes to Tulsi Gabbard, she just parrots what she hears from people who are popular like Tucker Carlson He's saying this so maybe she's saying it in response I don't know who said it first really to be honest So I think that she just kind of parrots what she hears in her circles Tucker Carlson I have no idea, but I think that the theory that is most persuasive to me is that like a lot of white nationalist far-right Americans he subscribes to the philosophy of Alexander Dugan who is a Russian philosopher who is a Theocratic fascist and a lot of far-right people in America like Steve Bannon, for example Have taken a liking to him and they follow Dugan and they subscribe to Duganism and the young Turks that a really great video Explaining this so I won't get into that here But Tucker Carlson has quoted Dugan on his show before not necessarily In a positive light. He tried to make it seem more neutral, right? This is something that the Russian military reads and it's better than what we read here in the United States And what our military reads so that's my theory I think that perhaps, you know a Tucker Carlson is more sympathetic to the Russian war of aggression because Perhaps he subscribes to Duganism. I'm not really sure but what he's saying is absolutely disgusting and You know like it or not. He's shilling for the Kremlin. He may not want to admit that but unwittingly or wittingly He's doing the bidding of the Russian state disinformation Machine and that's that's pretty gross and it needs to be condemned, but at the same time I don't think that he should be arrested for that if we start being overly skeptical of people Because we fear that they may be shilling for Russia or on the payroll of Russia That's when you also get into a different sort of dangerous territory, right? I mean with 9-11 Civil liberties were destroyed right George W. Bush signed the Patriot Act into law And we never got those civil liberties back that we lost the American people were scared And they thought that the government needed more authority and we now know that the government used their power that they got from the Patriot Act To do warrantless surveillance on American citizens So we have to be really cautious even if you're scared you shouldn't call for people To be investigated because you are suspicious of them because they say something bad You can condemn them for saying bad things without calling for legal prosecution It's really important that we uphold the First Amendment and you know That's kind of I think the point that Dave Rubin was making here But Dave Rubin gets no credit because Dave Rubin is a hypocrite He likes to pretend as if he supports freedom of speech, but when it comes to BDS, he's silent when it comes to Instances where left-wingers are being censored unnecessarily so he's silent So he is a hypocrite Having said that though to be fair to him the point that he's making here or trying to make I think is At least more valid than other points that he's made but for some reason he chose To do a parody and make a fool of himself rather than just being an adult and making a valid point ultimately the parody Dave It's bad and you should delete it from the internet if possible. I think that once that's out there You're not really gonna put the cat back in the bag. I mean I'm re-uploading it So I'm not gonna take that down, but just learn from this experience Don't do comedy Dave. You're not a comedian.  You're not a comedian. It's cringe worthy It's damaging to yourself more so than You've already done it just I'm thinking about the parodies honestly really just making me disassociate I I wish I didn't see it and you're probably gonna be mad at me for showing it to you viewers But I'm I'm so so sorry. I had to share it with you, but um, yeah Let's not start a new wave of McCarthyism.  I'm kidding of course, but that was bad.


Hello Team 5. Thank you very much for sending me your draft plan. This is some feedback. I'm just going to pick up on a few of the points and what I'd like you to do is obviously pay attention and discuss the points in this screencast, but really what you'll need to do after this is to take your copy of the Word document which I have commented and work through those comments in a bit more detail, basically clearing the pink boxes as you go and adding, in your case, quite a lot more detail to your plan. What I'm going to do in the screencast is pick up on just a few of the points. Okay so let's start with the podcast channel and your ideas for that. Looks like you've spent a fair amount of time planning out roles in the team so that's really good. That's exactly what you need to do. Start to get a sense of strengths, weaknesses and who wants to input in what way to your project. So it looks like you've had that discussion. Quite a big team. Okay let's move on to the podcast title because if I talk about this it will give you a sense of the approach that you should be taking at the moment. What was needed here is for you to come up with a title for your podcast channel. What you've given is some sort of sense of what it might be like. Now this is a really good starting point for you. It really helps you as a team to focus in on your planning and design. So the idea here is to come up with a title and create a sense of what it is you're going to be producing together. If you can commit to a good title as a team everything else will start to follow and fall into place. So this should have been the good fun starting point for you. So what we want here is an actual title for the podcast channel. Okay let's move on. You've given me some sense of a structure here. It is a bit thin at the moment. Okay we're talking about a two to five minute podcast so it's not too far off perhaps. One of the things here that I'd like to pick up on is this word informal and you're not alone by any means in using this word to describe the style of your podcast. But as I've said in the comments if you take a closer look at this caution here beware. Informal isn't really telling you anything. It's not telling you anything as a team about what you're going to need. It's quite good to have an agreement about the attitude that you'll be conveying in this podcast. But don't be misled into thinking that taking an informal approach will be easy and that it will result in an engaging podcast for your audience. It's going to be quite tricky to pull it off if you really are going to achieve an informal feeling podcast. It's likely if you're not careful to feel really quite amateur. So just have a bit more of a think about what that means and essentially you've really got a plan whatever you're doing whether you're going for the informal or not you've really got to plan your podcast. Okay let's zoom in here on production time. You've not actually put anything here but possibly you haven't understood really what what you're meant to do. But let's be clear about this. You need to be clear about what each of you in your team are going to contribute to this project. So you need to have that very early conversation right now about when are you each available? When is a good time for you to work together? When you're not together? When can you work on this project? And for how long? Have that conversation now as the team and then you'll be clear about the level of input each of you are going to make and what's expected of you until you start to deal with those questions as a team there'll be uncertainty about things like who's doing the most and who's taking the lead in what area of the project and that sort of thing. So please do have that conversation as soon as possible. That is something that is really very important for you to be clear about at this stage of it. Okay let's move on through here. I'm not going to spend any time talking about this but I'll flag it up. You need to do some work on defining your audience. If you don't know who your audience is you've really got no focus to work towards. You've got to conjure up really quite a real feeling about who your audience is. Right we've talked about channel planning. Let's talk about the episode planning now. Again we want a title because as I said before if you can come up with the title other things are going to follow. It's going to inform the style of the approach. It's going to signal to you as a production team as much as to your audience eventually what sort of approach what's going to come next. So come up with a title. It should be really quite straightforward to do that and it's certainly a good starting point when you're designing this sort of project. Okay so there's more comments there on style and approach and defining your topic. Now this area is also really quite important for you right at the beginning of your planning process. You need to have a brainstorm. You need to look at all the tasks that are going to be involved. Think back to the seminars. You know what is actually involved in this production process and what you need to do is come up with your ideas and convert those. Identify the tasks within your ideas and assign looking at the roles you've identified within your team. Assign who is going to take responsibility for leading each task. Not necessarily doing but leading each task. You need to have someone pushing away at each task to ensure that you get through to the end. There'll be a list here of at least 15 different tasks. That's what you should aim for when you have that discussion. What does Andrew mean by 15 tasks? Well think back to the seminars and that should become quite clear. Okay let's move on through here. File management. Let's just pick up on this because if you get this wrong you can have done a brilliant job the whole way through. Get to the end. Someone's ill. I don't know. Something goes wrong and you've got no sense of where all the files are. How the file naming conventions worked. Just agree that now. That's another easy thing to do right at the beginning. How are you going to manage the files? How are you going to name them? Where are you going to create a backup and you know there should be two or three people in your team that are very familiar with the way the files associated with the planning and the production itself. How are they being managed? Because you can do a brilliant job and it can all fall apart at the last minute because you've lost track of which file is which. Okay I'm going to leave the screencast feedback there. Please come and talk to me if you want any more thoughts about what you should be doing at the moment. But essentially what you need to do now is turn to each other. Now you've viewed this screencast together. Turn to each other. Take a look at the word document itself that I've sent back to you too and work through each of these pink boxes and basically finish that plan off. Now I think you're doing okay. There's quite a lot for you to do there but you have got quite a lot to do really in the next week to fine tune your plan and actually in some cases that there are big gaps in your thinking so far. So keep going and compared to other groups I think you're about there. You know most people have got quite a lot to do at this stage so you know don't worry too much but get it done now. Okay thanks.