Yeah, thank you. Good morning, and welcome to my talk streaming. Why should I care? My name is kristin Drebing I'm working at a blue yonder a German company doing Retail solutions predicting what they need what they should order. Yeah, so that's the background of Me and now let's start with a talk What will expect you in the in this task in this task at first I will give you a motivation on why it makes sense to look at streaming at all and yeah That's also the title of my talk and I will give you some background why I came to the question at all I want to give some introduction into What is streaming? What is the basics of that and then show you how you can use this within Python? Finally showing some challenges that might await you when you go down this road or maybe swim down that stream So that you are prepared to tackle them Okay, let's have a look at the situation We faced as a team for quite some time. We have a data processing application as I already said We are doing machine learning. We get lots of data from our customers and We had a quite monolithic application So what we have is we get input data from the customer that send it in large XML files We need to validate that data. We need to make sure that this is fine that this has to write quality to go into the machine learning We do the machine learning that takes lots of input data right start writes out lots of output data And then we have this output interface where the customer can query the data The results of our machine learning So this works quite fine. I mean The results are quite are really good But we have some we have some issues with operations and worth extending with that So yeah, we have this big database in between where we store all that data where all these boxes are accessing that We're using SQL alchemy to access it from Python We're using a limb big for database upgrades and all this works fine But if you have also an application of that type you might know some of the pains that come with that So for example We are several teams developing this application and now there are dependencies We all depend on this one database and the customer desperately wants a new feature in machine learning Also, the machine learning team says, oh, that's great. It's no big deal We just have to change our query a little bit We have to write to one more field to the output tables and then that's fine Then we are that's no big deal But the other team is also working on that and it's also working on the database and they are chain doing some refactoring on them They are changing that data validation All is very good that they do that but those conflicts and they say, well, yeah It just takes two weeks time just then we're finished and we need to see that we have no conflicts in that So really, yeah, now you as a machine learning team say well now we have to wait That's bad, but that's nothing we can do about This is one thing one challenge that you might face other things are for testing So it's always hard if you have one big application to establish clear boundaries to have Solid testing strategy there you try really hard, but well, this is at least not an architecture that supports you in that Yeah, I also I like the testing in layers talk from Monday and where you see all these nice little boxes And how you can how you can separate that into layers and for sure you can do that with such a monolithic application But it's just hard to do so Then I went to Europe Python already two years ago in 2015 I heard many talks about microservices and I really like that it was a great idea it solves these issues that That you split up the monolithic application into several boxes and you can handle that boxes separately Each has its own data each has its own upgrades the teams can develop independently. Wow, that's great And we had lots of discussions afterwards After these talks on how we can use that for us But Well as I said we have a data processing application and for data processing we have to process lots of data and That's where we saw that we cannot use this model in this purest function as We would have to to transfer lots of data between the services We have to save lots of data in each of the service as each service needs many of that data so Unfortunately as nice as this look as it looked but that was nothing for us Yeah, there we stood Microservices sound great, but I mean we had lots of case studies. They were used in billing They were used for more transactional applications and this worked fine, but as I said we could not use that and Then at other conferences so many people were talking about streaming and also here at at this Europe Python already heard many talks and it's great. It's great to see but Most applications used that in financial services For stock markets where they have lots of data volume where they have to process that really fast or they are using it for online Advertising where they have to process click streams or locks and doing that really fast and Well, I thought these are the traditional domains of stream processing and we are not in there also So what possibilities do we have? We're just standing in the midst Well, we can't do microservices and streaming seems to be the domain of other people Sounds interesting, but yeah, maybe nothing for us Fortunately during a different project I came into contact and do did some evaluations for streaming there and Suddenly it came to my mind that that this might be fine for us Also, we don't have millisecond where click streams to process many things that well might be more side effects of streaming and not just This this pure millisecond processing are really really good And we can use that to solve the issues we have and that's the reason for me for giving this talk Maybe if you are in a similar situation, I hope that this will help you also and give you some ideas on how you can improve your application so let's do introduction to streaming and Give you some basic idea of what this is my background as I said is coming from a database centric application And therefore I want to compare this and give you some idea. Where are the differences? So let's look at databases and streams. I Heard the talk from Martin Kleppmann, which he gave at the strange loop conference in 2014 Turning the database inside out and I thought this was a very good way of thinking about databases and streams Because essentially in a database and in a stream you have the same information When you have a database then this database has internally a change log And this change log tells what did change within the data your database tables and from this change log This is essentially stream and you can reconstruct the database content at every point in time So when you when you look at it here in the first entry It says well change a row with a key a to the value one and then you see this Table has one row with key a and value one then as next Another information comes in change row B to the value of five and you see it in the table Then comes change row C to the value of three and you see that in the table and then it gets more interesting You have updates for existing tables So a gets the value of eight then a gets the value of four and C gets the value of two and each at each point in time You have a consistent table in your database So that's the basic idea of well the It's a similar energy between tables and streams and databases used it internally for replicating to different nodes for example But why does this matter for us? The most interesting thing from in my situation was that different services We have can be in different states So we don't have this dependency to one single state because each stream processor each service Can have a different offset within that stream? Let's take this example here. We have to service one which now is on index on offset three and he sees the table as a Has is eight B is five and C is three We have a different service that is already at the offset five and their a is four B is five and C is two And this is totally fine both Services are in a consistent state and if service one that catches up to the offset of five Then you will have the same information as service to So the interesting thing is you can have several services that can operate at different speeds And this is one thing we have for our application at one point in time We might get lots of new data from the customer We might write lots of data into that stream and we have services that process that faster We have services that process that slower, but as long as they are able to catch up. That's totally fine and Yeah, you can add new services to that structure which can process the stream from the beginning on and You can you have just more control you have more possibilities How to scale your services so you don't have That all services need to be at the same speed, but you can Design by your needs to have services that need to get faster Do I have services where it's fine that they are slower? For example, just Aggregating some reporting mechanisms whereas on the other hand you want to answer the customer fast Okay, so we have that possibility. That's great, but how can we use it? I mean I said some services might be faster some services might be slower How can we influence the speed of our services? Well, we can program better. That's fine. So we can we can just improve our code But the success just a limited effect. Sometimes we need to have even more and There comes one idea into play. That's Also very helpful in streaming. You can partition your streams as means and and that's always a decision based on your Your business domain on what partitioning makes sense for us. For example, we get sales streams We get the information about in what location did we have which sale and we can use this for partitioning our stream So let's say we have here our sales stream We have three locations. We have Rimini where we are now with the bow the last year of Python conference And we have cards rule where our company is based So we sell spaghetti we still ravioli we sell pizza and we have this on different times and in different quantities And now we can have one processor that works on all these three partitions That's fine And that's maybe the best place to start but as we see we need to get faster We have the possibility to split this up to Introduce one or two new processors and they can work in parallel on these different stream partitions So each partition could be handled by a different processor and the state stream example will follow us throughout the rest of this presentation How does it look like so we don't have the database centric We don't have the microservices, but we have here the streaming platform now in the midst And the idea is that we can be at different offsets in that So what did we gain? We have Clearer boundaries for our services. We have all these ideas that we can deploy them independently independently that we can operate them independently and The data is mostly in the streaming platform Well, you might ask yourself Why do we have this data bubbles still in the processors? I'll answer that later and What is about these database schema changes? That I talked at the beginning when the data validation team needs to update it and also the the machine learning team needs to update that Also this save this question. It will be answered later So what did we gain as I said in independent development upgrade and we have more options for scalability Did we throw out the databases completely? We have the streaming platform in between. Well, think of these data bubbles and we'll come to that later For me one important question was this all sounds so great This is such a good idea and it seems to solve so many problems Sounds like magic and magic. That's always something that makes me suspicious. Maybe there's something I don't see maybe there are new problems that arise which I just don't know of at the moment and Yeah, it is not magic. It is a trade-off. I mean a database is very powerful You have many guarantees that are given you by the database You have the the asset consistency guarantees guarantees in your transactions You have the sequel language, which is the so powerful way to retrieve the data that is there in the database You can do nearly everything, but as we have seen this comes at a price We are depending on one single state and also the scaling is hard also scaling of the database is hard So We have to think do we need all these things that the database gives us in our application and What are things that we lose? So with streaming we don't have the asset guarantees anymore We have an ordering on a stream partition. So what the streaming platform will give us It will guarantee us when we feed entries into the stream each service that will retrieve these entries will retrieve them In the same order This might be a small thing compared to the asset guarantees But it's fascinating what you can construct from that and when you have several things on several streams Just given these ordering constrained you are able to construct many of the guarantees that you need and We don't have the sequel queries anymore So it is not possible to query a stream And this is something that you really have to get in your head when you are thinking about that You might be used to sequel you might be used that you can query for any Row with any while you are doing a join on that But this is not possible the stream just goes through your processor And it's your possibility to keep that state and to remember what was the last value of a and You have to decide whether you can live with that or not and what What mechanisms you employ to help with that I Mean you see what you lose, but at least I feel better now I know it's not a magic that might come back to me in the worst moment at all But it's a it's a conscious decision that you can do and you can see the trade-off on whether that's good or bad for you Okay, so much to the theory now We are using Python at our company really really love to use Python in real most of our services and How can we do that in Python? Just taking a step back Apache Kafka as a streaming platform. This is not Python. This is implemented in Java But we'll come back to Python soon. This is This is what we on what? This is an example of the streaming platform So you can have here produce us that put data into the streaming platform You can have consumers that retrieve data from the streaming platform and you have the stream processors that what they take that data and they Wrangle that data and they might they put it back to different streams then And you can also have some connections to the databases to get that data from that directly So that's a really cool Streaming platform. It's used by many people. It's very scalable and it's really better proof. So that's thing that you can build on and There are also Kafka clients in Python We also I heard just yesterday also from others that way that are using them and You have pie Kafka Python Kafka and the confluent Kafka client Which are the three I have seen and also other people already have done very nice comparisons On that for example here the attribution games You can have a look there in detail if you want the interesting The interesting differences between them are pie Kafka and Python Kafka Kafka are both depending written completely in Python Hi Kafka has a very Pythonic interface whereas Python Kafka more simulates to see interface And we have the confluent Kafka client This is not pure Python, but this is using the C library lip RG Kafka, but it's the most performant of these So we decide to use the confluent Kafka client There's many configuration options and it's really worth looking at them because this might You can use them for performance tuning So at first I was a little bit surprised when I used that client and it seemed also slow But it was just to my testing set up where I had very little records and the standard settings are not for these little records But for more it has more buffering in there And if you reduce that you can come to very low latency also in your test setup, which then feels much better So let's see how can we use these clients? First thing is to have a producer. I just have to give that producer with the bootstrap server Which is my Kafka node, which is default on port 1992 I have some data you can see here the sales data and Rimini on On Monday we sold some ravioli and we wanted to input that to a stream called sales input and we are using JSON to serialize that data Then we want to consume that and also the consumer it needs to know the Kafka node to connect to Well, it has some further settings of which here the topic config might be most interesting For you because topic config tells you do I just use the new values appearing on that stream Or do I want to process the stream from the beginning on so default is that you just look at new values But here especially in testing is always very helpful to start at the beginning We subscribe our consumer to the same topic to say its input and The most important things are here. We are pulling the consumer. We're checking that this worked fine that we have something in there And then we just print the received message and it will show up as the JSON string we put in there That's great This already works as we needed to now we use JSON as a serialization format That's also good for a starting point, but as you are working with many teams It's always good to have some defined schema and your databases the schema was defined by the database and this is good So that everyone knows what is in there and also for your for your streaming applications You can use more rigid schemas than just put on JSON What we decided for is for Apache Avro and This is a schema where you define this is something where you find the schema at first You give some data types or give some fields. It has many many possibilities to define that schema and It's also a very good compacted So it's not just writing the JSON format, but it has a compaction in it Which is also very good if you want to save some space But what excited me most about Apache Avro? It does it's that it defines also schema evolution this means You can enhance your schema you can add new fields and It it defines the criteria on how you can enhance that schema So for example for new fields you always have to give a default value Because this ensures that also processors that are at an older state can use that data And if they want to retrieve data that was written by an older service Then the default while you are supplied and if they are reading the data that was written by a newer version of the service then it already has this field with With what the service has put in there. So this is really great and this Solves one issue which I promised to answer you ten minutes ago This is when different teams wanting to want in to enhance the schema So by that they can use in a compatible way different versions of records in there And you don't have to reprocess all entries in the in the stream as you would have done with a database upgrade script So how does that look like in Python we have here the schema defined it has a name It says it's a record of many fields and I say what is the What are the types of that field we have strings we have quantities there are many other types also And as I said you can you can say whether it is optional you can say whether field has a default value or not So by using this and now we can use a different producer and consumer We use the afro producer and the afro consumer and you can see here Still we have to give the Kafka now now we have to give the schema registry URL So this is something what is schema is registered where every service can retrieve all versions of that and this schema registry This also checks that you only do compatible schema upgrades Which is great because as soon as you want to write a schema in a new version and it's not compatible It raises an exception and heads of work. This is wrong. So, you know it from an early point on The other things are mainly the same. We are giving default Schemas for a key and a value we can use our data here and we now don't encode it in Jason But we give it directly to the afro producer He uses the afro schema to encode that and writes it to the stream Also for a consumer the main new thing is that we have give that schema registry URL so that he does know how to interpret things that are on that stream and We are using here also polling and we just can check on what is the value on here So these are mainly the examples used from the confluent Kafka client So you can also have a look there to dig in deeper and to have some more explanations on that Okay, this these are the basics how we can write and read from from a stream What do we do with that so let's have a look at data validation as you remember That's the second box we had in there and what do we want to do there? We have to say its input and we need to check whether that's correct or not So we're separating the wallet and invalid sales records in the same way as during the Cinderella fairy tale She has to separate the good piece and the bad piece and this these are what we want to do here within our service So very basically we just pulled in your records Let's say we have a function which checks whether the sales record is wallet Whether the location is a wallet one whether the quantity is non-negative for example all these things you can think of and If it's wallet you write it to a new stream say it's validated And if not you can write it to a new stream say it's error and then read the other processors handle this information For example, how to answer the customer that he sent an invalid sales record That's fine a Very interesting thing about streaming is that you can add additional processors in very easy way so let's say either we want some new stuff we We want to have some more in monitoring some reporting on that and we write this monitoring or reporting to a new stream or We have a new validation logic, which we just want to try We don't want to put it directly into production But we want to write its results out for a different topic to check whether this work fine When we had the database centric application we would have to remember the processing states So we have one state and for every record in our sales table We would have known is it validated or not hasn't been processed or not so a second service for validation would be really hard because it has to It has to know well it this has been validated by that service, but I have not validated I need to introduce new field or so But this is not the case here each service can know how far it has processed So that you can work independently Which is a thing that might might not sound that interesting, but as soon as you have tried it. It really is Fun to work with that because it makes things really much easier Especially when you do a lot and you want to try out new things So these are the basics for using the streaming Now let's come to the challenges I said we do machine learning and machine learning Especially in the training. It's not a thing that you do in streaming and the the results The real machine learning answers where they might be different But for us as we do forecasts For sales in a daily way. This is also something we do in batch So how can we work with these batch-like processes within our streaming? Our machine learning application needs to get all data based for the training in one batch Or maybe in several they are working in a partitioned way But still there's many data in that so how do we get that input data and remember you don't you can't query that stream You can't tell give me for all keys the current value So somewhere we need to save that data with Several options. We can just keep it in the memory of our service. We can use a separate database So serving database that is Doesn't need to be that powerful as the database from our monolith But still we could use a smaller database or we could use a blob store which is Also just cheaper than a database and we can save the values in there and it can be used by the by all machine in an application And yes, that is data duplication It feels bad at the moment, especially if you come from a very normalized database scheme But that's the price we'll have to pay we have several advantages and we also then have to live with that data duplication But what's the idea behind such a duplication? How can you explain that and what I found very helpful in there was to differentiate between a right path and a read path So for reasoning about does it make sense or not? I think that's a very helpful distinction and In our old way with a database we had a relatively short right path so we put the validated data in the database and Then the data validation is finished and when we start our machine learning We are doing a machine learning query that needs to fat all the fetch all this data with a very big joint statement And then feed it into the machine learning so this is to read path and this Machine learning query is something that really puts the database under much pressure And you have to see your database sizing that it can work with such a big query and relatively short time frame Now let's compare this to how we would do it with streaming. You see the right path now is longer Because we have to data validation. We write that to our topics and as soon as we have new data on that topic We can all we we can already do the joining. So We are joining the new sales tower with sales data with the location data with the product information And maybe further data and write that to a blob store and there it sits and it waits until the machine learning started What we have lost is our normalize schema because Because now we have to data duplicate it But what we have gained is a very big operational advantage Because that right path can be more scaled As soon as the data arise we can write it there and it sits there until the machine learning starts So when the machine learning starts it doesn't have to do that big join that really puts the database under pressure But it can use the data in a format that it needs and that this is consumed by that So we have duplication, but we have gained operational advantages How would such a thing look like? So as I said, we have this location data We have the products data and we have the sales data We can treat that locations and that products. It's master data. That's not such a high data volume We can keep that in the in the service as a table and we join that additional information to the sales stream And then that joint data we can append to a file and this file then sits there and waits until the machine learning started This is the possibility to cope with a challenge that the machine learning is still batch you might have Noticed when I explained that that I said well the sales in the location and the product data They are kept in the processor. So that's a state that we have in a processor and state as you might know is the nightmare of every distributed systems engineer because It's hard to handle What state do we need there? I mean for streaming the data could just rush through but for example in our In our scenario, we need to know what is the master data so that we can join that information for that So that's the data you want to join with but there are also other things that might be more subtle So when you have some time window processing you want To aggregate a stream that comes in and you want to know the sum every five minutes Then you need to know the data of the last five minutes so that you can sum that up correctly And you need to know when to start a new aggregation And there are different ways of doing that you can have hopping time windows or sliding time windows So that's just different possibilities that have different requirements to the state Formerly the database did this for you. You could ask them You could ask for master data. You could ask for things in the last five minutes. Now you have to know that within your service Well, it's fine. You might think you just keep that in memory. You know what came in everything's fine But there are some challenges. Well a processor might fail and Then it needs to restart and all state is lost that was in its memory So from where does he get the data for restarting? Or something less dramatic But as I said one thing that we really want to use is scaling so at first we had a processor that took care of all locations and Now we want for each processor to have one location. So this also changes the state in there Or let's say we had three processors for each location and now we want to merge that into one So at least the location master data needs now also to be merged to this one processor. How can we do that? well As I said, we just keep in memory and maybe we just keep all state in memory That we might possibly need in the future. That's one option, but it's not the best one You can read process just a stream from the beginning on to warm that up. This might take a long time or Each processor could keep its own database instance and save the state in there So that it can be used at restart and you just need to know to connect to which database, which yeah might also be interesting You can save your condensed state in a stream, which is an interesting thing because you have that streaming platform already Or you can just ask a different service that hopefully knows all the master data and you can Tell him, please give me all things I need to know several frameworks already exist to cope with that problem and it's very interesting if you start with a You're very naive approach and using that to have a look at such frameworks what they do and Why they might do it and then you you can really learn from their experience even if you are not in the same language for example, you can have you can have a look at Kafka streams or at Apache Samsung and Up to yesterday my impression was there is nothing we can use in Python and Well, we have to think can we employ some learnings from these other things? Do we have to write our own framework? Does it make sense or not? And we really were searching for an answer there and it's very glad to be in a talk from Winton yesterday and they said they now open sourced yesterday the Winton Kafka streams Which is the Kafka streams doing in Python and I'm really eager to have a more look to have a look and that they also said It's at the beginning state and there are still some topics they need to solve But it's very great to have a starting point in there and to check whether we can whether you can contribute to that and Yeah, just grow this functionality in Python, which would be really great to build more applications on that So you see here. Let's dig it up link Winton code Winton Kafka streams Well, and that's the end of my talk Let's summarize what have we learned and that you have more options for your data processing application than you might have thought But you also know the trade-offs trade-offs and I want to encourage you just to broaden your way of thinking about your application To see there's more things that you could use and you need to check whether you can live with a trade-off Or not and you know the challenges, you know some possible solutions So yeah, no go on and build some credit applications on that That's it from my side Okay. Well a great. Thank you very much for the great talk. That's awesome. We have Almost 10 minutes for question great timing any question Any question for Christian on streaming otherwise I do have a question Well, I'll start with my question Very interesting technology. I've never used streams How do you deal? Or maybe the framework deals with missing Value say you have a network glitch somewhere Somebody trips on a network cable and you're missing. I don't know two or three values in your stream or do you care? Well, what the framework Guarantees to you is that within a stream partition you get all the values I mean if it cannot guarantee it, you will get an error. So this is something really can rely on but What you have to deal with is some streams might operate at different speeds than other ones So for us one challenge will be and this was just too complex to bring it into that short presentation Let's say the customer delivers you some location and some product master data and he delivers you some sales data and now the Location data is delayed this there's some issue on the note that process is that and you are at a At a past offset, but you are the newest offset for the states data And now you would raise an error because you say the sales record is invalid And you don't because that location doesn't exist But this will be back because the customer will tell you well I already send you that why do you give me an error and this is something where your application logic then has to cope with and we will do that as we have one input The data input it always will assign delivery IDs and processing timestamps when this delivery when this was delivered And now your validation logic needs to check that you are that you have processed also the recent master data records So this would be the solution to cope with that issue But for the stream partitions you always can rely that you will get all values in the correct order Okay, great Thank you. Any other question? We do have a Few minutes seven minutes more or less. I can keep asking questions folks But if you have any question and a question over there, sorry, I didn't see you Hi, thanks a lot for the great talk. It's really nice to see Somebody else doing the same thing that we do as well in our company. I have a question about recovering State in the case that your consumer crashes Did you try some sort of seeking back in the stream to to find the latest? Latest state that you could recover from Well, that's the thing you it's not seeking back in the stream, but You can always have a snapshot of your data so That you know that you have in in some data file You save what what master data did I receive for example? And this is used also when we do some to some Acquiries for that to do some data science in there and with that stream within that Storage you're also safe on what was the offset in the stream which corresponds to that So by that you always know from which point in time you have to reprocess your stream If you want to get the updates compared to such a blob storage So this would be the way well to to search in the stream for updates that are missing Thanks, I like this approach If I can just quickly share what we did we implemented time-based seek in In the stream so each one of our Avro messages just like you we use Avro Each of our Avro messages contains a timestamp So we implemented a sort of binary search in the in the stream so we can return to a particular point in time But it is I mean it works, but it's not very elegant. So I like your solution better Yeah, yeah, I mean it's hard with time-based things as we want to think about always in a distributed systems manner And then you have different processors that might have slightly different time things So what we want to do is we want to use that stream of the delivery IDs Which then is guaranteed to always be in the same order and if you if you reference these newly created Things then you can ensure that you are really in the correct sequence Cool very excellent. So any other question. I'm standing here so I can see you Okay No other questions Well, um, let me see a question here. I have a beginner question. How do you integrate the data? Different databases, right Excuse me, can you repeat the question? How do you integrate the data of those several databases you have? data paths and they are Saving the data in several databases, right or not Yeah, I mean for each stream processors You have for each stream processor you have different possibilities to save that in there And not everyone uses a database and not everyone needs to be queried. So we want that That each service They do not need to know about what technology the other services use. So the communication really is With the streams and how each the databases are more that each service can be restarted for that or Or that also other parts of application can say well, I need to know all Locations that you have in there, but then he queries that service for that. So it's it's not a question of database Integration all this integration should work while the streaming platform and they should not need to know about that It's the only thing where we need to know about it is Well, as I said, we want to do some exploratory data science on this which is not connected to this to the streams but directly To to the output data, but then the data scientist knows where the data lies in but within that application or communication should be wireless streams Okay, we have time for one more question Yeah, one question Do you have any particular strategy to handle the Migration updating of the data that it's already queued into the topics of your streaming Infrastructure if it changed the format or anything like that due to code changes or things like that Let me see what I understood the question correctly I mean if it's not just the the schema evolution we have but if you say that we that we need additional queer additional Fields in the query for our machine learning is is that a question how to handle that? Yeah, the question is I mean I suppose there is a service that puts your the data into yours to mean Q Okay, and the data. It's probably Formatted Marshall Dean some way if you change that format in any way. I mean, do you how do you tackle that? For that you then really have to reprocess your data. So yeah, I mean If you have a different output format and this does not Help you what you have done in there then you need to reprocess that data Actually, I was looking for the next speaker. Sorry So we have we have no more time for questions. Sorry I'm looking for the next speaker. Please identify yourself and come to the podium and let's thank Christian again The great talk great questions. Thank you all Thank you


I did, it was difficult to do so, but I did my best. I really get blindsided, I usually sniff out something, but it was such a great conversation. Welcome to theCUBE, Dave Vellante, my co-host.  I mean, obviously Ignition's doing later stage deals, right? Yeah, yeah, so Ignition typically is, we favor early, but in this particular case, this is a company that I've been involved with for a long time. Thanks guys, yeah it's always a short guy, they got to move it a bit closer. No, no, move it this way more, so that. Robert just moved the whole table. Okay, well that's Robert. That way you can talk down towards this. Okay. We'll do a reset, okay, go ahead. Back to Ignition Partners, what's going on with the funding? Yeah, so we favor early, but this is again, this is a series D for us, and so it is atypical for us to do a deal in this range, but I've been involved with the company for a while now. I was an advisor early to the company while I was on the Venture Development Team at Accel Partners before actually joining Zen Source. And so the project even back then was very attractive to me, and so I spoke with Mike over the years, and so when I turned professional, again after leaving Citrix, when it became time to do this investment, the existing investors and Mike came to me and said, what do you think? Said, Frank, you guys are doing typically early stage, but we've got momentum here, and that's what really drew me to the investment was the momentum, and of course the team, and also the thinking of the team around being a platform. So even before my time at Zen Source, I worked at Microsoft for 10 years, I worked on Windows NT for releases of that, Randy SDK team, my other partners built platforms like Internet Explorer, DOS, Windows 95, and so we have an understanding for what it takes to get a platform out there, and we think this team has the stuff to get that done. So we talked to Kirk Dunn, COO, as you know. He said to ask you the question, why Cloudera, and then he also mentioned that you looked at a lot of guys in the big data space or Hadoop space. Why Cloudera and why them over the other guys? Right, so again, so we did have the opportunity to look at lots of things happening in big data. We did make, we made an investment in couch base. Again, we chose a leader that we liked there as well in the schemaless area. But again, in the case of, in particular, of companies commercializing Hadoop or doing things in a proprietary fashion on MapReduce, there were a number of things that really drew us to the investment. Again, the first thing, even though this is a series D company, which from a venture perspective tends to be late, there's a lot to be said for the team. Again, our familiarity with the team and the skill of the team and the kind of things that they've done is really super important, and that's paramount. The second thing is real momentum.  Anyone can talk about number of downloads or number of people visiting websites, but Cloudera has real customers. And finally, it's the point I made a bit earlier around being a platform. The team has a thinking around being a platform and the patience and persistence to take the long-term view to get that done. And the strength of a platform is based upon the things which stand upon it and what we've seen just by virtue of even all the independent software vendors here that are building to Hadoop and working with Cloudera to ensure the best possible performance and deployment environment for those applications gives Cloudera the best shot at becoming something very broadly deployed. And that is the real strength of a platform and that's where valuation comes from. As investors, that's what we look for. And the market leaders.  So even if you mentioned that, they're doing well, but you've seen the wars, you've seen the computer industry. And obviously most recently with Zen where you actually were in the open source battles, you've seen all those movies before. This open source community, talk about what's different and some of the dynamics. We had Kirk on and we had Ameron. Talk about the Linux versus what's going on here, the differences, the similarities. Just share with us your view because you work close with Amazon. You saw with Zen that open source communities and how that all formed. What's going on today? Right, so the interesting thing when I went to work on Zen, it was my first thing that I did in open source. And so when the first people I went to speak to was Brian Stevens over at Red Hat. And he encouraged me to try to unlearn everything that I had learned on proprietary software. And one of the first things I learned that in general for any given project, it actually is a pretty small team of people or a group of people that are doing the really hard work.  And so it was funny I thought that everyone, at least in the Linux community, seemed to be on a first name basis. Linux and Zen community. So if you said Ian, you knew that was Ian Pratt. If it was Simon, it was Simon Crosby. If it was Brian, it was Brian Stevens and I thought that was pretty funny. And then, and so the parallels here on the, again with the Hadoop project, it's pretty similar that you have a small number of very aggressive contributors and committers and then a large number of reviewers, or rather reviewers and then even a wider circle of actually users and people downloading or incorporating the project. And that's very similar to what we had at Zen. The difference with this project in particular relative to what we had at Zen is the ability for the project to attract a very vibrant community of independent software vendors that are bringing solutions to end users. So at Zen, by definition, our key ISV partners were operating system vendors and that we had key partners that were some management software vendors, but we really didn't expose ourselves in any meaningful way to application developers. And so to the extent that the customer got a solution, it was just incidental it was running on our, in a contained OS running on our hypervisor. And again, so this has far greater ability to get real customer impact earlier on than most other open source projects. Frank, can you talk a little bit about the similarities and differences with Red Hat? You mentioned Red Hat. I mean, everybody kind of left Red Hat alone for a decade and they went off and built the $10 billion valuation. Fantastic. Cloudera's out in the early lead, looking good and all of a sudden all this other competition comes in and the big guys, there's one of the big guys is nah, we're not going to let them run away with it. What's your take? What are the similarities and what are the differences? Well, I think one of the similarities with the folks at Cloudera doing, again, similarity to the guys at Red Hat have done for a long time, is really look through the open source projects, look through the community and ensure that the things that have the highest use value actually become part of the project, harden that, make sure everything's tested together and can provide that as a well tested, well supported stack with additional management tools on top of that. And so that's one of the things that really made Red Hat as a commercial entity gain customer adoption. And yes, many of the, much of the, a lot of their revenue comes from support. Red Hat is purely an open source business. But it's that ability to maintain the relationship with customers who are using the product, find out what are the various bits and pieces of new packages and new projects or enhancements to existing things and then going and bringing on and making sure as many of the project maintenance and contribution that's feasible as part of the company. And so I see that as something that the team at Cloudera has definitely learned from. What do you think about the Excel's $100 million fund? Obviously, iPhone fund with Kleiner. They had a couple wins out of that. They had a couple gaming companies that paid back the fund. You and I traded some messages and talked about sometimes these funds are gimmicky. Well, that's what I said. But there seems to be demand for these apps. Is it the lifestyle business that they're trying to enable? Is it just throw the seeds out into the patch and see what grows? Or is it really a venture opportunity? Is it, should it be part of a side fund? What's your opinion on that and how you think it's going to play out? So, historically, the specialized funds have been hit and miss. Some have been very good. Some have been less than successful. There were some that were done early on around processor technology that didn't work out that well.  So I think it remains to be seen. My friends at Excel have the ability to have a fund of great size. And so the ability to slice off a piece of the fund or raise a side fund to do that is a testament to their ability as a partnership. To focus on big data right now just seems totally obvious. I think perhaps even more important than the creation of the fund itself is Excel's ability to pull together the team of advisors that they have around those investments. So whether it's going to be 50 million, 100 million or all of Excel 11 that's investing in big data, the focus there is something I think that is notable. Now, why do you think it took so long? I mean, it's obvious. Given it, it's so obvious. It's just still in the early days, but you guys are usually way ahead of the game. Yeah, I mean, I think it's okay. I'm very new to venture. And so I can just make some judgments even just on the last couple of years. I think that in general, the limited partners, the folks that actually put money into the funds, you'll prefer things to not be so hard and fast so that a fund can actually have some agility. And so which is why you rarely see a particularly focused fund in that way. But I guess it's my impression that at the time right now with the potential of things happening in big data across the spectrum, that saying you're going to invest a specific amount of money in big data related things, that it could make sense. If you recall back to the slide that my friend Ping showed, it's not the $100 million fund is not specifically focused at MapReducer, Hadoop. He'll actually invest up-stack and down-stack. So he'll invest in applications that are using big data and also even cloud infrastructure to make big data a better place. And so I think that's just a logical thing for a fund like Axel to do is invest big money across those areas of technology. What's your advice for the BizDev side of it? Because it's pretty crazy. We were talking last night at dinner and you again lived that wars before. There's a variety of different approaches. I'll say Cloudera, which you've now putting your money behind that horse, they should have a specific go-to-market or continue their bit competitive strategy as we were talking to Kirk about. We said he's the Google of Hadoop. Google has such penetration, their goal was to grow the number of internet users. So as that increases their share, Cloudera has kind of got the same mindset. The more people are using Hadoop, they get a bigger, their pie maintains larger. Maybe some competitors get a position in there. What's the approach of these companies, the startups in particular, or the big guys like Informatica or Dell and these hardware guys, the services company? What should people be thinking about how to approach the go-to-market? Right, so in terms of if you're a bigger company or a startup and you realize you're going to bring a solution to market. And again, Cloudera is far from being a big company, so in a sense we have small companies and big companies trying to work together. I think what's a real driver here is just the nature of the data that's being produced across the board. Whether the data is being produced by people or machines or sensors or analytics on top of analytics, is that the data sets themselves are very large, but also the kinds of things, the kind of problems and the kind of analytics that are being applied against them are very large. So there's no one company that can do it all. And so even like harking back to my time at Microsoft, we made the operating system. We never made the backup software. We never really did a good job at very high-end systems management. And so it was relied on an ecosystem and community to go and do that. But the community does need a company and technology to rally around. And the thing that Cloudera will do is become that rally point for the ecosystem, be able to provide the kind of outreach and provide the kind of thought leadership, both in working with and driving demand at the end user customers, but also being the place that will continue to do enhancement to the product and to the project based on the kind of use value that the ISVs desire. And so it is about people working together in early days and yes, there's going to be overlap and there's going to be competition with certain things here and there. But at the end of the day, this rising tide will float all the ships. We're here live in New York City for Hadoop World 2011 at Frank Artali, the ignition partner. It's just invested $40 million led the round, series D for Cloudera, a more rocket fuel for Cloudera to continue to push ahead and maintain that lead. People trying to nip at their heels. A key word is try. Amarau will be on at 2.30 to talk more about this operating system concept platform. Again, like you said, your platform is judged by the things that it stands on and up and down the stack and we'll see that.  One, what are the white spaces out there? Like Microsoft, you mentioned backup, they didn't do those little things and Norton Utilities did a bunch of it, they made a lot of money. And what gets you excited right now? Now that Cloudera is kind of on the books and you're going to help manage that, watch that pony run. What are you looking at next? What's it getting you excited? White space and then what's getting you excited? Okay, so on the white, and these are kind of, it's kind of combined into one answer for me, to be quite honest. So on the white space side, what I, when we made the investment, our investment thesis is that this kind of technology and Cloudera as a team and the kind of mindset, this type of technology should be, should provide solutions and enable solutions to businesses of all sizes. So when we see there at the conference today, by and large, the solutions are still somewhat specialized. You see a lot about shopping on eBay. Okay, there aren't that many eBay's out there or you see fraud with large financial institutions. So, at the end of the day, we believe that this technology is something that can help people in their everyday work or their everyday social lives. And we see it hitting the social lives already from the big social websites. But the things that people do to work with their coworkers or business partners everyday should be optimized by this type of technology. And that is a big opportunity because when you take it, take the product down from the very large enterprises all the way down to businesses of 10 people or more, that's a very big market. What gets us excited in the investment community is going and finding those companies to build the products that can be sold to businesses of all sizes that take advantage of Cloudera. And also look at perhaps some things in the infrastructure space to make the deployments even more efficient and less costly to adopt from a CapEx and OpEx perspective. And so we'll look at both of those areas both at the top of the stack and the bottom of the stack. That's it, Frank Artali from Ignition. Thanks for coming on your new adventure but you sound like an old pro to me. Okay, thank you very much, it's great to see you.


Well, folks, we've made it to a new lake and it is Blasting winds Blasting winds just talk to a fishing freak though He's been waiting for the wind to die down. He's got a limited boat as well. He's he said this crappie like But I don't know if this is gonna be good. It's muddy and windy y'all it is Not looking very nice, but this is a lake that I really wanted to stop out and just see See what we got going on. So we're gonna give it a try out here in the old crispy See if we can stay afloat 64 degree water folks My oh my I'm gonna idle into this pocket this Cove Try to escape out of the wind There's just nothing I could do with the wind and the crispy So I'm gonna look for floater crappies in this creek. You got to get a fish today You know get a fish in hand every day of the trip. That's nice Oh just lost a good one Hey first spot of the day Came on my goog and green hand-tied jig This is a giant. I don't know what it is. It's a crappie. It's it's a magnum Going in baby. I'm back in this brush pile Here we go. There's a crop. There's a big crop. Come here, baby Should I return it? I don't know it almost looks like it's spawning guys. Looks like spawning behavior You know what I'm not getting it very good. I'm gonna let that one go and just kind of see what happens here My goal was to get get on some crappie and carry him home on my last day, which is tomorrow That one spanked it really nice. That's That's what you want. There's another one boys These are good ones Let's get them in here I might need to keep a couple of these now Now that we're getting our bearings I'm basically just right in the middle of this creek where it splits And I just found like a little piece of brush right there. There's a big one in it man a dandy Big dandy That's where I've gotten these last two fish and I'm just gonna I'm basically just gonna pluck this thing apart. I think there's a big one in here. Oh my gosh There's hammer in it These are all good ones. I don't know if they're spawning on this thing or staging They are they're thumping my little hand tie guys. Look at this Look at me go All studs All right, we get five of those We're doing all right Five juicies There's a few more in here. Oh Big floater big creek cruiser right here. Oh Yeah, baby, we're doing a little work here Magnum here we go boys There's a good one This is a good one. Oh God. I got her.  Oh nice one Not as big as I thought it was Absolute magnet signature Perfect float over her Nose.  Oh Good eaters. Yeah, I hear catching stringers boys That's nowhere close to a limit. I don't want a limit. I maybe want like That to eight of them and I'll be good. I Like to catch a few keep them fresh and repeat but I am really stoked right now about Catching them on this. I just decided to tie a goog and green 16th ounce Little crappie jig and I put my my own materials in there I just use one of our one of our heads. I just matched it green looks pretty sick and it is breaking Another thing that I did was I put a little um, I put a little juice in there put some of this liquid mayhem Just put a little juice on it. So I just I rubbed it up in there. I rubbed her up. Oh Man, it feels good to get some fish on the line Sir getting them yappies the size of that one on this Garmin was insane base of a stump Absolutely gone can't get better than that dead gummit. I'm having fun My fish was just on the base of a big tree I knew as soon as I saw the stump and I saw a fish on it. I was like And there's even more there's even more oh Oh gosh y'all this is like This is happy days right here, man. Just look at these blobs Blobs there's that stump. I was talking about There's a blob. I had one guys I it felt you Mungus that came off one of these stumps. It just came unpacked. I don't know what the deal was This is a big one though. This is probably a you know 15 incher Right here Coming in for a little. Oh, yeah sees it Wants it Good God. That's a good one. Geez. That's a big one.  Oh Come here darling. Oh Just fat Well, we were reaching a Reaching the threshold here on How many I want to keep but We're searching for big. I don't they have to just come back here in the morning Look for Sally Ah floater boys big floater six foot of water Almost inside 15 to get inside of 15 Let's go time boys. This is it give her that shimmy right there This is a good one. You know when they they're tough to catch. They're good ones. Just go on to the bottom Couple of there's another one on the bottom right here. She's coming. She got it Under the boat. That's the big one. That's probably the biggest one. Oh God They're just all big. I just love them They're just all big. I love them. Oh my gosh. Oh She stared at it forever Oh I guess we'll put one more on there in the day boys coming up to this nice brush ball to this is Some crappies y'all. It's really hard to stop fishing right now because I know what's lurking in here One of my big fresh water goals three pound crappie. This is it. This is when they're yoked up This is when they're fat. I mean we got some Amazing crappie right here some some pretty good eaters in there just studs We're gonna take those home Alright, so now that we have had success on the lake I Want to take you back here to the treehouse and I want to show you a little recipe I'm gonna we're gonna do a little bit of traditional here With some of the new the boogin eats fish fry But I also want to show you one of our favorite family recipes and that is the fish cakes the kids love it There's a certain way you got to prepare the meat, but I'll show you that here in a second By the way, this stuff is available. You can use my promo code Lfg at googinsquad.com and save on this. It is delicious Gotta try the original. I personally really love the spicy But if you're not a spicy person original is tasty for these crappie cakes what we're gonna do On half of these fish that I Get here is I'm gonna go ahead and I'm gonna cut them up in chunks crappie also have this The bigger ones once you start catching, you know above 12 inches I like to keep that little portion right there.  That's uh, that's a little extra beat right there You don't want to waste the key for doing these cakes I've tried it in a food processor. That is not the deal The best thing to do is such a good looking crappie is I like to cut them in little quarter inch chunks quarter inch chunks and Sometimes I will just just cut it cut it like I'm dicing a tomato almost Just like that. Look at that amazing meat. I mean it's so Clean you can just see through it. We'll take it to the rest of the kitchen and I'll show you the rest of the recipe In the kitchen with the lovely OSG I Know I know there's some some of you out there that are waiting to see OSG for hot Here she is back home from a trip y'all and ladies are always happy when you bring home the fish I'm just saying we're happy when you bring home the crappie. Yeah, it's crappie especially so Get your crappie danglers ready. It's it's a it's a woman pleaser. I can assure you so first of all let me show you guys the finished product after I fillet the fish and Basically cubed it into a little quarter inch pieces so these are little cubelets and something that I like to do is Put them in a strainer beforehand, you know, I'll shake them around a couple times Just get the water off of them.  Usually, you know midday or a few hours before we're going to eat them And I typically like to clean my fish You know a day after sometimes even two days after so they're nice and chilled that meats firm Just seems to come out a little nicer, especially when you're going into summer But these are cold water crappies and they're gonna be delicious It's just nothing quite like harvesting your own onions. I'm just kidding. What did you just pluck that out of there? Like a weed. What was that? No, it's been in my herb garden We planted these last year remember when you picked so many you picked so many of these onions They were like overgrowing so I ended up putting a bunch of them in my herb garden and I guess they regrow every year Oh, so we're gonna have like fresh green onion. Yeah little choppers I got you Very nice The crappie cakes All right, so this is what you need for crappie cakes. You need mayonnaise.   Yeah, so we've never we've just full disclaimer We've never used that in our Our crappie cakes just trying it because you know, we got it on hand something new But I'm gonna take this I'm gonna take the filets I'm gonna get my frying pan and we're just gonna do the old-fashioned style couple of golden crispies Just cuz you know, I like to do them that way a little bit One reason that we'd like to do the crappie cakes as well is because they're actually pretty good reheated Do them in the air fryer and you can I like to put them in the toaster oven Next day might maybe microwave my smidge and then put them in the toaster oven get that little Christmas back in there And they're actually pretty decent reheated crappie. I don't I don't like to freeze them I feel like it just it loses some of its It's luster. So I like I said, I like to catch a few cook them up and then repeat literally rinse and repeat Keep the Daniel going I don't measure anything but I kind of go you do with your mayonnaise Cooper though No, I just I'm gonna spoon it out actually So a couple of dollops three or four dollops of the mayonnaise. This is a five or six crappie I think we did five last time it turned out perfect So add a little bit of the the fry the Guggen fry Some of this green onion if you don't have green onion, you can do yellow onion White onion That's the messy part. Yeah, I don't mix it up gotta get in there For the seasoning Dusting there This is you could also probably use like smashed up Crackers Ritz crackers Ritz. Yeah, you could probably go with that. These are gluten-free though That's why we get these because OSG is gluten-free And actually helps me too because Everybody's stomachs seem to do well. It does like honestly it helps Ben. Okay, look notice the consistency Scooper kind of smash it on in there. That's a fun. That's the fun part. It almost feels industrial Okay There's one of your cakes Notice this is a lot of crappie here. It's not like it's not like you're you know eating a hush puppy You're getting the crappie Yes, the first time I made these I was like, I don't know this don't feel right because the fish feel really big Because I think the first couple times that I've made these we've like pulverized the fish and that was yeah It ended up tasting like a Yeah, like a like a fish stick for my childhood, which wasn't great So I think he is the key. Yeah So with with five or six crappie that I'm gonna say average around 12 inches, which are good ones good eaters We'll probably get a dozen cakes maybe Probably about a dozen cakes. I'm gonna say which is Plenty to feed a family of four And Ben and Emmy love you, too. So here on my side of the table recipe standard diss Golden crispy baby Typically what I'll do is I'll throw the the fish fry in here first and I'll coat coat them one at a time But you you can just do a full bunch if you want to get that adhered and then I'll do another another one You know, especially if If you're dealing with like 12 to 14 inch crappie if we're talking like mega huge crappie I usually just throw them back, but you know 12 to 14 inch crappie just great Great eaters and I like to do double-coat if they're like 10 inches if you're dealing with like bluegill or Little white bass or something like that or a little large mouth. I don't know I'm gonna judge You may just want to do one coat on there Some of the other Googians have told me to just do one coat on the spicy I like to double-coat me and let me and Lojo or on that spicy terrain You know, but little little bit of a olive oil shot there. Yeah, this is actually ghee. Oh ghee ghee butter shot Wow Where'd you get that? Where'd you find that whole food? Yeah, whole paycheck calm load them up and I think we put them on the fish setting last time We did a second coat. You did a second coat on top. They're top. Mm-hmm. So I did at the bottom I don't know if you have to be the bottom and this is I'm gonna guess it's gonna be 350 is what the yeah 350. What does the fish setting say 358 minutes? I think that was too much, but I packed it in there Let's do eight minutes. Okay. I bet you it I bet you that's good Literally just turn it on the fish setting and go But that might be that might be set for like frozen fish, you know, who knows all right, let's move on to the skillet This has been heating up for a little bit So I'm gonna turn this down to about medium And we're gonna add our olive oil if this is smoking that means we're hot. We're good to go. Oh, yeah She's smoking really bad actually That's gonna be I do when I'm talking, you know to the camera and I'm distracted. I'm really excited about something after years of Cooking fish. This is my preferred method I would love to eat golden crispies every day deep-fried at 350 in an oil But that is not good for your bodily functions So what I have found that is semi-healthy and still gets that that nice golden crisp is I do a little bit of olive oil actually half olive oil half butter And that seems to work really well and just shallow pan fry pump. Let's pump it up just a little bit more Oh Yeah, gosh You know what's kind of it's coming out of the the corn the corn water is hitting the oil and it's making a clean up Clean up all the fish off Alright brown butter and olive oil this way we get some of that nice buttery case Olive oil Handles the heat of the pan better And we are now ready to add our fish folks, so we'll add our traditionals make our golden crispies a Lot of fish cakes are in the oven a lot going on. Let's keep up with but here we go There we go Cool down and do a couple of these at a time So here is the key with doing your fish like this. This is where a lot of people make a mistake Uh They get excited when the fish gets hot and they want they're like, oh, no I don't want to burn it and they they move it or flip it too early and It takes off your seasoning Some folks like to really like add a wet binder and really stick that stuff on there I just do it like this and then I use a stainless steel pan or a cast iron and when the fish will Kind of remove itself from that stick that means it's ready to flip Perfectly several several times and I feel like I never really trusted it because I have one of those people Especially with chicken that like want to move it every couple minutes. So it doesn't stick But I don't know you've got it figured out with cast iron or with the stainless steel stainless steel That's just kind of how stainless works it It's it's what the pros use that's how they cook on it and It's key just can keeping a consistent temperature, which I have another in here, but That is typically how it works So just don't get it don't get excited and flip flip the fish too early if you're shallow pan frying If you are in a deep oil bed, then it's okay because you know, you're suspended in oil speaking of oil We got like oil on the lens right here Smashing butter on the screen Get a little brown on the side. I'd love to see that It's gonna be a light crisp last time we cook these you said I did too much You're like, yeah, it's too much. It's too much on there too salty. Oh, it was salty Yeah, and so I really want to see now because I did I didn't heavily cut these like I got normally do That's good. Everyone's different different buds different suds. That's okay. Wow. Has it already been eight minutes? It's time for a check It just it just Is a little brown, but I'm not I'm not convinced Couple more I Would say it's a ten it's a ten minute deal Yeah, big chunky these are about ready for a flip So we'll grab our flipper. Let's just see what's cracking here See what this other side's looking like get one handed with the camera. Oh, I'm golden crispy You notice these fish Sliding sliding around they're just ready to be flipped Too hard to get that splash dead nice crust on that Gordon Ramsay where you at with that crappie, dude? I don't know if you can keep up with that Butter and all Notice the flakiness Flakiness falling apart You start to get that separation That's what you want. Now. These are not deep fried. These are just these are just nice little butter noble shallot pan fry So second batch going in Don't flip them until they're ready to flip if you're using stainless cast iron very similar very similar, but Follow the method you will enjoy did four minutes 12 minutes. Oh, I did see what's going on Yeah, we want them to be a little brown little more brown There we go. I feel like that's it. That's it.  It's really nice. You want to try it? I do have lemon do we just I don't even think you need it. No, I'll just try it taste the butter you like it taste the zingy. No, it's not overly salty Okay, good. You like this batch better Just beautifully golden Beautifully golden ready to go. Bye. Gosh, that'll just make you want to go catch more All right, fish is done cooking. Just got a couple of extra nuggets in there sizzling I just want you guys to see that there's hardly any Of that fry left in there that's what you want You don't want like a bunch of fry coming off of your fish So when cooked properly They should look like this Now we just need to try the fish cakes We ready I think already might be cold off any of you if you want to set it off you want to go up an extra notch What is that this is a sauce is a this is mayonnaise with ketchup and some fresh ground black pepper Pretty pretty amazing Yeah That's what dreams are made of it's got some bulk to it. I love a good Croppie cake that doesn't fall apart when you pick it up. What do you got on that? There's a lot more breading on the inside than I realized Outside it doesn't look like very much Oh, yeah, and that's that breading will kind of expand is what we've learned fish is cooked perfectly Quick personally that was 12 minutes. I think I would add just a tiny bit more of those seasoning of the fry Okay Breading is perfect. Not a mayonnaise is perfect. They don't taste mayonnaise, but it's a good bingo Fish is perfect. It's a tiny bit more of the seasoning fry It's there. I do I do I really do Big big croppie cake fan. I would agree with you maybe just a hair more of this just add a little zesty pop but You could also just throw like A little bit of salt in there Pepperers, you know like a splash of cayenne. My gosh, you know why it's perfect for the kids though It is just look at this Let's get a close-up shot Look at a close-up a detailed look It's just perfect Great kids play that is an ultimate kids play you kidding me just we just need some fresh okra from the garden Not quite time yet Folks at home you pick what you like try these both My gosh It's nice to have croppie options kids Love these things. I mean it's just like fish sticks. It's just more fresh and delicious. Well guys catching the fish It's really fun But sometimes what I enjoy the most is coming home and watching the family eat And really enjoying it truly enjoying it and I try to do a good job with the meat preparation And the cooking so that they really do enjoy it and uh osg does a great job As well just preparing A lot of the things that I bring home, you know, whether it's from the woods or on the water It really doesn't get much better than a fresh caught croppie And you know what sometimes Stephanie will come to me and she'll say like hey, do we do you have any more blue catfish in the freezer? And I I think we might be out and I'm like, you know what honey? I think we are out I think I need to go catch some more and it just keeps this Perpetual cycle Of catch clean and cook going and it was really cool today to catch some fish on one of my hand-tied jigs What am I little googen 16th ounce hand-tied jigs that I made googen green really cool Just a fun time of year to go catch them. So you guys Google squad.com link down below get get you some gear get some of the new tubes Get some of the jig heads And don't forget about the batter the fish fry. It is now available. It is tasty y'all. I'm telling you it is really good so Do yourself a favor and do me a favor try something link down below use my promo code lfg and save at checkout, but Thank you guys for being here on these adventures Next up we got we got one more adventure that we're going on And it deals with catching crappie more on the bank So, you know, we got our like offshore floater crappies and then we got crappies coming in So we got we got a little more of that if you guys are interested So stay tuned subscribe to the channel smash that like button and I'll see you guys