Hey everyone, welcome to the stock. My name is Sagar Patwardhan. I'm a software engineer at Yelp. I'm part of the distributed systems team and I've been working at Yelp for last two and a half years. So today I'm going to talk about Siegel distributed fault tolerant concurrent task runner that we built to mainly run our tests in parallel. So let's talk a little bit about Yelp. So Yelp is a mobile app and a website that connects people with great local businesses. So it's basically a search engine where you can find whatever you're looking for a restaurant, a dentist, a doctor maybe. To give you a sense of how big we are, we have 28 million mobile web users. These are the monthly active unique visitors that we get. We have 74 million mobile web users and 83 million desktop web users. So we are pretty big. So in today's talk, what I'm going to cover is what is Siegel, why we built it. I'm also going to give a comprehensive overview of Siegel, how it works. So deep dive into Siegel. I'm going to talk a little bit about the cluster autoscaler that we built at Yelp. Next thing will be challenges and lessons learned and last thing will be a future of Siegel. So let's get started. To give you a little overview of testing at Yelp, we have a hundred thousand tests that we need to run in order for our developers to push code to prod. We have a monolithic application which has a lot of tests and we also have service oriented architecture and over 150 services. So if you execute these tests serially, it will take you two days to complete all these tests. So we can't really waste that much time on testing. We have north of 500 developers and testing kind of has a direct impact on developer productivity. The more time you spend in testing, the less productive our developers are. So Siegel to the rescue. So as I mentioned before, it's a distributed fault-or-end system which runs tasks in parallel. So to give you a sense of how big Siegel is, we usually have 350 Siegel runs every day and average runtime for each Siegel run varies between 10 to 15 minutes depending on what kind of tests we are running. We spin up 2.5 million ephemeral Docker containers just for the testing use case. These containers are basically different services at Yelp. Developers want to test their service against different services. So we spin up a lot of containers. We use cluster autoscaler. So our cluster is always varying in size. So at night when the load on our cluster is minimum, we have 70 instances and our scale, our cluster goes up to 450 instances during peak time, which is in the afternoon US time. We have started using spot instances. So initially we started off with RIs and spot instances. Now we are just using spot instances. Every day, Siegel executes 25 million tests. It's basically to ensure that developers are pushing code that is okay to push to prod. So let's talk about applications of Siegel before diving into Siegel itself. So mainly it was written for the testing use case to run tests in parallel, but we have extended it to different use cases. So we have a load testing framework called Locust. This framework is mainly used for testing different endpoints of services. So let's say you are a service owner and you want to see how many requests per second you can sustain for a certain endpoint of your service. You would use this framework and make parallel calls to your service and see if it can sustain the load. Next use case is photo classification. So Yelp has tens of millions of high-quality photos and we use deep learning to figure out whether a photo was taken inside a restaurant, whether it was taken outside a restaurant, whether it contains food, whether it contains drink, etc. So we train the model and then we basically use that model to classify photos on Siegel. So initially we used to run this on Spark, but now we have migrated it to run on Siegel and we run classifiers on close to 90 million photos in less than a day. So let's talk about Siegel. So I'm going to walk you through a testing workflow and then I'll go into specifics about each section.  The first thing you will do is you will use a command-line tool called Run Siegel. You will specify which branch you want to test and which tests you want to run. After starting a Siegel run, we start a build on Jenkins. We do that mainly because there are certain things that need to happen before we can start executing tests on your branch. So one of the first things is we create an artifact.  This is later used by executors to run tests. The next thing we do is we need to figure out how many tests are there in your branch. So we run a job called Test Discovery. This job goes ahead and basically calculate, basically figures out how many tests are there in your branch. So next thing that we do is, as I mentioned before, we run tests in parallel. So we need to split these tests into smaller chunks or bundles as we call them. So we run a program called Bundle Creator, which divides these tests into multiple bundles. After doing that, we start the Siegel Meso Scheduler, which talks to Meso's master and it basically gets offers from different agents in our cluster. After getting all the offers, we start sending executors to the agents. First thing agents do is they download the artifact that we uploaded in the previous stage and they untie the file, they compile PYC files and start executing tests. After all the tests are done, the results are reported to Elasticsearch and Kibana, which is our main data store for storing test results. One thing that Siegel executor does, apart from reporting results, is upload standard error standard out of each executor. So this is mainly used for debugging. So we direct standard out standard error to a specific file called Siegel log and we upload it to S3 at the end of each Siegel run. We also report a real-time metrics to signal effects in DynamoDB from our scheduler. We have a UI service called TestResults. This is mainly used by developers to see results of their tests, whether all the tests in their brands are passing or not. And yeah, that's like a complete overview of Siegel. So this is specifically for the testing use case, depending on the use case workflow will vary. So now I'll talk about specific of Siegel. So first is Siegel Meso scheduler. So the scheduler itself is written in Python. We are currently using LibMesos to talk to Mesos. We have not migrated to HTTP yet. We have one scheduler per test suite. As I mentioned before, there are different test suites at Yelp. For example, unit tests, acceptance tests, integration tests, selenium tests, et cetera. So each test suite basically corresponds to a Meso scheduler. So at peak time, we have more than 50 schedulers running. So we rely on Mesos' DRF algorithm to distribute offers to us. And so far, it has worked really well. We have never run into an issue where a scheduler is starving for offers or a scheduler is getting a lot of offers. Siegel also has customizable concurrency. So because it is supposed to run batches, you can either run your batches really quickly by setting the concurrency to a very high number, or you can reduce the concurrency and let your batch run for a really long time. Siegel scheduler is also fault tolerant. So it can deal with agent failures. It can deal with bundle failures. It basically retries bundles. And I'll talk more about the retrying strategy. Let's talk about placement strategies. So basically, the aim here is to optimize for Siegel bundle setup time. As I mentioned before, each executor has to download tar ball from S3. The tar ball is typically a few GBs. And we also have to do some things before running tests, like compile, Python files, et cetera. So to optimize for Siegel bundle setup time, we use two strategies. One is affinity for slaves. So if we have used any particular mesos agent before, we'll try to use the agent again so that the executor tar ball is already available and ready to use for the new executor that is getting scheduled. And the other thing is we use as many resources in an offer as possible. So if you get an offer with, let's say, 30 CPUs, 200 gigs of memory, we'll try to fit as many Siegel executors into that offer as possible. Siegel is supposed to be a batch runner, so it's really time flexible. There's no upper bound on when a Siegel run should finish. It's okay if our Siegel run gets delayed by a minute. If an agent goes down and we lose all the executors on that agent, we'll reschedule them and they'll finish in a few minutes. This also simplifies scale down because we are using up all the resources that are offered to us. We are always left with a few idle agents in our cluster which can be terminated without causing any disruption in our cluster. So this UI here is what we call Bundle Visualizer. So y-axis is the time scale and x-axis is basically bundles. So the green bundles are the ones that have finished successfully. The red ones are the ones that have failed. So let's say a bundle fails. What we do is we split the bundle into two bundles. So let's say if a bundle is of 10 minutes worth of tests, then we'll split it into two bundles each of five minutes. This is to basically make sure that we are finishing within a reasonable time. We don't want to schedule the bundle again and spend 10 minutes on that bundle. Siegel Executor. So we have written a custom executor in Python. This executor is currently using Mesos Containerizer and different Siegel isolators. The main job of this executor is to do set up and tear down. So set up, as I mentioned before, is basically getting the tar ball.  Apart from set up tear down, it also reports utilization of our process. So we run a thread in that executor which is constantly getting CPU metrics, memory metrics, how many network connections we're using, disk usage, et cetera. We use that to allocate resources to our Mesos bundle. It also uploads log files to S3 and reports test results. We also have special constraints to run these tests. We have some resources that we call them cluster-wide resources. These resources are not tied to a particular agent. So the typical example of these resources is Selenium Connections and Database Connections. We want to allocate them without tying them to specific agents. So to achieve this, we use ZooKeeper, fmrl-z nodes. So there are two z-nodes in our ZooKeeper cluster. One keeps track of how many connections you can use, and the other acts as a parent node for fmrl-child-z nodes, children-z nodes.  You'll first check what is the limit on number of connections. Let's say for Selenium. Then you will query get children on the parent z-node and see how many fmrl-z nodes already exist. If it is less than the limit, then you will create a z-node and use a Selenium Connection. Same goes for Database Connections. We use ZooKeeper logs to make sure that our access is atomic and we are not running into any race conditions. As you may know, once the executor goes away, these fmrl ZooKeeper connections will be removed and will basically reclaim the resource at that point. Let's talk about monitoring and alerting. So we have real-time monitoring and alerting using SignalFX. So we send time series data to SignalFX and they visualize the data for us and they also provide alerting. So the graph that you see here is representing the number of bundles that have failed in the last one day. So green bars are corresponding to finished bundles. Red bars are the bundles that have failed. Blue bars are the bundles killed and yellow bars are the bundles lost. For each bundle in Seagull, we specify runtime. So for example, for a test use case, we specify the runtime equal to 30 minutes. So if a bundle goes over 30 minutes, we go ahead and kill the bundle, assuming there's something wrong with the bundle. We also do log aggregation in Splunk. As I mentioned before, we upload a standard error and standard it out of each executor to S3. We then ingest all this data into Splunk and we basically query Splunk to get statistics across multiple Seagull runs and across the cluster. So in this example, I'm searching for a Yelp serializable object validation error. So it's basically showing us when it happened in last four hours. This has been really helpful in debugging cluster-wide issues. So we have better instrumentation of what's going on. So as I mentioned before, we split tasks, our tests, into multiple chunks. So to do so, we use two different algorithms. First is a greedy algorithm, which is basically a bin packing algorithm. We keep track of test timings. So whenever a test is executed, we store how long it took to execute and we upload it to Elasticsearch. We then run a nightly cron job, which calculates P90 for last one week for each individual test and it uploads it to DynamoDB. And this data is basically used for bundling. So when we are creating chunks, we sort the test list and we basically split the tests into bundles of 10 minutes. This algorithm has worked really well for us. We've also tried different linear programming algorithms, but the gain that we get with those algorithms is not significant.  So some tests have to be executed together while some tests can not be executed together. So to achieve that, we have written a pulp LP solver. So this is a really simple solver which has three main goals. One, to make sure that a single test is present in a single bundle. Bundle never goes about 10 minutes, and number of bundles created is as minimum as possible. So as I mentioned before, we use an autoscaler for our cluster. You must be wondering why it's a bad system. Why do you care? So this is the weekly user strand of our cluster. So during weekends, the cluster is at minimum scale. On Mondays, it scales up Tuesday and Thursday, and it again, the resource utilization goes down. So all the dollars that you see here is the money that we save. Same goes for daily user strand. So we have developers in Europe which they basically push code in the morning US time, 3 a.m., et cetera. And during US office hours times, the cluster utilization goes up. You can also see a spike during lunch hour. So the utilization goes down during that time. So this is the overall architecture of our autoscaler, FleetMizer. So there are two components of FleetMizer. One basically gathers data from Mesos and other sources, and it stores the data into elastic search in DynamoDB. We use signal effects and sensor for monitoring the autoscaler itself. So it emits useful data regarding when it took a particular decision, what did the cluster look like and why it took a particular decision. And it basically queries Amazon API to scale up or scale down the cluster. So the autoscaling strategies that we use are we use two different strategies, CPU utilization and seagull runs and flight. So CPU utilization, our workloads are CPU bound. So we always run out of CPUs before memory. So we track CPU utilization and if the CPU utilization goes above 65% for the last 15 minutes, then we scale up the cluster. And while scaling down, we check if the cluster utilization is below 35% for 30 minutes. If it is, then we scale down the cluster. This signal was added last year. So we basically, whenever a new seagull run is triggered, autoscaler gets notified that a run is in flight. So this has helped us a lot. To give you an example, let's say we are about to scale down, our cluster has been idle for last 30 minutes. But all of a sudden, developers in the EU come along and send a bunch of seagull runs. So our autoscaler anticipates how many seagull runs are in flight and how many resources still need. And depending on that, it either prevents the scale down or adds more resources if needed. Scaling down is difficult. So we have implemented placement strategies to allow smooth scaling down. We also use AWS Spot Fleet. And one disadvantage of that is that we cannot specify which instances to terminate. So our autoscaler queries each individual mesosagent and figures out how many tasks it is running. And if a particular agent is not running any tasks, then it will select that instance for termination and it will go ahead, terminate the instance, and then readjust the Spot Fleet capacity. This is to basically trick Spot Fleet into thinking that we have terminated the instances that we want. As I mentioned before, we use Spot Instances. So back in May of 2015, we were using a combination of RIs, which are reserved instances, and on-demand instances. We slowly started ramping up on Spot Instances. And you can see in July 2015, we saw 55% reduction in cost. And we started using all Spot Instances around December of 2015. And the utilization, the cost of the cluster has gone down 80%. So that's a huge amount of saving. Now I'll be talking about some challenges that we faced while building the system and the solutions that we came up with. So first issue we ran into was bandwidth while talking to S3. So we are using NAT boxes to talk to S3, and there's limited bandwidth that NAT boxes have. So we also use Docker images, which are backed by S3 buckets. So we had to come up with a way to kind of avoid talking via NAT boxes. So basically the solution here was to use VPCS3 endpoints. They provide fast and secure access to AWS S3 without any bandwidth limitation. All the S3 limitations are enforced, but in terms of bandwidth, there are, they don't impose any restrictions. And one good thing here is that your traffic never leaves Amazon network. Your traffic never goes to public internet. It just stays on Amazon network. But one caveat here is that your data and your application has to be in same AWS region. So if your bucket is present in an AWS region, then you must co-locate your application in that same region to take advantage of VPC endpoints. After moving to VPC endpoints, we've seen lots of free bandwidth on our NAT boxes and other applications in the VPC are happy now. Central Docker registries. As I mentioned before, we run 2.5 million Docker containers and lots of different images we download from Docker registries. Docker registries are backed by S3. So they simply send S3 redirects after getting a request. But, so initial setup we had at Yelp was basically we had multiple Docker registries on a single host backed by an NGINX instance. NGINX was supposed to load balance across these registries. But the scale at which we were operating basically failed to cope up with the number of requests we were making. We also tried to put registries on different hosts and put them behind ELBs. But we ran into issues with sticky cookies, et cetera, and we couldn't really do it. So the solution here is that we run multiple Docker registries on each agent. So Docker registry is basically a Docker container which sends a redirects to S3 and it doesn't matter where it is running. So on each single agent, we have two different registries running and we use ZC hosts to basically resolve the result to local host. So this has been a game changer. We have never run into any issues with a bandwidth or Docker registry is not being able to cope up with the number of requests we are making. I highly recommend this if you ever have run into this issue. You should definitely try it out, try out this solution. The second main problem is spot instances. Spot instances can be reclaimed at any point of time by Amazon. So you have to make sure that your application is fault tolerant. It can deal with executors going away. It can recover from this failure. So what we do is we have a cron job which is constantly checking for termination notice. After getting the termination notice, we terminate all the executors that are running on the agent. Our scheduler gets task loss status update for each executor and it basically reschedules them elsewhere. We also terminate Mesos agent process. In order to prevent it from getting more tasks and basically losing those tasks. So this has also worked really well for us. We have not run into any issues with spot instances going away. As you may know, spot markets are pretty volatile. Bit prices for spot instance prices can go up, down, they can fluctuate and can have an adverse impact on your application. So getting the bit prices really hard, how much money you bid on a particular instance, it's hard to determine. And the trade-off that you have here is basically availability and cost savings. If you're willing to pay more cost, then you can get more availability and vice versa. So the solution here is to make your application fault tolerant and diversify into more spot markets. So for Seagull, we use 10 different instance types and three different AZs. So we have 30 different spot markets. We use a wide range of instances, C4, I2s, M4s, and basically this has allowed us to have compute cluster available all the time. We also use spot fleet that makes sure that we are never paying more than on-demand price of an instance. If price of an instance goes above on-demand price, then we basically terminate those instances and we get capacity from other markets. So this allows us to keep the cost down. Issues with Docker daemon. We have ran into a lot of issues with Docker, mainly because of Docker daemon. Sometimes Docker basically gets locked up and stops responding to our requests. We've also seen deadlocks in Docker daemon. Every time we upgrade Docker, we always run into some or the other issue. Docker daemon sometimes fails to resolve DNS while other tools like DIG are working fine. We've tried different solutions like Sego, DNS, et cetera, but it has not really worked for us. We use AUFS as our union file system and we are running kernel 4.2. And we've often run into issues where it basically causes kernel panic and our CPU is going to soft CPU lockup state where you cannot run any task on the agent. So we basically run a cron job, which periodically it SSHs to each box and it checks a D-message and sees if the instance has gone bad, if it has, then it goes ahead and terminates the instance. But the AUFS is probably one of the better union file systems based on our experience. Orphan Docker containers. So after running 2.5 million Docker containers, we have had a lot of orphaned containers. So what happens is that application tries to remove containers, but because Docker daemon is not responding, it gets an exception and just exits and Docker containers are left behind. These containers are not account, the resources that these containers use are not accounted for in Mesos. So Mesos thinks that there's more room to schedule tasks on the host and eventually our boxes run out of memory. So this used to cause a lot of issues for us and then we came up with a solution. So we wrote a tool called Docker Reaper. So it's a proxy for Docker daemon. It's written in Go, it's supposed to be transparent. So if you send a particular signal to it will forward to its children and it basically cleans up Docker containers after the child process goes away. So the way it works is we have Mesos executor. It creates a Docker Reaper. It basically creates a unique socket and sets the Docker host environment variable which is used by Docker clients, Docker daemon. It fork execs the child process. Child process then goes and creates containers. So create container API call is intercepted by our proxy. It forwards the call to Docker daemon. When it gets the response back, it gets the container ID, it records the container ID in memory and forwards it to the child process. When the child process goes away, it goes ahead and removes the container. So this has basically allowed us to reduce the number of orphan containers. But because of various issues of Docker daemon, we still are left with a few orphan containers. So we run different cronsops on the agents that check which instance of its Docker containers are running for more than 30 minutes and they go ahead and remove the containers. Mesos maintenance mode. So this is a great feature that Mesos provides but unfortunately it is designed to be used by a single operator. So anytime you make a post request to this API, it will override the existing maintenance schedule. So if you want to use this feature in your agents, you need to make sure that only one agent is able to talk to Mesos and make post requests to a particular endpoint. You need basically external locking to ensure that this happens. So you can use a Zookeeper based mechanism to make sure only one agent is talking to Mesos. So yeah, we are talking to Mesos folks and we're trying to figure out if we can improve the maintenance mode and make it usable by multiple operators. So yeah, future of Siegel. Where would we like to go next year or so? We'd love to use oversubscription. So oversubscription will basically allow us to use residual capacity in the cluster and run low priority batch workload. For example, photo classification. If there's no stringent upper bound on when classification should finish, we can just run it on the residual capacity in the cluster. We're also replacing the core component of Siegel scheduler. Siegel scheduler was written a long time back and it needs an upgrade. So we have basically written a library called task processing and we are replacing the scheduler with this library. We'd also love to use CSI plugin once it is available and replace the cluster wide resources with this solution which is more robust and we delegate the responsibility of allocating cluster wide resources to Mesos, which is great. We also wanna make Siegel easier to use. So more and more people at Yelp can take advantage of parallelization that Siegel provides. Executor improvements. We'd love to containerize everything. We love Docker containers. Most of our products in workload runs in Docker and we'd love to containerize everything. We'd also like to use Mesos containerizer and Docker runtime and basically eliminate the need to talk to Docker daemon. This will save us a lot of effort and money. Basically we don't have to talk to Docker daemon after using Mesos containerizer. So Mesos recently added this feature called nested containers and parts. This is a great feature, especially for Siegel. We can use nested containerizer and spin up Docker containers within that container and once our application finishes, the container will go away and it will reap all the containers that were started within the nested container. So we don't have to deal with often Docker containers. Autoscaler improvements. While our autoscaler works reasonably well right now, there are things we can do. There are better algorithms that we can use and drive up the utilization of our cluster. Right now we are using a single spot fleet request and this basically prevents us from using more spot markets. So with a single spot fleet request, you can only use 30 spot markets. We would like to use multiple spot fleets and expand into more markets and see if we can get more savings. And we would also love to use more instance types in our Siegel cluster. So that may drive it on the cost. So yeah, we are hiring in Europe in London and Hamburg office. We're looking for distributed systems engineers and managers. If you are interested in working at Yelp, please reach out to me. I can get you in touch with the right people. So yeah, that's all I have for this talk. Anybody has any questions? Thank you.


Welcome to Reader Syndicate 3.0, the next evolution of the look into counterculture that is Canada. My name is Matthew, owner of Riot Seeds, and this started as a one-man mission for strain history and breeding science. Over time, it's evolved into something bigger, better, and more of a team effort. We will be joined by members of the Cannaluminati and other friends throughout the season to hear their takes on grow techniques, breeding science, strain history, and more. Our mission is to combat the narrative that corporate cannabis and seed posers are obfuscating for their own financial benefit. Welcome to the Underground. We are The Syndicate. Welcome to Reader Syndicate. I'm Matthew, here with our co-host, Thousandfold, and today we're going to be talking to Patrick Pico. He's been a good friend of mine since I think we first met at Emerald Cup 2017 in person, and ever since we stayed in contact, and yeah, he's got a fascinating tale through the seed world and a lot of experience in the seed world, making seeds at scale and more. Thousand, you want to kick it off? Yeah, I guess first I wanted to ask everyone how their week has been. Fantastic. How's the new year? New year was great. Family man, so super mellow, which I'm grateful for, because otherwise I'd be feeling like shit right now and every day since the first. We did absolutely nothing. I like the slower pace these days. It's really gratifying, so it's been a great week. Do they celebrate New Year's in Kenya where you're from, Thousand? Yeah, it's traditional to just like stay up late and watch.   Our city is very conservative, so fireworks have always been really big here, and like there's never really been any pushback on fireworks, despite it being desert and like lots of brush and things catching on fire every year. But yeah, it was, they were lighting off boom booms loud. I don't even want to say that word.  It was wild, so my big old Anatolian shepherd was on my lap the whole time, but yeah, and it's usually never fun. What happens if you get old, I guess? I don't know. We did have a whole bunch of fireworks out here. They allow those there? They do, yeah. Wow. That does surprise me a little bit. It's a bit of a wish of a climate, I guess. It's probably not as risky as it is for you guys, but still, you know. Do they allow like bottle rockets and like firecrackers, or is it just like the safe ones? It's the safe ones, I think, relatively, yeah. Yeah, it's weird where we're at in California too, because like they import, like you just go over the border over to Nevada and people bring back like M80s and like big old monster things and like there's no way to really rationalize like that, that legality, because especially with online now, anybody can get anything, anytime, anywhere. It's really opened up that market, you know? Yeah, I mean like you guys were talking about earlier before we started recording. I mostly feel sorry for all the animals. Yeah, you have to put up with it. Or even veterans with PTSD or... Yeah, dude, yeah. You know, that was something I hadn't really considered until later in life.  It really affects them. So yeah, people need to take more consideration to that if they really do care about the troops and all that. Okay, well, I thought maybe we could just ask Pico Patrick here a little bit about what he's been doing. Obviously, that's the whole point of the entire conversation, but maybe just a little brief thing on what he's been up to just recently and maybe touch a little bit on the Bank Seed Hub. Okay, yeah. So my most recent job is in Arizona. I left California last year. I got laid off at the end of the season, which I'm sure we'll get into a little bit. Just prices plummeting, unfortunately. We had a great year, but it is what it is. And through luckily some good networking, I was able to find a job relatively quickly. It was something I had kind of passed on earlier because I didn't want to give up. I was running an outdoor operation and didn't want to kind of give up in the middle of the season and go take a job somewhere. So circled back on that and currently doing that, which is in Arizona. It's a 100,000 square foot facility. We're doing indoor, which is a back to indoor for me. I hadn't been in that in a couple of years. So it's fun. It's extremely challenging. And like any other aspect of working with the plant, I just learn new things every single day. And that really drives me to just continue on this journey and keep going. So that's been the most recent iteration. Very nice. And yeah, you like you've told me and I've read your interview in Skunk Mag that you've obviously been working at scale for quite a while. Right. So that's going to be quite cool for us to get into later on. Yeah. I mean, we can we can talk about the bank if you want to. That was that was a really cool project. It's a good thing to segue into. So initially, when I got the job at the bank, I was kind of doing the same thing. I was out there looking for work. I had gotten a license in Sonoma County, had fall out with a partner, had to sell out of the building that we own at the time and the license and made a partnership with an extra company to do that. That partnership just just fell fell apart really, really quickly and ended up leaving that position and then was pretty much freelancing for a little while doing some sales and some other things in the industry and then was looking for work. So an outfit that I had interviewed with prior due to some others, some good networking and try to get in with as far as just a grower any job worker in their nursery ended up circling back on me and interviewing me for what they were looking for at the time, which was just someone who could come in, make seeds, grow seeds at scale. They were doing 32 acres the first year I was with them and they wanted to plant all of it from seed. So there was a series of depth crops that were getting planted and there was also a lot of auto flowers and we're talking like, you know, one farm would have 170,000 seeds twice a year. So massive, massive amounts. So the first part of the job was making all the seed, which I was tasked with. And in the interview, I told them, you know, I haven't done this at scale. I had only done one reversal prior and they wanted all them. And so it was like, I was very honest with them and telling them like, I know how to do this, but I've never done it at scale. And I was lacking a lot of confidence to say the least, but they basically were very supportive and told me, look, if you will come in and take this job and try it, we'll learn together, you know, and fair enough. And really, it all worked out really well. The reversals were extremely successful. And we made three million seeds in the first year. So at the rate of, you know, growing a couple hundred thousand a year and making millions in the first year, I had essentially somewhat worked myself out of that job because we made so many seeds. Can I ask you, just kind of to interrupt you and ask, what do you feel like you learned from this one? Oh my God, from that job? Yeah. I mean, the entire reversal process, I'm very analytical. I keep a lot of notes. I've used a lot of data. So, you know, for what we were doing with STS reversals, just the proper amount, tracking it per strain, tracking it per specific varieties and looking at lineages to see if there were going to be correlations with other strains I wanted to reverse and how successful or unsuccessful they would be, how many plants per seed you're going to get per square foot, you know, how big the receiver should be those timelines of staggering the donors and the receivers, what the ideal timeline is depending on those genetics and their finishing times and, you know, there's also a graph as far as for seed maturation, how long you want the seed to actually continue to mature and there's four, I believe there's four, like four data sets that they're looking for that all hit their peak at a different time. It's like the germination rate, the storage rate and there's two others, one's vigor and one's another one. So, ideal timelines to where you hit all of those at their peak before you start to basically get diminishing returns or if you overdo the seeds you'll have the crack shells, which obviously nobody wants to buy. You can still grow them. They germinate just fine, but nobody wants to buy those. Isn't that wild like that they still grow? Like, I remember what line was the first time I saw that, Bodie did it with, it was one of the G13 hash plant lines. Chem91 G13 hash plant was the first time I saw it and like he sent me a bunch and they were kind of cracked like, I think they still grow and they did, which blew my mind, you know, considering I thought air was getting in and going to dry it out, but they did crack pretty well, nobody wants to buy them. Yeah, nobody wants to buy them. I mean, honestly, they crack a little bit easier because there's the water will penetrate, you know, into the embryo right away, right? And then they don't have to actually break through the shell to get up above the surface, depending upon what substrate you're using, if you're using like, you know, little Excel plugs or something or rock, well, those things can be challenging.  So there's certain strains that just, that happens. And even when you take it to the number of days that you would want to go to for all of the other things that you're looking for in a great seed, you know, some strangers got to take them a week earlier, two weeks earlier, so that crack doesn't actually happen. It doesn't open. So learning things like that. Again, I was doing crop registration on all this acreage. So with that came a lot of other plant diagnosis that was happening. First time I saw a beet curly top virus, and last time, thank God, that was a tough one, you know. I couldn't have been able to identify that in cannabis. I've never seen that before. It was, it's very wild. And luckily it runs its course fairly quickly, like 30, 35 days and it's out of the plant. And it goes from wherever the bite site is up the plant is affected. Everything below and on any other branch is good. It doesn't affect it. So how many times it got bit by the leafhoppers that carry it will affect how terrible the plant is in yield and everything else. But I mean, it'll take a plant that's about to be 10 feet tall and, you know, seven feet around and turn it into a tiny little squat bush that is like two feet by two feet. Oh no. Yield impacts are tremendous. And you have it, you have it.  It's going to run its course, but you're not going to get good viable, but off those branches and it hurt. That one hurt. So that was a great learning experience. Just everything at scale. I ended up running the farm ops on that, you know, with that outfit that the prior, or the next year rather, because I'd made all the seed and we didn't really need that as much anymore. So now it was just about growing the seed and running the farm ops. They liked what I did from a plant diagnosis standpoint. I took a lot of videos again, shared a lot of information, would give reports back on all the crops and where we were at with the acreage and then would try to map out the harvest plan, which was insane because, you know, if you're running that many acres, it's all ready at the same time. Yeah.  Like we were talking about another episode that people really don't take that consideration, especially when they're new. And like that is a major, major consideration. What do you dry it all? And if you're making seeds, how do you store it, dry it? And when the seeds start falling out bit by bit, like how do you catch them, identify with what plant? Yeah, there's so much that goes into that. Well, the seeds were made and kept in a totally different area.  So that was like independent. And then all of the growing acreage was elsewhere. And it was across multiple counties. So it was like putting a lot of miles on the car and doing a lot of driving, which was also awesome. I mean, it was like the ideal job. The fact that I had a job just to make seed was the coolest thing in the world to me. And it was like, I'll do whatever it takes to keep this job. That's a seed maker's dream to be able to grow at scale, pop a bunch of stuff that you want to see and get paid to do it. I've never found that. And I've been doing this a long time and that is something special. It's not something for those listening that aren't really that keen on what goes on in the seed business. It's not a common thing to be offered that kind of job. So it is special. Definitely. Yeah, I don't know anyone else had it before since. Nope. Nope. So I was super lucky to be in that position and right place, right time. Networking has been so important and everything that I've been able to do.  So with that job, with its unique challenges and with what we basically want, I had to put any of my ego and any of my prior work kind of to the side. I was working for someone else. I had to fulfill what they wanted. What would sell for them wasn't necessarily the stuff that I was making. And so for those years that I was with them, I was really doing a lot of OPW, which is other people's work, which I was fine with. Because again, I had my dream job. So who cares? I'll make it right. I'll go do crop reg and I'm still doing the work. So it was all good from my perspective. So with that, the challenge that we had was genetics, sourcing genetics, always having good genetics and always having the next thing. The way I was doing it, the way I was being asked to do it was using other people's work from the previous season or the previous cuts that had come in and mixing and matching and doing all that kind of shit. And it was always catering to a very thick market that wants something new all the time. Yep. So with that, there's no actual line breeding or learning. You're not really getting heritability of traits and the things that are really interesting except for the auto flower work, which was probably one of the greatest learning experiences because of that. Oh, yeah. Definitely. So with that, I started the Bank Seed Hub with the partner who was the owner of the company, which the idea that was so many great breeders who had amazing brands already had been left out of legalization for one reason or another. Whether it was they were in a garage and they were never going to get the zoning correct to be able to have their seeds enter the market. This is before the 2018 Farm Bill. So there was still a lot of very, very gray area as far as how anybody would get seeds anywhere on this scale and two facilities that were being opened up across the country that were totally legal and compliant, so to speak. So with that, I started the Bank Seed Hub and the idea of that was to basically be a conduit or a bridge to all these other amazing breeders who were out there to come to this 10 acre facility, make your seeds here, or give us the plans, tell us what you want us to do. I'll make the seeds for you. We will sell them back to you at a certain price if you like. If you want to go hit the market with them, we'll grow out a shitload of them as many as you possibly want us to grow out so that we can do the crop registration, phenohuns, and keep building for the next season and basically just offer the resources that we had to all the amazing breeders who were already out there. So that was the whole idea with the Bank Seed Hub. It worked for what it was designed for, but as you guys know, many of these breeders are very hard to kind of get in with. They're very guarded. If I come with an opportunity like that from someone that they don't know and have enough business with before, it's going to be very skeptical in which it was. I had to develop relationships first with the people that I knew and had relationships with that I thought would be a good jumping off point. And from that, I knew that in the short term, it would quickly kind of start to snowball and grow. And I think if we had another year or two, we would have seen that really start to come around table, kind of coalition of breeders. What I envisioned and I called it was the Sunday Mass, which was the Mutual Admirations Society of Seedmakers. And we would all be together. Someday we would come together and we would be able to form this group where everybody is gaining in the same amount. We take away the competitive element a little bit, even the playing field. And then we could have a very real brick and mortar or some type of actual bank where all the brands represented, all the breeders are there. It's all 100% legal and compliant. We can do business with anyone we want to do business with. You know, that was the whole idea. I just wanted to say that we've heard different, heard of different people's attempts or initiatives to collectivize in different ways. Like so we've heard from Little Hill, Matt obviously with the Breeders Syndicate and now you with this. And it's interesting, like each of you has a slightly different take on collectivization, but it's super cool to see and hear about. Again, it's not common for Breeders to even get along to accomplish a goal like that. So it is something that takes years and years and years to build with trust. And yeah, you're well on your way. Everybody's got a little like bone to pick with everybody else it seems. Yeah. Myself included. Yeah, right. I mean, some others with you as well. Yeah. And you know, it is what it is, but things change quite a bit when you all are in the same room together. When you're all physically talking and have the same goal in mind or the same thing to gain.  That's what it should be about. So the last thing that we did with the Bank Seed Hub in the last year, things were pulled back quite a bit. We did a lot more from Cologne and we're running less and less of the seed. And so I did an acre hunt and I got like five or six friends that all had fenn seeds to throw in with me.  It didn't really work out the way I wanted with being able to produce some content, give ready on site. Everybody has different schedules. People had moved. It was harder for them to get back to California to kind of see the work. So I was documenting it all and trying to share it as much as possible. But that was like the beginning of the fruition of all the goals to have even half a dozen breeders that would throw their work in together. And then if I could have gotten them all together to that site, you know, then we're viewing each other's work and like making selections together and picking out the little nuances of what we liked about this cross or that cross or, you know, whatever. That was really what I envisioned. And we almost got there at the end.     So I unfortunately was let go from that position and had to, you know, do the new thing. So it was great while I lasted and the idea is still there. The dream is still there. I hope that I can continue to kind of see that mission through and exciting that where I'm at in Arizona they're giving me pretty much carte blanche to do any genetics I want we're doing massive seed hunts there. So I just filled a room with 2,500 plants all from seed. There's six different varieties. So we got a couple hundred of each one going. That's going to be the first like big phenohunt project on that site. And then I am doing a reversal in the greenhouse right now, which should net quite a few seeds. So really exciting projects. And then I hope to kind of be able to bring other folks back into that fold and continue the same idea and offer that space. Like now I have indoor space, which is one of the biggest complaints about the Bank Seed Hub was we were doing all outdoor. Nobody wanted to take pictures and content of their outdoor flower because for whatever reason they think it's bad or they can't sell seeds with it. But now I have the basically the same idea that I think I can replicate bring it inside and hopefully make, you know, a mutual admiration and a mutually beneficial group of people that would want to take play can take part in that. And for the company I'm working for, it's all good for them too because they're getting access to all these genetics and all these different things for their sales that, you know, they wouldn't otherwise have or they'd have to pay for or whatever. Absolutely. Yeah, you let us know if we can help in any way. You know, obviously we have a platform of sorts and have a certain kind of reach so you should just hit us up if you can think of anything that we can help with. Yeah, I definitely like to deliver completed ideas. So it's in the works. Let me continue to work on the mission right now and make sure that We want it now, bro. Yeah, yeah. I have to make sure that I can execute and be successful for everyone else, you know, before I can, before I can really just open up the floodgates. But that is the long term goal. So, you know, if anybody's interested in having those conversations, I certainly am. Very nice. I'm sure you're working, you know, your previous gig working on that scale probably forced you to systematize a lot, right? Oh, yeah. I always have, though. I'm, operations are a big, just part of my flow with anything with the part. So I'm very, yes. Yeah. Okay. Well, I thought we could kind of go backwards in time a bit and maybe you could talk to us about what the early period for you was like making season growing. You know, what drew you to those things? How did you get into it to begin with and how did you learn about those things? Well, so I'm from Ohio originally and in Ohio, the first grow kind of group that I was a part of, they were growing all from seed. So there was no clones, none of that was even happening. And it was all DJ Short, Sensi, just a bunch of old school stuff, White Widow, Blueberry, Super Silver Haze and with that, with that early sort of introduction like seeds were the first thing for me. You know, so when I ended up returning back to them around like 2010 or so, it was like a, it was like a homecoming. It was like, Oh, this is the way we were always supposed to do it. You know, we like kind of lost sight for a while. So those early days back in Ohio, all from seed, everything was selected in-house. And it was just because it was a very close group and didn't want anybody to know what the hell we were doing. Didn't have friends that had clones and probably wouldn't have, you know what I mean? Like was parked the car like two miles away and carrying grow equipment down the street. Seemed like a better idea than parking your car after Green Merchant, right?  So that was kind of the, you know, that was kind of the viewpoint and it was very, it was very stigmatized in Ohio in my area, even from my close friends who are now are totally cool that it was, it was still like, I was kind of off to the side on my own a little bit. So anyways, from there, I ended up, one of the guys who was part of that group who was really just more of a, he was more of a scammer kind of, he was more like just in it for the money part. He didn't, kind of guy. Yeah, he didn't even really smoke weed.   But that was his whole thing was just sort of like, I'm just trying to make money, you know, which was fine too, because it gave me a job. So I ended up moving out to the Bay Area to do some trim work for him and just help out because it was exploding. And at the time that the OG Pounds were 58, 57 down in LA.  So relocated from Ohio to California to do that. And everything was from clone. So from there, we're growing OG, snow cap, blue dream, Blackberry Kush, Nightshade, you know, some other random stuff and eventually that whittled down to like, you know, maybe 10% anything but OG, you know, and everything was OG all from clone and that was flow for a long time, you know, and everything was going to LA. And what year was this? This was, I moved out to California 2007. So right around there, 2007 through 2012 or so, that's pretty much what we were doing. And then what became pretty apparent, like, you know, right away was if we, it was always like, oh, we need the next OG cut, you know, now we need Skywalker, now we need King Louis, now we need this, now we need that. And it was like, well, we can just get this from like, we can just go backwards by the seed, make selections and have something totally unique that nobody else has. Yep.  That's all I see. It was splitting hairs between the cuts and a lot of people don't realize, like some of them, they could be the same cut with just two different names. So I get real boring. 90%, 90% of them were the same cuts and it was just little conditional differences from environment that would really change, you know, the appearance or what was expressing in the plant was more so based on the environmental changes based on these people's grow rooms and how they built them, which is kind of an interesting thing, like your signature is kind of in your grow room. It's kind of an interesting concept. Yeah. But yeah, it was just really boring. So I was like, fuck, I need these flavors. I was in love with the banana flavor. I had banana OG. I wanted to find that in a seed form. I couldn't get a cut anywhere. So I went to seeds and I think some of the first seeds that I bought when I was in California were off of seed bay, you know, started with some of that Dutch growing stuff, like the one that we talked about, that's the pre-roll that I gave you, that was that super number. I still talk about that. I still talk about it. When people ask about banana stuff, like, and I did a lot of banana work myself. So I've got a pretty good working knowledge of that line and how it breeds, but that was exceptional. And people ask me about banana flavors. I'm like, well, there was one very specific one that was the most extreme banana bread Terps that it was just unmistakable. That would smell bad. And that was like a seven-weeker. It was like, and the Ogre 99 was made probably years prior and probably not kept in great condition. So by the time I even got those seeds, it was like, they were probably, I don't know, five years old, six years old. I don't know when she made them, but they were old and only two germinated. So the one was a female, one was a male. He has 11.  Perfect. I don't love the odds on that. I know, I know not only that, but then I used the male because I'm like, well, there might be some banana in this guy too. So I used the male in my earliest breeding work just to try to continue that. And he put out amazing work. I mean, that's still the backbone of what I grow today and what I breed today in those lines is still rooted in that Ogre 99 cut and the male from that pack. Those are still like the basuses. And then I had some, what was the other one? Sweet tooth number three from, I think deluxe had done a remake or something.  Oh yeah. Oh my God, that plant was so amazing. It was like nobody, the flavor was just unbelievable with the tangerine. NYCD G13 haze? Yep, exactly.  And everybody who smoked it who was looking for like a downer, couch-lucky weed, hated it because it didn't do that. But I had a very specific experience where I was down in LA vending. I'd always go to Malibu to first point to go surfing. I had smoked some OG with my buddy.    And I was like, this is the fucking, like this is more powerful than that OG. It's just a completely different effect that you're not feeling from like a sober to high state. But it's like, if I was already high and too low, man, that thing brought me right back.  So that was, that was a magical one, but I used the sweet tooth to try to like preserve that flavor as much as possible. So those were the two earliest breeding project. I did really was using the banana ogre and I mail and I made F twos, but I did all sorts of other work with him and then trying to preserve the tangerine haze as much as possible by using the sweet tooth F two, the sweet tooth number three F two, which I don't know how many times that's in crossed, but that was pretty much the earliest stuff and the results were fantastic.  And then, and then I got a little pushback on them. That's when they stop liking seeds as much as when they see the entire room. That's exactly when they stop liking seeds so much. At that point they got referred to as art projects. And I was no longer. Which is honestly what ended up being the impetus for me leaving that outfit and starting my own thing was based 100% on that. I was like, well, this is the ticket. Like we got to keep doing this. And they're like, no, we got to do none of that and just go back to growing blue dream. And I was like, all right, this I have a different vision. I have a different vision than what I do. So that's when I kind of broke off on my own and left, you know, the safe haven of a very tight knit friends group that I had and had moved across the country for and everybody that was working with us at the time was from Ohio for the most part. So that was sort of the impetus for leaving that group was the souring of popping seeds for them. You know, not for me collateral damage, man. We just ended up with more seeds. Yeah. I mean, she didn't even complain. That's a, that's a plug. Right. Right. I wanted to take a little tangent here and want to throw this bone to Matt as well, which is the forums you mentioned, I think, in one of in that interview in Skunk Mag that the forums played quite a big part in your early stages. Definitely. And a great, great point to mention or a great place to mention that.  Oh my God. So when he went to jail, all the guys in jail called him cool white because of the lights. I like that. So anyway, he'll probably love that. Shout out cool white. Super ego maniacal guy. So he'll definitely enjoy that. Oh yeah. Here's his name for sure. Yeah. Yeah. But so he was on overgrow. He was big on overgrow.  So I can only imagine how bad he was on overgrow. But I was on IC Mag mostly because of the seed bay linkage. And that's just where I was buying seeds. So I was super paranoid still, you know, the same mentality had followed me through and was a lurker.  That wasn't my regular IP address. Go even surf it and look around on IC Mag. But yeah, watching some of the threads on IC Mag, especially the Jack Mayhoff or stuff, which I think is Ivan, right? Yes. And it was like the early cocoa beds that they were doing. And you're referring to Jungle Boys Ivan for the people who don't know. Right. Yeah. Jack Mayhoffer. So Jack Mayhoffer's threads were epic as far as just like scale growing and what they were doing now, or what they were doing then, they don't do it all now. But my idea and sort of what I was following was fully soil organic growing, living soil. And so his idea of these giant beds in your room was just like right in tune with what I was trying to do. So I use the tech that he was kind of providing on IC Mag and all the other information about nutrition and living soil and all the other stuff that was available on there and really just ran with it. But it gave me a lot of confidence to kind of know what to try and learn from other people's failures, especially. And that was quintessential. And then so much stray knowledge. And now I really, I haven't gone on there in so long, but just so much information. And the thing was that all the bullshitters got called out. That was the great thing. We don't have that anymore. Like Instagram is such a step backwards from the sharing of information because. Agreed. And I don't think Twitter does it either. I'm not on Twitter, but I would imagine that doesn't do it either. But the power is in the words, the words or the knowledge, the pictures support the words, but the words are where it's at on Instagram. It's totally different. It's just the facade. Here's what it looks like. The words are almost meaningless. You throw a couple of emojis up there. And then, you know, Q conversations that are also meaningless, which is usually just a circle. Instagram for me is kind of just such a huge step back. It's a necessary platform. I think it's the only one that we really have, but it's just such a step backwards as far as like sharing information and sharing reality because none of it is steeped in reality whatsoever. I think, I think that's a common sentiment too. Among people who've been around and made it through the forum era and stuff, being able to have that in that kind of community and then going to Instagram, it's just totally, it's a night and day difference. And it's actually pretty depressing. Yeah, it is. I mean, I just appreciate it was policed a lot better. Yeah. You know, and I don't know. It was just, it was just, it was generally really solid information. You can kind of tell what wasn't and what was, but generally really solid.  Yeah, all right. So they're like, I see mag was really, really awesome. I really enjoyed it. And the first kind of the first few people that I met up with and you know, it became like in real life friends with were from IC mag too.  So I'm super grateful for that form and all the forms that proceeded it as well. Yeah. Are you going to jump on the can of Kibana? Now that it's all opened up. I will. Yes. Unfortunately, time is so restrictive for me right now between the job and the kids that it's just, I just have the time. Yeah. I don't have the time to do it. I don't even have time to smoke as much as I'd like, honestly, or a place to do it. I'm in my kids room right now.  It's quiet and not in the garage looking depressing. Yeah. So I just don't have the time for it right now. I wish I did, but I'm so, I mean, I'm in my own forum world, so to speak in the, in what we're doing because the 100,000 square foot is perpetual. I've got nine different grow rooms that are all going at the same time. And they're all a week apart. So I can see very much the entire like week one through nine of what's happening in a room at all times, which is really cool to just try to digress and, and kind of make my new changes and see the results of those types of things. So I just get so much value out of that that I don't really go to the forums quite as much anymore. It was really a little bit of a crotch, a lot of information. And now it's because it's, there's so little activity there. It just wouldn't be as valuable for the things I would probably use it for today, which would be more like verification of cuts and, you know, provenance and things like that. And we have our discord too, where we can all jam on that, which is really nice. I mean, that's been, that's been a great thing. Honestly, the discord has been awesome. And I've, and that's been the main type of forum thing that I've interacted with in the last six years. I think I've been on that thing now with. Yeah. Yeah. I remember messaging, messaging Denali the night I was in the hospital and he just turned five. So. That's wild. Wild. Yeah. Wait, so did you two cross over on the forums? No. No, we met at Emerald Cup in 2017. Yeah. So I remember exactly how we started talking. You had said something on a post. I posted something at the time I had, my company was called the moody. And I was running that Instagram. And your bio had a stone roses. Yeah. And I love the stone roses. And so few people know about the stone roses that I was like, he's already in me.  The first time I sent you something, I sent it to cinnamon girl. Oh, that's right. So wild. Yeah, dude. Yeah. Nobody even listens to them anymore. Yeah. They're the best man. Music and weed. Those are the unifiers. Those are the things for me that I've always like brought people and made friends has been like, if there's two ice breakers for me, it's music and weed. And those have always been true even to this day. Yeah. For sure. For sure. I still use that quote. I think it's in my profile still to this day. I figured you were cool when I saw that man. It creeps people out though. Like, cause they don't get that it's a quote from stone roses song. It just sounds like I'm an evil Satanist. Even though. I can. I can hang with it. Anything else on the forums? No, I enjoy. I wish I would have done more. Honestly. I was more of a lurker. I wish I would have interacted more with people and I wish they were still as used as they are today. Yeah. I mean, going back to what you guys are talking about between say the forums and Instagram or Twitter, the architecture, you know, obviously lends itself to different things. So on Instagram and Twitter, it's very hard to sustain a conversation for more than a few days, right? Because it just disappears into the timeline, the abyss of noise. And you're so limited on what you can say. It's not like you can really express yourself very well, kind of like Twitter. You know, you're limited to a certain amount of character sets and people don't want to read long sentences. So you'll find that most people just respond to the first few things you say or the image and not read what you wrote. So there's a lot of miscommunication that goes on. Yeah.  Yeah. It's just about like your reflexive knee-jerk reactions. Yeah. All the time. I definitely agree with you. I definitely enjoyed the linear nature of the forums because I would wake up in the morning, go back to wherever I left off, catch up on all that, you know, the chit chat, whatever was going on, and just follow along with the conversations. So I enjoyed the linear nature. Once it got, it got ridiculous to a point.  I wake up or at like 600 and something.  I can't even catch up. So I guess there's a downside to it too. When it gets too much use. Yeah. But yeah, I missed the forums, man. I really liked that. That was a good, that was a good time. San Diego's finest got so big, they had to blow it up into like a second and third, I think now it might just still be on second. But yeah, that was one of my favorites. Yeah. When threads become their own forums. Yeah. Right. And they walk it at a point. Yep. Well, I thought we could kind of segue to Pico's.  You talked about the banana stuff. Can you tell us a bit more about some of your own recent work? You know, nightcap, hazes. Yeah. So, um, so from that initial project with the older 99, I crossed out to Blackberry Kush, which was a staple for us at the time. And from those found some really cool plants. I called it Gargamel. It was an interesting plant. Nothing like too crazy, but it had a very, very good look, got great color, had good flavor. It was kind of like pretzel dough with like a hint of licorice on the back end, which I really, really liked. And then, um, I ended up getting a cut from a buddy that was supposed to be Gorilla Glue 4.  That's not Gorilla Glue 4.  And it might be better than Gorilla Glue 4. Cause like, I never even experienced anything like this. That one, we ended up taking it to market before I knew it was not Gorilla Glue 4. We took it to market and, you know, taking it to like Oakland and they were like, nah, that's the saying it. And their best guess at what it was, was, um, Grease Monkey, which was Gorilla Glue 4 and something else. I forget what the cross was on it. Yeah, I think it was a exotic deck. Yeah, that one, I remember that one. It was exotic. Yeah. Yeah. It was exotic genetics. We ended up taking that name and calling it Grease Monkey 2.0. We didn't, we couldn't confirm it was or wasn't. Sure. The guy who, uh, but if that was what they wanted to call it and that was what it was, you know, we could sell it to the club because it was super dank.  Um, I ended up making some S ones of that, which may have been an S one already because it was the fake Gorilla Glue.  Um, but the S one of that that I popped, I popped one seed. Every time I'd make something that I just throw one or two, you know, in, in just to kind of see and get a look under the hood. And this plant pops and it was like the, the first set of true leaves after Carl Eden was like covered in trichomes, like covered in the, the Cecile type, you know, the type without the glandular stock and, um, and it was glistening and it was like amazing. And I could smell it right out. There was like fresh pine and coriander was unbelievable. The plant continued to grow up in veg and it just kept growing in these trichomes on every single leaf. Like the fan leaf is covered in triches. I'd never seen that before. It was so fragrant.  It was just meant to be. Um, so that's what ended up becoming boss frost for us. Boss, boss frost like ruled the day for us for a minute. We went to the 2016 high times cup and that was the only thing I've ever entered into a cup. It hit 6.6 on the Terps and 33 on the total can. It blew everything else out of the water. The, the NorCal cup was, um, the lab was steep hill. I did the NorCal cup and they were, they were very, very rigid. Um, and so they had to test it multiple times even to, to, to confirm that that's what it had tested at, but that's what they ended up finally releasing for the test results.  I was very disappointed. I thought for sure because of just the test results and everything else that it was going to kill it. It didn't really matter. Um, but that crossed into the gargamel ended up being nightcap. And again, I just did a small seed run, um, you know, like one plant, a couple hundred seeds, nothing big, threw two of them into pots just to see what was up right away. And one of those two became nightcap. I never even went back to the seeds again. Nightcap was better than boss frost. Somehow it was like danker, more earthy, heavier. You know, it, boss frost was like very, um, ground coriander, really bright, like really bright flavors. And the nightcap was just way, way danker. And, um, that one, we ended up hitting 8.1% terps on super high in alpha pining, but does have a backbone and mere scene too. It's like kind of one to, or two to one alpha pining to mere scene in that. And, um, was the terp jam for SC labs for a couple of years till I think, um, those guys up at the ridge, ridge farm, ridge something, they, um, they ended up usurping me with a couple like 10 sub point, you know, on that. And at the time we were putting out in the living soil, we were putting out amazing stuff, our own genetics. I guess my own genetics, although I had some help. Um, and everything was killing it in the Terps from SC labs. Like boss frost, I never saw it go below four and a half. And that was on like the shittiest, shittiest grow. And nightcap was always just killing it too. So the cool thing about nightcap is that, um, the terpene profile, this is where I started really getting into terpene profiles, like around 2014, 2015, when I started my own outfit and was doing this living soil and everything else, I opted in to lab testing immediately because I wanted the data, right? So I had to pick the best labs. I would use Steve Hill or SC and I just wanted to keep one, you know, keep it as simple as possible. So that I would get as true a data set as possible, um, through all the different things we were doing in the grow, trying out different things and running different strains and whatever.  It would take me weeks to get my terp test back because they kept having to retest it to confirm that these numbers were accurate enough for them to put out. And at the time, SC labs was, was, um, making public all their test results. If you didn't ask for it to be private, they would make a public and you could go and you could scrape all the data off of their website and you could pull tens of thousands of tests. So after that, and I had a pretty decent relationship as like a client to Alec Dixon, one of the founders of SC labs, and he would shout us out all the time and we had a lot of conversations because my test would take so long to get back. I would always be calling him like, what the hell's going on? And we would have a lot of conversations and he was just like, you know, really impressed with the amount of terps we were getting off all the strains that we were growing, especially these unique lines. And, um, and what he shared with me, and we could see from the data was that all of the cannabis that was tested was falling into these like seven categories, you know, now that's what the Emerald Cup is using as their groupings for it to get away from the sativa and indica designation. And they're using these, you know, I forget what the words are, dessert, something, whatever the words are kind of tacky, but the exotic dessert dreams. Yeah. Sweets and dreams. Yeah. Right. So the, I mean, they're trying to simplify it and that's a corporate move, I'm sure, but regardless of that, the idea that we now had, we take this vast super plastic genome plant that can do so many different things and we make it finite, right? And we have seven categories now that like almost all the week fits into these seven categories. That became super interesting. And my idea was that if I play within these categories, then I will be able to breed homogenous strains much, much quicker because I'm going to be replicating this chemo type over and over and over again. And so at least the expression of the chemo type is going to be the same. Now there was a lot of misinformation about the like integral part of terpenes on flavors, you know, which I kind of could tell right away because of that banana 99 cut, I was searching like, okay, what's the terpene that's giving us this banana flavor, right? And it's not at all. It's not. It's amyl acetate. It's an ester. You know, same thing you see in alcohol production when you drink, and you get that banana. That's amyl acetate. And I had a background in brewing beer as well, just for fun, but that's how I knew about that ester.  But the idea was the same that I follow within these terpene categories and do my breeding work within those to add or take away certain traits that I wanted and keeping other ones intact, mainly the flavor. And what I knew for the terpene profiles is that they would have a similar effect for me anyway. So I would what was interesting about the nightcap is that it's the same flavor as all the non-terpene hazes or the PIF. Yeah. PIF was something I was searching after, you know, as much as I possibly could at that time. And again, this is like 2015. Nobody was circulating cuts. You couldn't nobody had to keep black.   There was some work here or there. But I ended up going to JJ. I got a bunch of his stuff at one of the Emerald Cups or one of the high time shows and started running through all his stuff. That was supposed to be, you know, as close to that as possible. He got a mango cut that was super incency from somebody else on the IC forum. I forget the dude's name and made a bunch of stuff. I ended up finding some really nice, very incency non-terpene haze expressions from his stuff. Use that, bread it down a couple generations to get what I wanted out of it. And then I paired that to the nightcap because it had the same terpene profile. I was able to basically take the flowering time and reduce it tremendously, increase the bag appeal and the actual like density of the bud from what the haze that I was growing at the time was and, um, and increase the increase the terpenes and the cannabinoids by like a shitload. So it was basically allowing me to have sort of a secret weapon in nightcap to breed hazes and shorten their flowering time and do everything, but it did affect the high, you know, the effect was affected. The flavor profile stayed the same, but the effect was affected. So, um, nightcap was super influential in with my work. And from that line breeding with JJ's into the nightcap, I made nightpiff, which is like a true eight week flowering, you know, haze type. The effect is a little stronger, but it's still very heavy. It's very up. Um, and it's incense is held when it, when it burns in the room and it just hit all the traits that I wanted right away. Um, so that was kind of some of the idea of, well, one, that was the nightcap and like what the nightcap has become. Now I've worked that down several generations since then. I have a nightcap, exotic align where I added an exotic in. Um, at the time that I made it, I didn't want to sort of piggyback off of the success of cookies and what they had done. But now with the cats out of the bag, because other people had already leaked the lineage. So it is nightcap Gary Payton. And then from there, I bred that into another exotic, which I will not. Actually, well, that one was the purple cast, which nobody can get anyway, because that was a chesedero strain. Um, that I was gifted by like a family that wouldn't let anybody else have it. So that's the nightcap, exotic of season two. I continue to do to do that line. It's really high pining. Um, it's the pining. The other thing that's interesting about the pining is that the effect for me for high pining strains is very, um, it's very lucid. I don't get, I don't get the inner monologue.   Yeah.  And, and it's also extremely potent. And, um, the reason I called nightcap nightcap is because that terpene profile for me with the potency, with the high THC, allows me to drift off to sleep without any inner monologue or like my voice inside going, which is one of the things that gives me paranoia and stops me from falling asleep. So for me, it was super easy on the nightcap to fall asleep because that terpene profile, it allows a very smooth transition from sober to high or, you know, whatever you want to call that. Yeah.  I'm too high, you know, it's like, I can maintain the same train of thought. Um, and so at the time that this was 2016, I think that that one popped and we were doing a lot of work still with medical patients and they were giving me the best feedback that they were able to sleep through the night, didn't have to get up and go to the bathroom, getting restful sleep, which if you're ailing from anything, like getting restful sleep is like one of the two best things you can possibly do, regardless of anything else that you're taking or doing. So, um, talking to cancer patients and talking to people with true medical need, who were impacted by that plan was, uh, and still is like a highlight for me. Super amazing. So my cap's been on its journey. I continue to use her today. I continue to breed her today and she's a magnificent plant. Again, it was two seeds. So recently on my own discord, I auctioned off the, the half of the remainder of those seeds that were left and some lucky gentlemen picked them up. That's cool. Yeah. That's really cool. I kept half. I need that. But, you know, I just with her in the stable, like there's almost no need to go back in. And the, and there was quite a bit of variance even in the two that I popped. It was like one was super squat. And the one that I kept was really tall, lanky and had more of what I was looking for. Um, so yeah, I'm happy to let some of those things go. And I have so many seeds now. It's like, what's, you know, if I'm not going to get around to it, let's let somebody else grow it out. That's about where I've gotten with a lot of my collection too. After a while, you just start to realize like these things do have a time limit. And, and I want to see him grown out. It's, it's, you know what I mean? Like it's either you get greedy about it and let him die with you or you get him out there and be able to get data points at least. Yeah.  It's a perishable good. It's like, right?  When I see people with these mat, like flexing with their massive collection, it's like, grow them. What are you doing? Yeah. You know what I also learned too on the forums, the people who had posted their massive collections were always the first to get shipped down and like scammed. And it was just setting yourself up so bad, so bad to do stuff like that. Yeah. So it's a very FOMO driven. Yeah. I'm guilty of it too.  Like I early on, I think that's what we saw was cool.  So of course we'd follow suit as younger guys. Yeah. Well, the endorphin, the endorphin experience is like in the purchase and then receiving the package. And then after that it's over. So it's like, I gotta do it again. I need more seeds. Yeah. Yeah. I have these now. I don't need these anymore. And the flex. I honestly, the psychology of it is really interesting to me because seeds themselves are obviously not plants and they're not flower. They're not end products. They're kind of promises of something else. I love that. And then when you, even the purchase of a seed is like a promise of a promise and it's like these little gambles that you're kind of getting high on really. Yep. That's great. When we do this massive hunts now and I'm, and I'm talking to like the crew that's working under me. That's, I never have used the term promise, but I'm going to now. But I always say, Yeah. If you let one of these seeds not, if you let it dampen off or you don't, you know what I mean? It's, you're not like growing the seed properly or doing justice. If you lose seeds in a germination run, it's like you might have just lost Michael Jordan. Yep.  You know what I'm saying? You can't lose Michael Jordan, man. You got to see all those to their true potential to know. And that's the FOMO for me now is like, if I lose a plant or a plant is, I'm like, it's not doing as well as it should. I'm not going to see its true potential. That's like where the FOMO is for me now. And I'm just like, I need, they have to be perfect. Yeah. For sure. Yeah. You could be missing that blue, that beautiful blue. It's funny because I found, I found really great things and really small runs. I found really, really amazing things in massive runs. And, you know, the potential is there. And that's, that's the thing I don't want to lose is the potential. I was just trolling my buddy here who is popping some blue. I exceed and didn't get a wall. I was going to mention it. I was going to mention it. All right. And that's going to be our show for today. I wanted to thank Pico for coming on. A thousand of course for always helping setting things up, being there with me to ask some of the harder questions that I'm not really good at. I'm only good at asking bringing stuff. Also want you to go check out our Patreon. That's how you get access to our Discord. That's where we hang out all the time. We've been doing more live chats. Actually, I'm going to go run one today. It's really nice. We get to interact with the whole community. Everybody gets to put their input on what they want to see in shows. Gives us ideas for different topics because there's, you know, I'm running out of ideas over time. You know what I mean? But I am trying to take it back to more breeding centric type show. So expect that. Definitely expect that. We have riotseeds.com with our seeds and spray. We have the diesel drop, the diesel hybrids. We have the blue dream drop from goat farm. We're going to have some new influx of stuff from high and lonesome, I believe, some mango haze hybrids coming up shortly. Going through testing right now. You can also find our seeds at riotseed co-Europe for European people in central Europe. We have GERT by seeds for our people in the southern hemisphere of Australia and New Zealand. And then we have lifted seeds also here in the U.S. but they can take like direct credit card payment stuff and they're awesome people. They're doing a great job of carrying extremely good breeders. Not every seed bank cares about that. Even if they say they do, I don't know any other ones. I really truly don't. So with that, just thanks for showing up. Thanks for caring.  There's been a lot of times where it's almost ended. So Bowsen deserves a lot more credit than I think a lot of people have been giving him in the comments. Because without him, the show would have been dead after the whole explosion. You know what I mean? Not everybody knows about it. I don't want to go into too much detail. But we talked about it in the Patreon crowd with everyone as it was going down. There's a reason that co-hosts change. It was necessary. We hold each other to high standards. We absolutely do. And sometimes life gets hard. People slip up. Things change. Morals change. Moral fluidity as we talk about it.  And just know that it's a good group of dudes all holding each other to really high standards. And if someone falls back, we put them aside for a while. So with that, we'll see you next week for some more breeding chat with Breeder Syndicate. Are you able with the Syndicate? Check out our Patreon and our link tree or description below. Our merch site is officially live. We have all sorts of shirts, hoodies and goodies to sort you out and shipping is super fast. And most importantly, the quality is top notch. I've been saving old designs for years for this purpose, so please check it out. SyndicateHere.com. We also have an underground Syndicate Discord where we get together and solve old strain history together daily. We're a community of learning away from IG and it's an amazing resource for old catalogs and knowledge. We hope you join our union of breeders and growers. Come check it out.