I'm sorry, this is a very technical talk in a very sleepy talk slot, so if you fall asleep in the middle, I will be super offended, but I won't call you on it too hard. So yeah, I'm Stella Cotton, if you don't know me. I'm an engineer at Heroku, and today we're going to talk about distributed tracing. So before we get started, a couple of housekeeping notes. I'll tweet out a link to my slides afterwards, so they'll be on the internet. So there'll be some code samples and some links, so you'll be able to check that out if you want to take a closer look. And then I also have a favor. If you have seen me speak before, I have probably asked you this favor. So Ruby Karaoke last night, anybody go? Yeah, totally destroyed my voice. So I'm going to need to drink some water, but otherwise I get really awkward, and I don't like to do that. So to fill the silence, I'm going to ask you to do something that my friend Lillie Shalene came up with, which is each time you take a drink of water, just start clapping and cheering. All right, so we're going to try this out. I'm going to do this, yeah? All right, so hopefully that happens a lot during this talk so that I won't lose my voice. So back to distributed tracing, I work on a tools team at Heroku, and we've been working on implementing distributed tracing for our internal services there. And normally I don't do this whole Brady bunch team thing with the photos, but I just wanted to acknowledge that a lot of the trial and the error and the discovery that went into this talk was really a team effort across my entire team. So the basics of distributed tracing, who knows what distributed tracing is? Okay, okay, cool. Who has it at their company right now? Aw, I see you, Herokai. So if you don't actually know what it is or you're not really sure how you would implement it, that's, you're in the right place. This is the right talk for you. It's basically just the ability to trace a request across distributed system boundaries. And so you might think like, Stella, we're Rails developers, this is not distributed systems conference, this is not Scala or Strangeloop, you should go to those. But really there's this idea of a distributed system, which is just a collection of independent computers that appear to a user to act as a single coherent system. And so if a user loads your website and more than one service does some work to render that request, you actually have a distributed system. And technically, because somebody will definitely will actually me this, if you have a database and a Rails app, that's actually technically a distributed system. But to simplify things, I'm really going to talk more about just the application layer today. So a simple use case for distributed tracing.  Monolithic architecture, you got one web process or multiple web processes, but they're all running the same kind of code, and they're going to return information. Users, orders, users have many orders, the orders have many items. Very simple Rails app. We authenticate our user, our controller, going to grab all of the orders, all of the items, render it on a page. Not a big deal. Single app, single process. Now we're going to add some more requirements. We got a mobile app or two. So they need authentication. So suddenly it's just a little more complicated. There's a team dedicated authentication, so now we maybe have an authentication service. And they don't care at all about orders. So it makes sense. They don't need to know about your stuff. You don't need to know about theirs. So it could be a separate Rails app on the same server, or it could be on a different server altogether. It's going to keep getting more complicated. Now we want to show recommendations based on past purchases.  So naturally the answer, microservices, obviously. But I mean, seriously, it might be services. So your engineering team, your products grow. You don't have to have this microservices bandwagon to find yourself supporting multiple services. Maybe one is written in a different language. It might have its own infrastructure needs. Like for example, our recommendation engine. And as our web apps and our teams grow larger. These services that you maintain might begin to look less and less like a very consistent garden and just more like a collection of different plants and different kinds of pots. And so where does distributed tracing fit into this big picture? So one day, e-commerce app, you go to your website, it starts loading very, very slowly. And if you're going to look in your application performance monitoring like New Relic or Skylight or use a profiling tool, you can see recommendation services taking a really long time to load. But with these single process monitoring tools, all of the services that you own in your system or that your company owns are going to look just like third party API calls. You're getting as much information about their latency as you would about Stripe or GitHub or whoever you're calling out to. And so from that user's perspective, you know there's 500 extra milliseconds to get the recommendations. But you don't really know why without reaching out to the recommendations team, figuring out what kind of profiling tools they use for Python who knows and digging into their services. And it's just more and more complicated as your system is more and more complicated. And at the end of the day, you cannot tell a coherent macro story about your application by monitoring these individual processes.  So what can we do to increase our visibility into the system and tell that macro level story? Stripe to tracing, that can help. It's a way of commoditizing knowledge. Adrian Cole, who's one of the Zipkin maintainers, he talks about how in increasingly complex systems, you want to give everyone tools to understand this whole system as a whole without having to rely on these experts. So cool, you're on board, I convinced you, you need this, or it at least makes sense. But what might actually be stopping you from implementing this at your company? A few different things that make it tough to go from this theory to the practice with distributed tracing. And first and foremost is that it's kind of outside the Ruby wheelhouse. It's not represented, Ruby is not represented in the ecosystem at large. Most people are working in Go or Java or Python. You're not going to find a lot of sample apps or implementations that are written in Ruby. There's also a lot of domain-specific vocabulary that goes into distributed tracing, so reading through the docs can feel pretty slow. And finally, the most difficult hurdle of all is that the ecosystem is extremely fractured. It's changing constantly because it's about tracing everything everywhere, across frameworks, across languages, and it needs to support everything. So navigating the solutions that are out there and figuring out which ones are right for you is not a trivial task. So we're going to work on how to get past some of these hurdles today. We're going to start by talking about the theory, which will help you get comfortable with the fundamentals. And then we'll cover a checklist for evaluating distributed tracing systems. Yeah. All right. So start with the basics. Black box tracing.  So an example of black box tracing would be capturing and logging all of the traffic that comes in and out at a lower level in your application, like at your TCP layer. All of that data, it goes into a single log, it's aggregated. And then with the power of statistics, you just kind of get to magically understand the behavior of your system based on timestamps. But I'm not going to talk a lot about black box tracing today, because for us at Heroku, it was not a great fit. And it's not a great fit for a lot of companies for a couple of reasons. One is that you need a lot of data to get accuracy based on statistical inference. And because it uses statistical analysis, it can have some delays returning results. But the biggest problem is that in an event driven system, so sidekick or a multi-threaded system, you can't guarantee causality. And what does that mean exactly? So this is sort of an arbitrary code example, but it helps to show that if you have service one, kicks off an async job, and then immediately synchronously calls out to service two. There's no delay in your queue. Your timestamps are going to correlate correctly. Service one, async job, awesome. But if you start getting queuing delays and latency, then a timestamp might actually make it consistently look like your second service is making that call. So white box tracing is a tool that people use to help get around that problem. It assumes that you have an understanding of the system. You can actually change your system. So how do we understand this path that our request makes to our system? We explicitly include information about where it came from using something called metadata propagation, and that is a type of white box tracing. It's just a fancy way of saying that we can change our Rails apps or any kind of app to explicitly pass along information so that you have an explicit trail of how things go. And finally, another benefit of white box tracing is real-time analysis. It can be almost real-time to get results. Very short history, metadata propagation. So the example that everyone talks about when they talk about metadata propagation is Dapper, and the open source library that it inspired called Zipkin. So Dapper Paper is published by Google in 2010, but it's not actually the first distributed systems debugging tool to be built. And so why is Dapper so influential? Well, honestly, it's because in contrast to all of these other systems that came before it, those papers were published pretty early in their development. But Google published this paper after it had been running in production at Google Scale for many, many years. And so they're not only able to say that it's viable at a scale, like Google Scale, but also that it was valuable. And so next comes Zipkin. And that's a project that was started at Twitter during their very first hack week. And their goal was to implement Dapper. And they open sourced it in 2012 and is currently maintained by Adrian Cole, who is not actually a Twitter anymore. He's at Pivotal and he spends most of his time working in the distributed tracing ecosystem. So from here on out, when I use the term distributed tracing, I'm gonna talk about Dapper and Zipkin like systems because white box metadata propagation distributed tracing systems is not quite as a Zipkin. And if you wanna read more about things beyond just metadata propagation, there's a pretty cool paper that gives an overview about tracing distributed systems beyond this. So how do we actually do this? I'm gonna walk us through a few main components that power most systems that are of this caliber. So first is the tracer. It's the instrumentation that you actually install in your application itself. There's the transport component, which takes that data that they collect and sends it over to the distributed tracing collector. That's a separate app that runs. It processes, it stores the data, and it stores it in the storage component. And then finally, there's a UI component that's typically running inside of that that allows you to view your tracing data. So we'll talk first by the level closest to your application itself. That's the tracer. It's how you trace individual requests. And it lives inside your application. In the Ruby world, it's installed as a gem, just like any other performance monitoring agent that would monitor a single process. And a tracer's job is to record data from each system so that we can tell a full story about your request. You can think of the entire story of a single request lifecycle as a trace. This whole system here captured in a single trace. Next vocab word, span. Within a single trace are many spans. It's a chapter in that story. So in this case, our e-commerce app, calling out to the order service and getting a response back, that's a single span. In fact, any discrete piece of work can be captured by a span. It doesn't have to be network requests. So if we want to start mapping out the system, what kind of information are we going to start passing along? So you could start by just doing a request ID so that you know that every single path that this took through. You query your logs and you could see that's all one request. You're going to have the same issue they have with black box tracing. You can't guarantee causality just based on the timestamps. So you need to explicitly create a relationship between each of these components, and a really good way to do this is with a parent child relationship. The first request in the system doesn't have a parent, because somebody's just clicked a button loading a website. So we know that's at the top of the tree. And then when your auth process talks to the e-commerce process, it's going to modify the request headers to pass along just a randomly generated ID as a parent ID. Here it's set to one, but it could really be anything. And it keeps going on and on with each request. So trace is ultimately made up of many of these parent child relationships. And it forms what's called a directed acyclic graph. And by tying all of these things together, we're able to actually not just understand this as an image, but with a data structure. And so we'll actually talk in a few minutes about how the tracer actually accomplishes that in our code. So we've got our relationships. If that's all we wanted to know, we could stop there. But that's not really going to help us in the long term with debugging. Ultimately, we want to know more about timing information. And we can use annotations to make a more rich ecosystem of information around these requests. By explicitly annotating with timestamps, when each of these things recur in the cycle, we can begin to understand latency. And hopefully you're not seeing a second of latency between every event. And these would definitely not be in user-readable timestamps. But this is just an example. So zoom in to our auth process and how it talks to the e-commerce process. So in addition to passing along the trace ID, parent ID, child span, we'll also annotate the request with a tag and a timestamp. And by having our auth app annotate that it's sending the request and our e-commerce app annotate that it received the request, this will actually give you the network latency between the two. So if you see a lot of requests queuing up, you would see that time go up. And on the other hand, you can compare two timestamps between the server receiving and the server sending back the information. And you would be able to see if your app is getting very slow, you'll see latency increase between those two things. And then finally, you're able to close out that full cycle by indicating that the client has received that final request. Let's talk about what happens to that data. Each process is gonna send information via the transport layer to a separate application that's gonna aggregate that data and do a bunch of stuff to it. So how does that process not add latency to your system? First, it's only gonna propagate those IDs end band by adding information to your headers. Then it's gonna gather that data and just report it out of band to a collector. And that's what actually does the processing in the storing. For example, Zipkin is gonna use Sucker Punch to make a threaded async call out to the Zipkin server. And this is gonna be similar to things that you would see in metrics like Librado, any of your logging and metric systems that use threads. So our data collected by the tracer, transported via the transportation layer, collected, finally ready to be viewed in the UI. So this graph that we're viewing here is a good way to understand how the request travels, but it's not actually good at helping us understand latency. Or even understand the relationship between calls within systems. So we're gonna use Gantt charts or Swimlanes instead. So the opentracing.io documentation has a request tree similar to ours. And looking at it in this format, you'll actually be able to see each of the different services in the same way that we did before. But now we're able to better visualize how much time is spent in each sub-request. And how much time that takes relative to the other requests. You can also, like I mentioned earlier, instrument and visualize internal traces that are happening inside a service, not just service to service communication. Here you can see billing service is being blocked by the authorization service. You can also see that we have a threaded or parallel job execution inside the resource allocation service. And if there started to be a widening gap between these two adjacent services, it could mean that there's network request queuing. I still can't help myself, but like saying and do a little dance when I do that. All right, we know what we want, how are we gonna get it done? So at the minimum, we wanna record information when a request comes in. And when a request goes out. How do we do that programmatically in Ruby? Usually with the power of rack metalware. If you're running a Ruby app, the odds are that you are also running a rack app. It's a common interface between for servers and applications to talk to each other. Sinatra Rails both use it. It serves as a single entry and exit point for client requests that are coming in the system. The powerful thing about rack is that it's very easy to add metalware. So that can sit between your server and your application and allow you to customize these requests. Basic rack app, if you're not familiar with it, Ruby object. It's gonna respond, call, takes one argument. And in the end returns status headers body. That's the basic of the rack app. And under the hood, Rails and Sinatra are doing this. And the middleware format is a very similar structure. It's going to accept an app. Could be your app itself or another set of middleware. Respond to call, needs to call, app.call at the end so it keeps following down the tree. And at the end return the response. So if we wanted to do some tracing inside of our middleware, what might that method look like? So like we talked about earlier, we're gonna wanna start a new span on every request. It's gonna record that it received the request with a server received annotation like we talked about earlier. It's gonna yield to our rack app to make sure that it executes in the next step in the chain and is actually gonna run your code. And then it returns back that the server has sent information back to the client. So this is just pseudo code. This is not actually a running tracer. But Zipkin has a really great implementation that you can check out online. So then we could just tell our application, use our middleware to instrument our request. And you're never gonna wanna sample every single request that comes in because that is crazy and overkill when you have a lot of traffic. So tracing solutions will typically ask you to configure a sample rate. We got our request coming in. But in order to generate that big relationship tree that we saw earlier, we're also gonna need to continue to record information when our request leaves our system. So these can be request external APIs like Stripe, GitHub, whatever. But if you control that next service that it's talking to, you can keep building up this chain. And we can do that with more middleware. If you use an HTTP client that supports middleware like Faraday or Xcon, you can easily incorporate tracing into the client. I'll use Faraday as an example because it has a pretty similar pattern to rack. So match the method signature just like we did with rack. And honestly Faraday's is very similar to rack. If you're using like Xcon, it's gonna look a little bit different. But this is just an example. So pass on our HTTP client app, we'll do some tracing and keep calling down the chain, it's pretty similar. But the tracing itself is gonna be a little bit different. So we're actually gonna need to manipulate the headers to pass along some tracing information. That way if we're calling out to an external service like Stripe, they're gonna completely ignore these headers because they don't know what they are. But if you're actually calling to another service that's in your purview, you'll be able to see further down. So each of these colors, it's gonna represent an instrumented application. So we wanna record that we're starting a client request, and ensure that we're receiving client requests. Add in the middle where just like we did with rack, it's pretty easy. You can even do it programmatically, like automatically for all your requests for some of your HTTP clients. So we've got some of the basics for how distributed tracing is implemented. Let's talk about how to even choose in this ecosystem what system is right for you. So the first question is, how are you gonna get this working? I'm gonna give a caveat that this ecosystem is ever changing. So this information could actually be incomplete right now, and it could be obsolete, especially if you are watching this at home on the web. But let's talk about whether or not you should buy a system. Yes, if the math works out for you, it's kind of hard for me to really say whether you should do that. If your resourcing is limited, and you can find a solution that works for you, and it's not too expensive, probably, unless you're running a super complex system. Light step, trace view, examples that offer Ruby support. Your APM provider might actually have it too. Adopting an open source solution is another option for us, the paid solutions just didn't work. So if you have people on your team who are comfortable with the underlying framework, and you have some capacity for managing infrastructure, then this really could work for you. So for us, for a small team, we're just four people, four engineers. We got Zipkin up and running in a couple of months, while also doing a million other things, but partially because we were able to leverage Heroku to make the infrastructure components pretty easy. And if you wanna use a fully open source solution with Ruby, Zipkin is pretty much your only option as far as I know. So you may have heard of open tracing, you might be like Stella, what about this open tracing thing? That seems cool. A common misunderstanding is that open tracing is not actually a tracing implementation, it is an API. So its job is to just standardize the instrumentation, like we kinda walked through before, so that all of the tracing providers that conform to the API are interchangeable on your app side. So if you wanna switch from an open source provider to a paid provider or a price versa, you don't need to re-instrument each and every service that you maintain. Because in theory, they're all being good citizens, they're conforming to this API that is all consistent. So where is open tracing at today? They did publish a Ruby API guidelines back in January, but only LightStep, which is a product in private beta, has actually implemented a tracer that conforms to that API. So if a tracer, existing implementations like Zipkin, they're gonna need to have a bridge between the tracing implementation that they have today and the open tracing API. And the other thing that is just not clear still is interoperability. So for example, if you have a Ruby app, Open Tracing API, everything's great and you have this paid provider that doesn't support go. You can't necessarily use two providers that use open tracing and still send them to the same collection system. So it's really only at that app level. Another thing to keep in mind is that for both open source and hosted solutions, Ruby support means a really wide range of things. At the minimum, it means that you can start and end a trace in your Ruby app, which is good.  It's not a deal breaker. We ended up having to do that for Xcon for Zipkin, but it may be an engineering time commitment that you are not prepared to make. And then unfortunately, because this is tracing everywhere, you're gonna need to rinse and repeat for every language that your company supports. So you're gonna have to walk through all of these thoughts and these guidelines for go or for JavaScript or for any other language. So some big companies find that with the custom nature of their infrastructure, they're gonna need to build out some or all of the elements in-house. Etsy, obviously Google, they're running fully custom infrastructure. But other companies are actually building custom components that are tapping into open source solutions. So Pinterest, Pinterest is just an open source add on to Zipkin, similar to Yelp. So if you're really curious about what other companies are doing large and small, Jonathan Mace at Brown University published a snapshot of 26 companies and what they're doing. It is already out of date, like one of those things is already wrong, even though it was literally published a month ago. So 15 are using Zipkin, nine are using custom internal solutions. But yeah, most people are actually using Zipkin. So another component about this is what are you running in house?  What do they want to run in-house and are there any restrictions? There's this dependency matrix of the tracer and the transport layer, which need to be compatible with each one of your services. So JavaScript, Go, Ruby, and so both the tracer and the transport layer need to be compatible across the board. So for example, for us, HTTP and JSON is totally fine for a transport layer. We just literally call out with web requests to our Zipkin collector. But if you have a ton of data and you need to use something like Kafka, you might think that's cool and then it's totally supported. But if you look at the documentation, it's gonna say Ruby and then you're gonna be like wait, no, if I dig in four layers deep into this documentation, it's only JRuby. So that's like a total gotcha and so for each of these, you really should just build a spreadsheet because it's pretty challenging to make sure you're covering everything. The collection and the storage layers, those don't have to be, those aren't really related to the services that you run. But they might not be the kind of apps that you're used to running. So for example, Zipkin is a Java app, which is totally different from the apps that my team runs. Another thing you need to figure out is whether or not you need to run a separate agent on the host machine itself. So for some solutions, and this is why we had to exclude a lot of them, you actually need to install an agent on each host for each service that you run. And because we run Heroku on Heroku, if we can, we can't really do that, because we can't just give root level privileges to an agent that's running on a dyno. Another thing to consider is authentication and authorization. Who can see and submit data to your tracing system? For us, Zipkin was missing both of those components and it makes sense because it really needs to be everything for everybody. And so also adding on authentication and authorization on top of that for every single company to use that open source library is not really reasonable. So you can run it inside of EPN without authentication. The other option is using a reverse proxy, which is what we ended up doing. So we used two build packs, apt, and then run it build pack. And so we are able to get nginx on our Heroku slug, which is just like a bundle of dependencies and your code with apt. And it's just a package manager for Linux. So we can download and install a specific version of nginx to run as a reverse proxy. Run it allows us to run our Zipkin application and nginx alongside each other in the same host. And we didn't want anybody on the internet to just be able to send data to Zipkin. Like if you just suddenly started sending data to our Zipkin in an instance, that would be pretty weird. So we wanted to make sure we're only having Heroku applications interacting with it. And so we decided to use basic authorization for that. We used HT password to set some just team-based credentials in a flat file, because we only had about 25 different basic auth configurations that we thought we'd be using. And it ends up looking like this from an architecture diagram standpoint. The client makes a request, nginx is going to intercept that, check against the basic auth and make sure it's valid. And then if it is just forwarded along to Zipkin, otherwise it returns an error. And so adding authentication on the client side itself was as easy as going back to that rack middleware file and updating our host name with both basic auth. So that was a really good solution for us. We also didn't want any of y'all to be able to see our Zipkin stuff on the internet, which right now, if you just run a Zipkin instance, there is nothing to keep you from seeing anybody if there's no authorization. So we use Bitly that has an OAuth2 proxy, which is super awesome. It allows us to restrict access to only people with Heroku.com email addresses. And so if you're on a browser and you try to access our Zipkin instance, we're going to check to see if you're authorized. Otherwise this OAuth2 proxy is going to handle the full authentication. So it's configurable with different low balancers slash reverse proxies and OAuth providers. So it's actually really cool if you need to run any kind of OAuth in front of a process. But even if you're going the hosted route and you don't need to handle any of this infrastructure, you're going to need to ask about how are you going to get access to people who need it. Because you don't want to be the team who has to manage this handoff of like sign-ons and sign-ins and oh, you need to email this person. You don't want to manage all that. So just make sure it's clear with your hosted provider how you're going to manage access security. If you have sensitive data in your systems, which a lot of people do, there are two places specifically where we had to really keep an eye out for security issues. One is custom instrumentation. So for example, my team, the tools team added some custom internal tracing of our own services using prepend to trace all of our postgres calls. And so when we, like we did with the middleware earlier, you just, we're wrapping that behavior with tracing. But the problem here is if you're calling SQL.2S and that SQL statement has any kind of private data, you want to make sure that you're not just storing that blindly into your system, especially if you have PII, any kind of like security compliance information that you're storing. And the second thing is that you need to talk through before it happens what to do when your data leaks. For us, running our own system is a benefit because if we accidentally delete data or leak data into a third-party provider, it's easier for us to validate, sorry, it's easier for us to validate when we own that data that we've wiped that data than having to coordinate with a third-party provider. It doesn't mean you shouldn't use a third-party solution, but you should ask them ahead of time. What do you do when data leaks? What's the turnaround? How can we verify it? You don't want to do that when you're in the middle of a crisis. The last thing to consider is the people part. Is everybody on board for this? The nature of the distributed tracing is that the work is distributed. Your job is probably not going to end when you just get the service up and running. You're gonna actually need two instrument apps. And there's a lot of cognitive load, as you can see from the 30 minutes we've talked about this, into understanding how distributed tracing works. So set yourself up for success ahead of time by getting it on Teams roadmaps, if you can. Otherwise, start opening PRs, is the other option. Even then, you're probably gonna be able to need to talk through what it is and why you're adding it. But it's a lot easier when you can show them code and how it actually interacts with their system. So here's the full checklist for evaluation. We'll cover one last thing before I let y'all go. If you're thinking this is so much information, where do I even go next from here? My advice is if you have some free time at work with a 20% time or hack week, start by trying to get Docker Zipkin up and running. Even if you don't plan to use Zipkin at all, it includes a test version of Cassandra built in. So you just need to get the Java app itself up and running. And you don't have to worry about all of these different components right off the bat, if you're just instrumenting Ruby apps, then Zipkin is compatible. You can even deploy this onto Heroku. So once you're able to just get this deployed, the UI loaded, just instrument one single app. Even if the only thing that app does is make a third party stripe call, it'll help you turn some of these really abstract concepts into concrete concepts. So that's all I got today, folks. If you have any questions, I'm actually heading straight to the Heroku booth after this in the big expo hall. So stop by, I'll be there for about an hour, come say, come ask me any questions or talk about Heroku or get some stickers. So, see ya.


And as part of measures to unite the Yoruba race from across the entire southwestern region in Nigeria, a sociocultural group named the Council of Yoruba Elders, CYE, has emerged from former Yoruba Council of Elders, YCE. Consequently, former Super Eagles head coach and FIFA executive member, Chief Adik Boye Unigbinde and Ye Ye Odua, of the SIRS, Irelu Abyola Dosumu, have emerged as president and vice president of the new group. The convener of the event, Dr Victor Tywa, who is the national president of Odua Redemption Alliance, who, as well as a pioneer member of the Yoruba World Congress, set the central motif behind the birth of the new sociocultural group, our unity and wonders among the Yoruba race worldwide. Chief Oigbinde discloses that the task ahead is hercling and as such require all well-meaning Yoruba sons and daughters to embrace, to brace up for the real pursuit of unity in Yoruba land. So Yorubas, if we have problems, then we cannot run away from crisis. A good leader must not run away from crisis. So this one, if you listen to my address, okay, we are the Yoruba people who are speaking for the Nautanas. It's indeed we are speaking for the Easterners, we do speaking for the Nidubes, and they are speaking for the Yoruba land and they say we are intelligent, they say we are wise, we are not wise, but somebody must rise to make a difference. This is what you are doing. As you are here now, this is Yoruba nation. So Nauula was a Nigeria is what is near geographical expression. So it's not about that right now. What is important is that we are in a nation and we have to operate within the nation at the moment. Yoruba nation is not something we are trying to create, it's something that has been in existence. So I would say yes, I would support the president fully. But we are not united, you know, to bring all this success together and form, and that would be my target. I know what I am going to do as a result of Nauula. I am going to concentrate on immediate target is unity and unity, because I know if we are united, we can go very high. Hello, hope you enjoyed the news, please do subscribe to our YouTube channel and don't forget to hit the notification button so you get notified about fresh news updates.


Hey what's up guys welcome to a new video. So today I'm going to show you three really awesome products with huge potential over the course of the next few months. So with Christmas coming up then now is the ideal time to get into drop shipping just because typically then it's the most profitable time of the year. So if you have been meaning to get your store going but you're not quite sure what products to sell then here's three really awesome ones with huge potential and I'm going to show you why as well. Now before getting to product number one I just want to very quickly mention as always in every single video I am giving away a free one-to-one call with me. So a chance in for me and you to have a chat one-on-one and I can help you with your drop shipping business in any way you need it. So it could be a store review we could go through your Facebook ads we could talk about products and for your chance to win that then it's dead easy to do. All you have to do is like this video subscribe to my youtube channel and leave a comment down below. In terms of the comment it can be whatever you want it can be a video suggestion it can be your opinion on some of these products it can just be the word e-com just any sort of comment then and that will be enough to enter you into the draw and with that being said then guys thanks again for tuning in I hope you enjoyed the video and let's get straight into it. So product number one then is this heated vest that you can see on your screen now when it comes to choosing products then I like to choose products based on evidence it's not just my gut feeling I want that reassurance that it is actually a good product so that's what actually I'm going to show you in this video today I'm not just going to tell you why it's a good product I'm also going to show you too. So number one then is very current and trending as it says there I'm going to show you the google trends ratings a simple search for a heated vest and this is worldwide by the way over the past five years and you can see that there's peaks and there's troughs but during the peaks it's kind of around October time where it peaks in actual November December time and then slowly starts to come down and as you can see as the years have gone on it's just got more and more popular with more and more people searching for this particular product so if the trends repeat themselves then this year coming up now you can see it's just starting to creep up it should go up and above this making it a record year then if you like for the popularity of this product so now the evidence suggests is the time to start selling this product. Number two then is the huge scope and I mean that in terms of marketing opportunities I'm also going to take you through the different tags and options in which I suggest for every single product in this video and in terms of huge scope then there's so many different people you could target with the products like this it could be hikers walkers campers all kinds of different people there'll be more information on that in a second. Number three then there's multi-country sourcing to give you an example here is the product listing on Aliexpress and as you can see this is shipping to the UK by the way there's even more if you're shipping to the US and there's loads of different countries you can source this from and to give you an example then if we were shipping this to the UK we could source it from Spain it increases the price very slightly but now if you look at the delivery time we're looking at less than two weeks which makes this a prime and ideal product to drop ship especially this time of year. Moving on then there's good upsell opportunities this supplier themselves also sell thermal socks so there's lots of different product ideas which you could add on to this and once you get somebody to say yes to spending say 30, 40, 50 pounds it's a lot easier to get them to say yes to spending an extra 10 or 20 pounds which is overall just going to increase your profit margins which brings us on to the next point which is good profit margins again I'm going through the complete numbers with you guys so you know exactly what you need to sell this at and what you need to spend on Facebook ads in order to make a profit and then the final one is it's easily marketable to give you an example then one of the first ads I found was this particular ad here which I'll play for you now and this is just one of the multiple ads I found on Facebook too if you were one of those people that wanted to trap upload to different videos and make your own one then there's already plenty of content on the internet in which you could do so. So if you're an example then of how somebody's advertising this I've actually opened up the store in which this links to and as you can see these guys are selling it for 60 pounds and if we'll just go back to my document I am basing the sale price on 40 pounds or 40 dollars so this is a worst case scenario in the more likely circumstances you'll actually be able to make a lot more profit than what I'm going to go through in this video with you. So talking about profit then the total price of this including e-package so this is shipped to your customer no additional costs is 16 dollars 44 cents to sell it at sale price of 40 dollars would leave a 23.55 profit margin if you could achieve a cost per purchase of 10 pounds on Facebook maybe even 15 pounds then you're still looking at 10 to 15 pounds profit per sale if you were to sell this at 50 60 dollars and still achieve that same cost per purchase then you're looking at 20 30 dollars profit per sale which is a very decent amount you don't have to sell 10 products per day to make 200 dollars profit which equals out about 17 to 80 grand profit per year so as you can see potentially a very profitable product once you do the maths then and divide the sale price by the profit you get 1.7 breakeven ROAS and as you can see that's quite achievable in terms of the actual marketing on Facebook then this is where the huge scope comes in there's multiple different ways in which you can market this product what I recommend is that you have the same ad copy in terms of the video but then the actual text itself is geared towards individual customers the more specific you can make a Facebook ad for a particular customer the more likely is to resonate with them the more likely they are to purchase it so for example then you could have male and female or of ages but then you could have an ad for hikers and ad for walkers and ad for campers make sure you include online shoppers which you can actually choose as an interest therefore you're more likely to target the people that buy and post by products like this you can also go and pass purchase behaviors and of course dog walkers this would make an ideal product for people walking a dog in the winter when it's cold so they can keep themselves warm moving on to product number two then guys which is one of my favorites I think this has a huge potential and for the following reasons then number one is a hugely trending products right now again to show you on google trends you can see this is the past five years this is worldwide too the kind of light october time is the beginning if we look at the current time right now it's on the upward trend so again now it's the perfect time to get in on this product in terms of the actual countries too you can see it's very popular in the US very popular in the UK Canada Australia and Germany and in terms of the related queries as well this is where this product has so much power and potential in my opinion because it gives you the ideal markets in which you can go after so when you're advertising your products we can see the warm leggings for snow is currently on the breakout so you could perhaps order one of these order a pair of these yourself and then film a video in the snow because you know that there's a demand for it there's a demand for warm leggings for hiking so again you have a specific market for this product and that's people interested in hiking back to the document then moving on to the next point it's a massive market size obviously anybody living in cold conditions this time of year this product is going to appeal to number three then in terms of the upsell and cross-sell opportunities then there's just multiple multiple different ones obviously somebody's buying this product to keep them warm in the winter so just think about any other product that would help do that it could be certain hats it could be gloves it could be socks there's just multiple different opportunities for upsells and cross-sells the next point then is good profit margins which I'll go through with you in a second and the final point then which is problem-free shipping so with a cheap product like this that contains no electronics no batteries nothing like that then you're more likely to get a problem-free shipping process so there's going to be no delays nothing like that and typically then from my own experience I've found products like this to arrive a lot quicker too so all that being said then let's get into the numbers the most important bit how much profit can we make in terms of including e-packet delivered to your customer looking at $5.98 that's on average depending on what size you could sell this for easily $25 if not a lot more again but I like to prove I like to base my numbers on worst-case scenario so in reality if anything you should make more so with a $29.99 sale price that leads $19.01 minus your Facebook marketing costs which leave you at a 1.31 break even ROAS so again if you can achieve that 10 pound cost per purchase on Facebook you're still looking at about $10 profit per product which is pretty good for a cheap product like this in terms of the tags and then this is where the market size like illustrates just how big it really is because in terms of your ideal customer then it's it's just absolutely huge so obviously I would probably stick to just females with a product like this but you never know I would go all ages I would make sure I include online shoppers and purchase behaviors that way you're more likely to include those people that buy impulse buy products and then in terms of actual specific interests you could go for hikers walkers dog walking campaign fitness anybody living in a cold country the possibilities really are just pretty endless moving on to the product guys is this LED bike light it's not just a generic one though I'll show you the product page in a second it's quite interesting it's quite a cool looking product and the reason I like this so much then is purely because the cycling niche is such a passionate one I do sell in it myself so I do have a good knowledge of how the market works anyway these are the reasons why I picked out then number one is it's a consistent seller all year round but also has peaks and to give you an example of what I mean then again we've got the Google Trends search history you can see that that is quite popular throughout the year this is from 2004 by the way by the way but then it also does have peaks as well and those peaks are kind of like this time of the year as you can see so again now is the perfect time to sell a product like this going back to the google document then the next point is easily tagged to what on facebook I will go through with that you in a second the next point is huge upsell slash cross sell opportunities for example then if somebody's looking for a rear bike light the chances are they're looking for a front bike light and then there's a whole load of different gadgets and accessories that you can you can put on top of that so you can get different kind of like bags for your bike for people to put their phone in you can get phone holders to put in handlebars there's just a whole range of different gadgets and accessories that you could upsell and cross sell with this product the next thing as well is that solves a high value problem obviously road safety at points can be a matter of life and death so this does solve a pretty significant problem for people which ultimately ultimately makes it a good product to sell and the final thing then is this product here is rechargeable and this is huge and the reason being for this then is because I used to sell these bike lights they were led and they were you needed three triple eight batteries it was a really good bike light and it was cheap and the rechargeable version was about ten times the cost so I stuck with the battery version and a lot of the comments and complaints I got from the product was that the batteries didn't last very long and people cycle because they're conscious of the environment therefore they much preferred a rechargeable bike light because it was better on the environment you weren't constantly buying batteries and throwing them away so you might not think it's a big deal but trust me from my experience then people much prefer a bike light that's actually rechargeable another big thing in which I forgot to mention probably one of the biggest things as well certainly here in the UK it's getting darker a lot quicker now especially when the clocks go back it'll be inevitable people will have to cycle to and from work in the dark in the dark therefore there'll be more and more people looking to buy a product like this that being said then let's get into the numbers how much profit can we make with this four dollars seventy nine include an e-packet you could sell this for easily twenty dollars to show you the actual product page itself the product itself looks really nice again there's loads of content behind it and there's all these different options as you can see so again different potential different routes into market potentially different customers you can target so all in all just a really nice product in my opinion so back to numbers then that leaves fifteen dollars twenty cents profit which gives you one point three two break even rewards which is very low for facebook plus you've got to consider all the different cross out opportunities you could have on this as well if you can sell the front light for say ten dollars only cost you three dollars plus a stand for the mobile phone then potentially you can make a significant amount of profit off this one product in terms of the tax and then it's quite a simple niche to target in my opinion because it's so passionate i would go for male and female thirty five plus and then in terms of the actual interest themselves i would go along the lines of different cycling brands cycling in general you can target as an interest you can target commuters cycling athletes cycling competitions there are just really hundreds of different ways you could go about marketing a product like this in terms of the content for your ad i would really invest in the product itself and film a really nice decent video ad for it to give you an example then i have a drone and one of the things i did was buy a cycle light and actually video somebody cycling down a road with the light and that just worked really nicely with all that being something guys that is all three products thank you very much for watching this video if you're still with me i want to very quickly share with you the bonus product just as a reward then for watching the whole video which is this led umbrella just to take you through the points very quickly number one is untapped i really struggle to find anybody selling this product on facebook number two there is a brand new star wars film coming out in december 2019 so star wars products are going to be in demand and something like this is just really cool in my opinion however one point to consider then is do not market this as a lightsaber umbrella purely because the term lightsaber is coined so you may be susceptible to certain licensing and trademark issues so don't call it anything star wars related however as soon as somebody sees this they're going to see this exactly as a lightsaber umbrella in terms of the money then it's going to cost you about eleven dollars including the packet delivered and in my opinion you can sell this for thirty five dollars if not a lot more certainly if you come across the right audience that are super passionate about star wars i can see somebody paying about fifty dollars for this that would leave you with a minimum of twenty three books profit per sale which gives you a one point four six break even row as so as you can see potentially a very very very profitable product in terms of the marketing for this product then i would really go for male and female i would go for all ages too and then just anything star wars related and obviously make sure you flex with online shoppers too that way you're more likely to include people who impulse buy products related to star wars one thing i would say is make sure you go for quality over quantity on this audience only the true star wars fans that are super fanatic about it are more likely to purchase a product like this with all that being said then guys thanks very much for watching the video that's all for products and without any further ado let's get into announcing the winner of the previous video for that one-to-one call if you want your chance to win it obviously please do hit the like button subscribe to my youtube channel and leave your comment down below so here we are then guys on my previous video massive thank you to everybody who watched it and left their comment down below i'm just going to take the url then head over to our random comment picker get the youtube comments and let's start the raffle then so these competitions are 100 percent random i don't pick the winner unfortunately and the winner of the previous video then is daniel cohen so thanks very much for your comment help me up on instagram or we can get that call arranged and guys if you just want to get straight down to business and book a call right away you can actually do so just make sure you check out the links in the video description below and with that being said then thanks again for watching the video i hope you enjoyed it and i'll see you in the next one


rates in Pennsylvania and he thought that voting in the way that you have described was dangerous. This is what he has said on Twitter. I'm not sure if you can see it or not, but for the benefit of our viewers who can't, the Supreme Court decision on voting in Pennsylvania is a very dangerous one. It will allow rampant and unchecked cheating and will undermine our entire system of laws. It will also induce violence in the street. Something must be done. Is that a particularly presidential tweet? Absolutely and he is completely correct. Real quick, before I let Sebastian Gorka here get to his smackdown of this propagandist, it really just goes to show how these media narratives they continue over across the pond in another country's media. It's completely insane. All for the last few months, last five to six months has been nonstop left-wing violence by groups like BLM and Antifa who don't hide who they are and are very open about their goals. Their anti-media have attacked journalists and reporters, but they cover all that up in order for them to push forward this completely fake false narrative that it's Trump supporters that are going to go out and riot if the election doesn't go their way. All these businesses are boarded up because of left wingers, not right wingers. Let's continue. I'm not sure how being concerned about violence is an incitement to violence. That's a very perverse way of looking at things, especially after the last 10 months, when it is a left-wing Democrat voters from Portland to Seattle to Baltimore to DC that have created us in havoc and looting to the cost of more than $3 billion to small business owners. So, you know, the reason that DC is boarded up, walk down DC today, Kay, and look at the shops that have been boarded up. It looks like Beirut. And why are shop owners painting on the chip board that has boarded up their shops Black Lives Matter if not to placate the radical left? Exactly. It's not the Trump supporters. Maybe it's because they think that Black Lives Matter. Can you believe that? She just said that these businesses are putting up BLM on their businesses because they believe that Black Lives Matter. Listen, nobody out there, besides a fringe few people, think that Black Lives Matter. They're putting BLM on their stores to placate the organization BLM, which is completely separate from the phrase Black Lives Matter. Okay, this is well established. We all know this. But it's like, are these reporters really this ignorant? Are they playing dumb? I don't know.  Just watch this. No, it's because they want to make sure that they're not the ones who have their shops burnt to the ground. Come on. Show a bit of common sense, please. Okay. What about when? If it were Trump supporters that were the threat, they'd say MAGA on the shop fronts and they're not. So the violence has been normalized by the left and it is a disgrace. Okay, what about Trump telling the far out group, the Proud Boys, to stand back and stand by? How should we interpret that? Hold on real quick. Real quick. They mangle. What is it with Trump quotes in the media where they just completely mangle it? Okay, when Trump said it, I kind of cringed.  And they always get that wrong. But as Gorka is going to point out here, he was telling them not to do anything. And I'm going to get into that some more, but let's let's go through this here real quick. Exactly as you said it. Don't take action. To stand by. Stand back, stand down, stand by and wait. They didn't say stand down. They said stand by, didn't they? Stand by means don't have. He said stand down. Can you tell me what they are? Tell me about it. Tell me what they mean to you. No, you tell me. No, you tell me. No, no, I'm the interviewer. You tell me what they mean to you. You raised it. You raised it. So tell me why you're asking me. I'm asking you because it's a far right organization as far as I am aware. And they are real quick. I just want to point out how the media is something that you can look for. I talk about patterns you can look for. And there have been studies to back this up. But the media uses the term far right, right-winger, extreme right. Those kind of phrases, they use those with abundance. If they want to discredit an organization or their political opposition, somebody that's maybe bringing up information that the Democrats don't want their voters to hear about, they'll just demonize those people as far right. What you will never hear them say are very rarely are left-wing, far left, liberal, communist, socialist, Marxist. They don't use any of these terms. Very, very rarely. And if you do find it, it's in print media. And this group, what makes them far right? Who knows? What violence have the Proud Boys committed? I don't know. The only thing I know about is that they've gotten into fights with violent communist mobs. That's about it. Other than that, as far as I know, they haven't done anything and haven't threatened to do anything. But let's continue. Who runs the Proud Boys? Do you know who runs the Proud Boys? I'm completely ignorant of them. Please tell me all about them. Well, that's a real indictment of Sky News, that you are ignorant about the organization you are asking me to comment on live.  So how would that be far right? Okay, so you don't feel that they are a far right organization? A far right organization wouldn't be run by a black Cuban American, now would it, Kay? I don't know. And I'm glad that you have made your view on that. I'm sure that my view, I'm sure that my viewers will make their view. Alright, well, we just went through that video and I found it highly entertaining. Gorka pointed out several glaring problems with this. This reporter doesn't know anything about the Proud Boys, but just calls them far right as a boogeyman to use as a cudgel as a weapon against Trump, his supporters, his spokespeople in the media. They don't know that all the violence that's going on out there right now, all the violence being threatened, is being threatened by left-wing groups, by far left Democrat voters. It's not Trump supporters going out there threatening violence or committing any violence. The media rages and freaks out about Trump vehicles being on the road with a Biden bus, literally just driving down the road with a Biden bus. They freak out about that. But five months of nonstop violence killing, threats, intimidation of just regular innocent people out on the streets, taking away their right to travel by blocking highways. That's all been done by left-wingers, not right-wingers. I mean, she didn't even know the exact quote that Trump said.  He absolutely said stand down. The fact of the matter is, and we all know it's true, if Biden wins tonight, there's not going to be any violence. Trump supporters aren't going to commit any violence. You know what we'll do?  And we'll hold their feet to the fire 50 times what they did to us in return.  You know what's going to happen? If there's any violence, that's going to be committed by left-wingers if Trump wins. Maybe even if Biden wins, they may still riot and protest. So anyway, just wanted to quickly get that video out and comment a little bit on it. I want to remind you all that tonight we'll be hanging out in my discord chatting, watching election coverage. So I welcome you, invite you to come and join us. I may also do some discord live streaming. I'm not sure yet. Tons of people already out there doing it. But come and hang out with us. It should be a good time no matter what. If you like this video, please hit that like button, share and subscribe. And thanks for watching. Keep coming back.