Hi everyone, I'm Vishal Parveer and I run a company called Cloud Cover. That's me. So actually we have two companies. Act Development is a software-developed company and we've been building stuff for a while now. I started Act Development in 2002, so we've been around for 12 years now. And Cloud Cover is a new thing that I've started which is it deals with cloud consultancy. We're cloud agnostic, so we help companies achieve scale on all the various cloud offerings. Could I have a quick show of hands? How many folks are working like exclusively with cloud hardware? And is anybody here that doesn't work on any cloud stuff? Maybe we just need a bit of help. Sure. As loud as you like. Anyone who's not working on any cloud stuff? It's wrong with you guys. Come on, get on to the cloud. Alright, so I'm going to jump straight in here and I'm going to actually back it up and come into each of the individual elements as to why we chose to do the things we did and specifically talk about the problems that we had. But before we get to that, let me just tell you what this product is. So this is a customer of ours and I'm kind of their acting CTO right now because they don't have anybody that fits the bill. They've been our customer now for five years, so we've kind of scaled with them and we've got them to the point where they're capable of doing the work that they do. PySmart is an electronic pre-paid recharge system. So what that means is that they do top-ups for your dish TV, your Tata Sky, your airtells and also all the mobile carriers. So they also do it for like Vodafone and Tata Boko. They basically serve everybody. So any of the major satellite operators, any of the major telephone operators in India, they offer those services. So this is a B2B model. They have literally 50,000 different retailers all over India. They have 2,500 distributors and they're literally in every single town. And the way that it works is you send an SMS to PySmart. You send an SMS or you use the Android application or you use the website and all that stuff ends up at PySmart and then we talk to each of the different operators that are there. And we have to do this, so there's separate integrations with every single one of the carriers and every single one of the carriers has a different API and every single one of them is using a different backing and effectively it's a nightmare. It's like every single one of them is horrible. So things that you have to keep in mind as to the complexity of the system or the reasons why it was hard for us to scale, there's lots of concurrent connections. A typical top-up would take at least three separate HTTPS connections in which we are polling some service and we are waiting for a response and in some cases the SMS is coming in from an external aggregator. In some cases obviously it's coming in directly from the website or from an Android application in which case it's a little bit easier for us to handle. But when we started out all this stuff was synchronous and obviously one of the learnings that we had was let's switch everything to being as async as possible and I'll get into some of the ways that we achieved that. Some of the vendors were absolutely horrible. Some of them took 60 seconds to respond. I'm not naming any names but there are certain TV companies down south that their systems are absolutely horrible. I don't even know how they managed to get this far. It goes down every day, you know, there's... And the thing is that when we first built the system we kind of had multiple services sitting on the same instances and like you could have cases where one vendor goes down and takes down half your system. So it was absolutely horrible. But we've grown past that tension. Okay so I really hate cricket and I really hate cricket because not because of the sport. The sport, you know, I had Azharuddin and stuff and that kind of spoilt me on the topic. But not counting them. As far as bias point is concerned we have massive traffic spikes every time that there's an IPL starting up or there's a Champions League starting up or India, Pakistan, what's the big... Number of transactions that we see on a daily basis can double or quadruple on what I consider that day, what sales considers ability. Month end closing is always bad. And the absolute worst is the time we're entering right now which is your end closing where everyone is trying to close their books at the same time. And since this is a transaction and a B2B system obviously there's finance involved and obviously you're going to have lots of reports that need to be generated around this time of the year. So these are the difficulties that we face and all of that led to massive amounts of failure.  We were down for, you know, at one point in time we couldn't even door on the instances there was so much traffic coming through it. We couldn't get to them how many times you would have had an instance that you can't even SSHA do because it's just absolutely destroyed by our applications. We face that. So why exactly did it start failing, right? As a totally straightforward thing when I say fail I mean the application crash or the service tanked out basically what it amounts to is that you can't respond to the user in time. You know, the credit 500 server error, right? When that starts happening you have to kind of look a little bit further to figure out why, right?  You have only a certain infinite amount of compute you have a finite amount of memory you have a finite amount of storage, database and network bandwidth and these things in conjunction cause issues in most cases it's just because they're not possible, right?  Given that fact and given the fact that we've looked at these problems from various different angles now we've been doing this not just for Buy Smart but also for other companies that have similar problems of scale and, you know, 95% of the time the problem always comes down to the database it's almost like the guaranteed failure point because they haven't got the appropriate level of caching or their queries are too slow or they're using too many inner joins there's a lot of issues in terms of the way the application itself is built that is very little that you can do at a DevOps level to kind of fix you have to actually fix the code so that basically comes down to, again, disk and compute and memory and it's various different aspects of how much and how little you should have in order to have a system that is reliable so the first solution that everyone obviously comes up with is let's just throw money at the problem, right? What does that mean? That means let's add more instances, let's increase the size of the instances let's just basically throw more money at the problem and hope to God that it doesn't fail, right? This obviously is very, very expensive we have, I mean, at one point in time we were spending upwards of $25,000 on ourselves just to keep the application up and running and to give you an idea of how far away we were from where we should have been today with even more capacity we're running at a run date of about $6,000 a month for advice month so that was the saving that we achieved through just fixing the problems behind the scenes this is only useful as a bandaid, it is not sustainable at some point in time your CEO is going to come knocking and say fix the problem because I cannot afford to run this business if you're charging me so much for just to serve the hardware so there's a very real business problem that's underlying this, right? this is not just a case of hey, it's a technical issue this is actually going to matter in terms of profitability at some point in time so second solution unfortunately is to fix the problem I can't think of anything else that you can do short term besides just throw money at the problem but that will buy you enough time to hopefully start fixing these things and there's a lot of stuff to fix so what I'm going to do is I'm going to actually take you through the steps that we did now we didn't necessarily do them in this order but I have organized them in firstly the most logical order I could think of and secondly hopefully the order in which it is easiest to start so you start gaining results right away and the first thing that we started doing was we started caching static content this seems like an absolute no brainer I'm sure that a lot of you are already doing it if you aren't you really should because this is something that costs you very very little in fact it's probably going to drop your costs it is very very easy to achieve and it genuinely reduces the amount of work that your web server has to do the reason for that is so content distribution network is basically a place where you can upload static content and it gets served from edge locations all over the world you've got multiple options in terms of which cloud to use or which provider to use I've just listed four of them the Wikipedia page literally has like 50 of them so you don't have any kind of like concerns in terms of finding one that fits your bill and you know achieves what you're looking for you don't need a web server for a lot of this stuff CSS files, JavaScript files, images, God forbid flash files all these things are static content once you've built them you save them they're not changing there's no execution that's occurring at the server so these are very very good candidates to just push onto content distribution network change the link that you're using to access that resource and who you've got you know 20, 30, 40 less requests going to your web server every single time somebody loads the page how many of you guys are using web applications how many of you are administering web applications every single one of your problem at that point in time look at your web page and tell me how many of them have CSS files how many of them are JavaScript files how many of them are image files every single one of those shouldn't be going to the web server shouldn't be asking the question of your web server number two manage services so if you're in the cloud and you're running your own database server on a standard, box standard instance my recommendation is get the hell off it because there are managed services available from pretty much every single one of the cloud providers and a whole bunch of other people besides people like Engine Yard people like Heroku are offering you specific servers that take care of your database workflow they are very very good at administering them they are very very good at monitoring them they are very very good at scaling them don't try and hit with the wheel use stuff that other people have built that are already operating at scale so in our particular case we were using MySQL we used Amazon's RDS which is the relational database service and we used that for being able to create lead slaves we used that to create replication of replicas so basically you can create a copy and then create a copy of that fantastic for reporting because you don't need to have that data coming from your live server and it takes care of creating the instances easily and effectively like an object of a button or a script execution so it gave us the freedom to concentrate on other stuff so the relational database system that's provided by service that's provided by Amazon that's what we use, we use MySQL on that it's basically a managed service specifically for databases so you have the option of using MySQL, Postgres, Oracle and SQL Server and you can choose the instance, the type that you want and it can just basically fire up an instance and it's fully configured and ready to go so they give you an IP address and if you have a username and password you can use that so the technical application starts consuming the database you can change the size whenever you feel like it you can change the size of the instance itself at least the compute and the memory or you can change the size of the hard drive dynamically it takes back up 75 minutes automatically and those backups are kept for a duration of time so you can use them as point of time storage stores and if for God forbid something horrible happens to your instance you can get another one in about 10 minutes so in terms of downtime, it's insane because you get it back with the last backups data so you can basically say this is my master server I need to have a read slave that I can start mission for just read queries basically right click create, read save, wait 10 minutes then this happens so you take away that problem of oh my God I have to worry about backing up my database how do I transfer the data, these problems they bog us down and really we have no business doing these things because there are other people that have already solved this problem why are we reinventing the view so we've used RBS exclusively for our database stuff I can't say enough good things about it there are equivalent services from all the other major clouds and from a lot of other smaller players as well you want MongoDB, that's available on Rackspace just go over there, start up an instance and we're ready to go you want SQL server, it's available on Azure so you've got so many different options these days you really need to go do your research, find a managed service for your database, migrate your data over there and you will see benefits in terms of the amount of time it takes you to register and the amount of flexibility that you have and these resources become fungible because anytime you need another one you can just right click create another one it's not like that huge headache of how do I migrate my database it kind of takes away the problem of that how much time do you have for it? it's typically the same as what you would pay for a normal instance what's that? okay so let me give you the most expensive one they've got if you're running Oracle on RDS in a large server it's about 54 cents an hour and that's like with the Oracle license be big team so you don't have to buy the Oracle license, it's actually charging you for it for an hour which means that when you stop it you don't have to buy the Oracle so the next point was at elasticity this looks super simple, this is the hardest slide doing this is the hardest hardest part because you have to make sure that your application whatever the point at which you want to start scaling it out you have to make sure that that application is actually capable of running in parallel and that means that things like sessions need to be stored outside the web server things like your database needs to be common because you're basically calling the web server once twice twice four times in order to deal with scale horizontally and then you'll be on balancing so you're taking every single request that comes in and you're saying okay I want you to go to server one, I want you to go to server two, server three, server four and it has to be smart enough to do that and not screw up the user experience which means that your application has to be clever enough to deal with the fact that I'm not going back to the same server every time I hit F5 I'm not going back to the same server, I might go to a different server and that server has to be just as capable of answering that user's request as the first one so this took time for us to figure out luckily we were using external session state management we already had a common database so this part for us was not killer but I had enough problems dealing with customers of us where they don't have this stuff sorted out to begin with and that's a huge, huge asset hopefully your application and the operation that you already got going has some of this stuff sorted out but this is a killer if you don't have this, this is like step one to being able to scale and again, auto scaling is something that so let me touch upon auto scaling here auto scaling is this concept which is very cloud specific because it requires a certain amount of virtualization for it to work effectively what it means is you set an arbitrary parameter whatever you think is right so this is actually figuring out what that parameter is also very difficult you have to choose based on your application what is the point at which this server cannot serve one more guy like if somebody else wants to come and ask this server for information it would fall down, it would die at that point in time, hopefully not at that point, 75% of that you want to have another server startup automatically that will deal with that next request that comes in and more important than that is to remember that when that requirement goes down again that server needs to shut off as well because again, it comes back to price so right now we have three separate auto scaling groups at BySmart all of which are dealing with different aspects of the application that I showed you I'll give you a network diagram of how we've built it at the very end so you basically figure out at this point in time, let's say 50 requests a second on this box if I put one more request on there, it's going to start dying I need another server up right now and you build that script, you figure out how to measure the point at which you want to do that you figure out how you want to scale it and the cloud offers you APIs that let you scale and then the new instance that starts up gets added to the load balancer and it just starts handling requests and the second that it's done, you remove it from the load balancer and drop the instance does that make sense? number four, cash this is hard, not as hard as the other one if you haven't done it but it's still hard before this, I think how many requests have you got on that? I have like 50 requests I get another one, if you want to build that so how do you calculate it and cut it and that? I want, since I want this 48 requests then we will cut it down how does it work? so when I say that it can serve 50 requests today we have a mechanism which tells me how many concurrent requests it serves and I'll get into how we got to that point where we can actually figure that out it's not something that is available to you as a metric it's not something that you can just say, hey this is the parameter that I want to use now when this parameter gets hit, scale the instance usually scaling is done on things like CPU utilization, RAM utilization how many concurrent requests are there these are the things that you can sort of get from your application or from your VM layer and those are fine and maybe those will work for you we have had a bad time doing it on CPU because our application, if you remember in the beginning I mentioned the fact that a lot of the vendors take a long time to respond so if I'm communicating with, say, for example, say, Dishne and Dishne has a problem and it's taking like 2 minutes to respond back to me but you know what happens there is, now I have, you know, a thousand threads all of them eating up RAM, all of them just sitting over there waiting for a response from Dishne, well, that takes up no CPU plan but my RAM is there, like I'm done, there's no space left and the next one that comes is a server file number you have to fit your scaling requirements based on what you feel are the appropriate parameters so now for that particular box what we're looking at is how long is Dishne taking to respond for us that is the most important parameter so we're using that to define whether you should scale or not so you have to figure out what metric to use or the combination of metrics to use in order to achieve almost scale correctness it's not straightforward and anyone who sells it straightforward hasn't done it in practice when you're actually doing it, you realize that you can't just say oh, CPU hits, you know, 90% of the scale but then it's already too late and the application is already failed to hit 90% so finding those parameters is not easy but getting them right is like, once you've done that you kind of fixed the core underlying problem you fixed the fundamental issue and that makes the rest of it much easier to deal with we use this metric now which is cost per unit of work and unit of work is whatever you define so like let's say in our case a unit of work for Buy Smart is one complete transaction so we're going to go and get an SMS to deal with the financial implications of that within our system to forward it to the vendor, to get the response from the vendor and send the response back to our retailer like that entire leg is one unit of work what is that cost in terms of hardware? what is that cost in terms of software? what is that cost in terms of computing power, raw computing power and then up to $32 figuring that out was the most useful thing we did because once we had that number when that number went down we knew we had done a good thing when that number went up we knew we had done a bad thing so we introduced a feature to the web application that drove that number up by 20% a feature, right? it ended up being a bug because it was screwing up it was a new graph on the dashboard and every single person that was logging in was hitting that graph and every single one that was doing that was causing our CPU to slightly spike hard and when you amalgamated it over the course of time that thing cost us 20% which was insane for a small feature like that which nobody really asked for it was just one of those things developers thought would be cool so they put it on the dashboard started screwing up he forced them to optimize it, moved it to cache and then it started working okay so cache is like magic if you do it right cache ends up being like magic because all those queries that were taking forever to execute suddenly, magically they come back in milliseconds because what's happened is you've taken something that used to use a screwing disk to get the answer and you moved it into RAM and when it's in RAM, access to that is almost instant I was having a conversation at JS School in Bangalore last year with one of the guys who was building Book My Show and they hit a hard wall in terms of their scale as well and what they did was they just moved everything into Redis like the vast majority of their queries just moved into Redis and suddenly, magically Book My Show doesn't go around anymore because they've taken care of the bottleneck the bottleneck like I said in the beginning almost all cases ends up being the spinning disk behind the radio because that thing is slow it's always been slow, it's going to stay slow until we start using SSB back storage and even then it's still going to be slower than RAM no matter what you do so it always makes sense when it is possible this is expensive because you're putting all your data into RAM and RAM is not cheap and RAM is volatile and if that server dies you've lost that data but if you can figure out what to put into RAM it can make the difference between life and death as far as your application is concerned figuring out what to put in RAM is another thing it's like we looked at the top 50 queries that were taken too long and what we realized was in all those queries the thing that was taken too long was the fact that we had to check whether this user was allowed to buy this product and that requirement to check if the guy was allowed to buy the product required us to go to the database pull out all the products that he had access to and figuring out whether in that product list this particular product is allowed to be bought then we do another query to find out whether this guy had enough balance to be able to do the purchase then we'd have another query to check if he was blocked in the system first thing that we did was we combined all these queries into a single query that improved performance a little bit but it was still a spinning disk at the end of the day and all that happened was instead of firing three separate queries we were firing one query with lots of joins so it didn't really give us that performance boost that we were hoping for what we ended up doing was we moved that entire dataset of whether that user was allowed to access that particular resource into the cache now I have an object for every single user in my system when the question gets asked do I am I allowed to buy this the response comes back in sub-MS numbers so and the database just stopped heating up magically just stopped heating up so we're still using the RDBMS at the back to store that data if we ever change that data we still write it back to the DB but rather than every single transaction requesting that data from the DB we requested once when the data changes and that's when it's stored in RAM so by using SQL as your canonical store you're basically taking away the problem of the volatility of RAM because you're never changing the RAM back you're always changing it on the disk and then updating the RAM what that forces is that because that data is already on the disk if this server was died I hadn't lost anything because all the data is still in the database loose coupling loose coupling is this concept where rather than having your application call things in sequence and rather than having all these parts live inside a monolithic application to split up pieces based on units of growth so in our case we split it up into this is the SMS piece it deals with getting all the SMSes from the aggregators then we had another piece in the middle that dealt with the transaction itself that is cutting the money from the guy's basket making sure that he has the appropriate funds in the basket all that stuff is the transaction engine that's like the core of the system and then we have another piece which was the aspect that actually has to talk to all the external services so the one that's talking to video car and the one that's talking to dish TV the one that's talking to TataSky all of those are now separate services so we broke the application, exploded it into as many pieces as we could think of made them into small small small small little modules and now you follow these little modules you want them to talk to each other right how do you make them talk to each other so we use choose and we use publication and subscription so how many of you know what pub service is super simple really quick all it is let's say that this is a publication and any number of applications can subscribe so let's say that for Iraqis you receive an SMS publish the fact that we've received an SMS and I have one subscriber whose job it is to write an SMS to the dean I have one subscriber whose job it is to write an SMS to the cash I have one subscriber whose job it is to write that to a law one to write a review tomorrow if my CEO says hey listen, if you get an SMS I need to be able to set a notification out to this particular other service but that's really easy now because that's become one more subscriber so that changes the way you start thinking about architecting the application and it also gives you this amazing flexibility to deal with problems that they file that your business keeps throwing at you if you split up all various activities of your application into individual publications just like thousands of publications now we've actually built a tool where this thing sits over there and it subscribes to everything and I have a law, like a console law that shows me in real time every single thing is happening on all my servers because there are subscriptions, there are publications I get those notifications on my browser window and this keeps scrolling like this because we've got so many different transactions that are happening the value of having that really fast scrolling list is when errors start occurring you start seeing rows get marked in red as the errors are occurring so if you see like a scene of errors suddenly start coming you know something is up one of your servers is behaving so we use that automatic obviously now we have stuff that subscribes to these things and deals with the errors individually but seeing that visually made a huge difference initially so yes, publication subscription queuing I think all of you all should know what it is basically building a queue to start pulling out items from it it's a way of using many many small services to deal with the same dataset so let's say that in our case every single one of these transactions has to be very close every single one of these transactions whether it's come from the android whether it's come from web, whether it's come from sms I have to process that transaction so I put it in a queue and the appropriate services pick up those transactions deal with them and put them back into another queue and that queue is then processed to deal with whether you have to send a notification over the web whether you need to send an sms out all those things are handled from there so we have split everything up into publications and interviews the value of that is it allowed us to rebuild our autoscale because now all of a sudden we had all these small small small parts so you took an application that used to live on one massive server that was able to autoscale but I was autoscaling this huge server every time so when I ran out of space, when I hit that threshold and I had to scale, I had to scale another huge instance now when I hit the threshold I'm hitting the threshold for just the Tata Sky service or I'm hitting the threshold just for the Sundirect service and only the Sundirect service starts scaling which is a small little instance so again, my costs went down because I had all these little pieces that I had to scale rather than this one big application I'm going to go back here so consider no SQL it's not some magic bullet that if you shoot it at your application it's going to solve the problem but it does help in specific instances it makes a huge difference in terms of like for instance, for us when we started using DynamoDB we've been using separate DB for a very long time and now we started switching to DynamoDB because separate DB is done it took load away from our database and it gave us the flexibility to be able to just keep adding more and more rules without having to worry about the schema we didn't have to think about, oh, this particular vendor of ours is returning some completely different type of data from this other vendor I was able to take all that data irrespective of whether those fields matched and put them into separate DB it allowed me to basically worry about the problem later and that allowed us to scale without killing our developers and that's important as well we've used MongoDB in the past, we've used a lot of Redis which is not on this list but it's also very very useful as a key value store I encourage you to look for the right tool for your job each of these things does something totally different each of these things makes a different trade-off in terms of capacity or availability or partition problem and you have to choose which one you want you have to choose which ones you care about and deal with them for you it's not an easy choice and it does take a lot of effort in terms of figuring out the right stuff but it is valuable and it does actually give you benefit in terms of scale once you've got other stuff, you have to optimize the cost you've got to fix the problems in terms of reserving capacity so this is the case for most cloud services but it also makes sense for specific instances that you've got running even local stuff, if you've got hardware that you bought in like a data center you have to choose how much of that you need so resource planning is super super important in our case we have to choose what to reserve how much to reserve because at peak load on a Sunday evening of an India-Pakistan match I might see 600-700 transactions a second at that point in time it's too late like I have to scale but I can't afford to reserve all those instances because I'll use them only once a year or twice a year so you only reserve the stuff that you know is your baseline that's what you have to use every single day no matter what come rain or shine this is an iterative process you reserve only when you show and then you choose later on when you see that this particular thing is getting used a lot again if you size your instances correctly it allows you to size scale in smaller increments so you use very very tiny servers that are capable of doing just that job and they have many many of them because you have no limitations in terms of how many in the cloud you can have hundreds of servers running nobody cares but each of those servers has to be sized right otherwise it will take much more than that particular computer class so let's talk about bismarck today after we did all this crap that's how many stores we address all over India some of them are like you know off in the islands of India that is the footprint of the number of detailed locations that we've got this is our architecture it's highly simplified for this life there's many other moving parts and there's stuff that obviously you can't see like you know our development environment, our staging environment all that stuff is going in here but in and out this is what we've done so we use firstly this entire thing is running on Amazon we use root 53 for DNS resolution which is there in DNS servers that's for bismarck.co.nl all our users come here we've got load balancers for SMS we've got load balancers for the web and the android application is now consuming the web API as well so basically we used to have another line with your Android now all those lines have ended up being reverse so this thing passes SMSes that come in from our aggregators our aggregators send us what's effectively an HTTP post we pass that HTTP post speed it up into all the pieces so it's like the syntax is top up dot password product name dot amount dot phone number so we have to speed that up into all the parts that's what the SMS servers do and then funnel that into the queue and this is pretty big it uses ssd backscoring it's a lot faster and more reliable that's what we're doing right now that's part of our development process so sqs is the queuing system what we do is we put the data in here and we add an element to the queue of it and then these are all the services that we're using for all the third party vendors and this thing scales horizontally each of these services is separate so there's one server set for videocon there's one server set for data skype and so on and so forth so firstly that's not just one it's an autoscale route it's actually 32 separate autoscale routes for this piece and that scales all the time to use it so as you can see by mostly coupling all the services that's our initial database system so we have one master server we actually have a show on all of them and we've got at this point six separate readscapes all of them will be various distributed and they sum them up small sum them up large and tell you about the book any questions about this? what I want to say? so separately there is no security and there is RDS we're using both of them we have to use both because our transaction engine needs consistency because we're dealing with money so I have to cut the money and make sure the money is cut and then I can pass it on what we're using simply in SPS score is to deal with these services so every time there is a transaction every time there is a top up that has to happen once I've figured out some RDS that you have enough money that you're allowed to access it then I write an item to the queue and I write the item to the security queue and then these services pick up the item from the queue say okay this is the key that I need write the key from security and then process the entire transaction and write it back to the queue it's actually a different queue that it goes back to like I said this is super simplified but it goes to a different queue which is the back in dog queue and writes to a different security queue with the data that came back from the venue and then that gets processed once again via SMS or our web server but your SMS user does not come through without anything in it no basically these guys well they still use bysmart it's not bysmart.co.in we still use root53 for DNS but those URLs are basically set inside our aggregators so our aggregators have a specific endpoint that's been defined for them to come and hit on our SMS servers and each aggregate has a different URL so then we can keep them separate and make sure again they split that any other questions so we're actually in EU because when we started this Singapore wasn't available here so 5 years ago and what's the reason for combining the android and web server third line? Android was basically consuming over the API right and what we realized there was no good reason to have that good base be separate because every time that we made a modification to the API of android we also had to take the same modification to the API of the web so we ended up combining the APIs and we unified the android in the web application so the web application now is like a full HTML5 socket based thing we're using Node.js for publishing data we're using Redis to push notifications out to it so you can do stuff like you can start a top up and then do another top up and do another top up and do another top up and they all start queuing up on the side and then the socket sends back data which is data back once this entire process is finished so once the queue has processed once the data comes back one more publication occurs and that publication pushes it to the socket that was corresponding to that request that's over with the top up that took us a while anything else? how many subscribers do you have? if you have questions then please ask them in the mind this is live streaming so people will be able to hear your questions as well what is the how many subscribers do you have? so right now the server number specifically of reserved instances is about 30 but depending upon the scale level obviously the numbers go up and down so I actually don't know how many servers are on the right now because depending upon the workload it might be more it might be less are you using the CDN for that? are you using the service of CDN? yes we're using Amazon for that as well we're using cloud for that actually when we assign the stars that is moving then moves and moves what happened if I try to play there is a you have to try to play with one but it's very very slow so what I'm thinking like we are using this on live streaming so part of the line CDN CDN place yes I'm already in India so you're probably pulling from somewhere how do you see the idea of what CDN provider is good for we only use Amazon and Amazon has two locations in India one is in Delhi and the other might be in Chennai or in Bangalore I'm not sure so there are two locations in India nothing for western, Maharashtra or India but it's pretty fast, we haven't really faced the problem in terms of stream we tend to be a lot quicker in terms of base growth now because we're using the CDN exclusively for even solving the application so what we did was we built a static HTML application with javascript files and all this dynamic content is being pulled over javascript so we don't our web application is more than being served from a web server at all it's just a web app that gets delivered over HTML and then javascript is doing all the rest for okay so just a few numbers these are the number of transactions that we perform on a good day we do about 500 transactions a second at peak load and that's being managed with the server capacity that we have in the sql about about 6000 dollars worth of hardware and we have about 30% headroom so we leave our servers running at about 60 to 70% capacity to deal with that eventual spike that might occur and autoscale only triggers where it passes that 66% threshold so basically keep 33% headroom on the servers before triggering our autoscale servers right yeah so you know some other numbers are very interesting to a large extent we don't have downtime at all because we kind of built it so that all these different parts we might have like sort of downtime for one service because the vendor is not responding like you know dish TV decided to do a reboot of their server and suddenly you can't get dish TV to go through but that's outside our control we try to deal with that gracefully at the client level but by and large our servers don't go down that's good and you know to a large extent that's going to hopefully stay the case we are making a whole bunch of changes in terms of we're starting to add more and more Node.js code because we find that Node.js is very very flexible in terms of being able to sit lots and lots of concurrent connections open without eating up and destroying your CPU so we're moving some of the services specifically the stuff that's talking to vendors to Node.js beyond that some of the stuff that we're trying to do we're trying to use MapReduce so some Hadoop stuff to be able to mine all this data that we're collecting so that's on our roadmap sure thank you this is about testing strategy and this is the main infrastructure and the migration from one to the other and again I'll provide an application and your API to basically combine that and another related to your code an application infrastructure how do you manage your export strategy so for us testing was a case of two different very important things one was the actual testing the application didn't fail but a second more important thing was performance which was a lot harder for us to engage in testing because a change like I said when a change occurred in one particular place they added a new piece of code to the dashboard and suddenly our performance level went up or it went down by 20% stuff still gets through the gate stuff that should not have ever been in production the ends up in production because we have a very disparate system in terms of the load 90% of the load is actually not from our system it's coming from an external person it's coming from how long it takes for a particular vendor to respond so it's very difficult for us to kind of load test to the extent that we have to I can't foresee where the failure will occur here so what we've done is we've split it up into pieces so that when a failure occurs it's isolated only to that part so that was the major learning that we do as far as testing is concerned obviously we have automated testing we have testing that's baked into each of the modules that we're building and then we've got three people whose job it is to break the application so they're testing on all the various platforms and that's just the manual like every time the feature has to go out it goes through the usual checklists and things our deployment strategy is everything has to be up on the staging server and fully tested and certified by everybody prior to moving into the production server and the act of moving it from staging to production is completely automated so we basically flip a switch we say okay this is the version number from git that needs to be put on to the production servers we initiate rolling restarts if required and that's all scripted that's all like I'm using shell scripting and aws so aws lets us control the instances themselves through a command line so I can drop an instance from the load balancer and I can add it back from the command line so I can build it into my script that drop the instance, restart it when it boots up it sends a ping back okay now add it back to the load balancer start the restart on the next one so there's a script that basically deals with that rolling restart all the way across yes so we have our own custom monitoring tool that we've built plus we use that on top of cloud watch metrics which is something that amazon themselves provide us with amazon's metrics give us data at the VM level they are not technically allowed to look higher than that so they can't see what's happening inside my application they can't see what's happening inside my RAM so in order for us to actually figure out what was happening for our application we ended up building a suite of tools ourselves and that's something that we are considering packaging as a product because it's gotten to the point where it's pretty advanced as well so let's see maybe I'll be back here next year to show you the product I think that's rich yeah that's it that's all I got sure well it's kind of hard to say at this point because like I said we run two companies right once the software developer company cloud infrastructure company EPRS and BySmart that the company that owns BySmart is called EPRS were our first customers as cloud cover right and a lot of the systems and things that we developed were around this installation so today I would say that 90% of it is automated there's a guy one guy that has a dashboard open that sees what's happening on the servers in the event of something horribly going wrong obviously it's all hands on deck but routinely like you know day to day very little of our resource maybe 5% goes towards just the upkeep and maintenance of the system but it's gotten to that point after many years I mean it's not something that we woke up in the morning and it was that easy it's got to that point because we had to do all that stuff together sure what do you mean? what do you mean? so we used all the CSF we used Redis for publication and we used Node.js for the application itself we used most TML, CSS, JavaScript for the client dashboards using these three graphs that kind of stuff yeah are you not using any are you not using the publication? we're using Chef so we tried Pocket we settled on Chef after Raging Eternal Debate I'm still a partial to Pocket actually but I lost that part so far we also used a couple of the tools we tried out the stuff that Amazon provides as well they have a couple of DevOps management tools that they have so we tried those as well at the end of the day we felt like we needed more customization we needed to do more stuff so we ended up going without this is actually an instance running that just takes care of the Chef requirements okay anything else? thank you very much guys


Okay, please, please everybody, take a seat, you hear me, please, back to order, okay I have a few announcements before we start, after this Jean's talk there'll be a persuasion and beer would come in the room over there, be right there, there are several things that you have to respect here, so if you get a beer here you have to show you ID, okay, there's no exception that's a rule here at Colorado University, so everybody has to show their ID, even me, look, so everybody's gonna be here and showing the ID and you get a beer, you cannot go outside of these rooms here, okay, you have to stay in these rooms, please stay in the rooms because we don't want the people here, the staff to get a bad rep with the rest of the university, okay, so be nice and stay in the rooms, there's enough space here, the posters are here, so stay here, okay, and also don't worry there are parties outside but you know if you don't get out of the room you're not gonna get beat up, so stay in the room and you know it's gonna be fun, okay, so now we're gonna go on with the workshop and we're gonna have a talk by Jean-Bron about parametrizing surface processes and their response to tectonic climatic forces, thank you, thank you for the invitation, it's a pleasure to be here, so like the previous two speakers I'm a transfuge from geodynamics tectonic modeling into surface processes modeling and what I'd like to do today is address my colleagues, my former colleague of the geodynamics tectonics community, modeling community, by making the assumption that geomorphologists have done their job and they have produced parametrizations or equations that do actually represent or have a predictive power in terms of representing out-surface processes and I'm going to take those explain them very briefly and mostly use them to show the consequences they have for this coupling between tectonics erosion and I must say climate because back in the days this limb that we were trying to show and then we trying to demonstrate using models in erosion and tectonics really was trying to relate tectonics to climate and another thing I'll show that so let's start by looking at the response of surface processes and potentially feedback to tectonics forces and to do that I'll simply do what I've already started to tell us and others that think of our origin mountain building as a cycle been in the geomorphology research for many many years and was maybe more recently expressed by Howard and from the tectonics point of view they also realize that there may be some link with surface processes so that you can get to this idea of a steady state with uplift tectonics and erosion proposed by Jemison and Roman in 1998. So the idea if you take a model or if you think about the real world that you have a region that which uplift is caused by many convergence between two plates and the uplift causes erosion and you go to the surface and after some time you reach steady state with erosion and we think for example that this is happening in New Zealand in the southern part of New Zealand. I won't say much about it but it is already a lot about New Zealand but if you just look at the rate at which it has been converging and for how long it has been converging total convergence is on the other hundred kilometers and the mountain is tiny but clearly you must have reached some kind of steady state. How long did it take to get to this steady state? You can actually say in New Zealand it didn't take more than a few million years at most because we know when the conversion started and we also know roughly when the stronger graphic control actually happened. So we have an idea of when the present day topography and it didn't take more than a couple of minutes. Now the end of the cycle is if you stop uplifting it just dies away. You move the topography by and as I will show you in a moment for many origins actually it looks like that stage lasts much longer than the growth. As an example I'll just take the Pyrenees which you may know and I'll be friends in Spain or Syria and the rest of Europe. Collision stopped there about 25 million years ago and if you look at the Pyrenees they are still a pretty substantial topography feature and if you compare the Pyrenees to the Southern Alps you actually ski on both. So they are very similar in height and shape. So I'm going to do a little bit of math so you don't worry I'm going to explain and I just wanted to get the essence of it not the detail. So one way we model erosion is by using the stream parallel which we say is incision by rivers and we say that erosion is proportional to drainage area and slope to some power. In the care here we've also put precipitation which you can take out which mean precipitation and which also it must be to the same power as A. So what I've done here is I've actually solved that equation for you and looked at time versus height and as I told you as you go and look at the height it grows until you reach some kind of steady state then it stops and then if I remove the uplift then it decayes away. What I've done here is actually scaled the height by what this is mean height but you can just think of this being the height divided or the maximum height scaled by the final height. I don't want to reach a steady state. For the stream parallel if you use that equation you can actually figure out what the actually the theoretical value of that maximum height is and what you can also do is figure out the response time tau which is also a function struck from the equation and the key thing what you do when you dimensionalize your model parameters or your variables here if you scale h by h0 and t by tau you can actually vary the value of uk and s the power exponent you will always get the same curve. What's interesting is that you can also show that tau here is simply h0 divided by you divided by rho prime and rho prime is something that measures the efficiency of isostasis. So it really is a function of the elastic plate thickness that you assume if you are on the half-liter sphere there is no isostasis and then every time you roll one kilometer the topography goes down by one kilometer if you're in local isostasis every time you roll a kilometer you remove less topography. So the key about this is that you can actually derive the value of how long it takes to get to steady state simply from the final topography the uplift rate and this isostatic factor and if you assume that a mountain has reached steady state then the erosion rate must be equal to the uplift rate. So what I've done here is quite simple is I've actually used thermochronology efficient track dating to estimate the present day erosion rate in many mountain belts and assume that for most of these we must be very close to steady state so that must be equal to you. I can measure the height of the mountain with the maps and then or the rebound or the isostatic effect I just look at either global compilation or local studies that give me the elastic plate thickness and if you do that then what you can do is regardless of what you assume as an erosional process if I come back one slide in this equation now we don't have to assume anything about the erosional process okay what happens is that you can predict the originic time scale or the response time the time it takes to actually get to equilibrium for all these mountain belts and approximation but I believe quite a bit and you see that it ranges from let's say one million year maybe for the west coast of New Zealand to something of the order of 15 million so in the range one to 10 million which is also what seems to be observed by looking at what I said for example even but also the sedimentary record sees that when do you reach this constant sedimentary flux coming out of here and you can see well one thing that I've done with that is actually try to answer a question that all of my former friends and colleagues from the geodynamic community ask me they say oh do you have an erosion model that I can put on top of my tonic model so I give them you know the stream power law or version of it and then they ask me oh what should I use for k m and m and I go hmm okay it depends okay the only thing we know the thing we know that the ratio m over m is about 0.4 0.45 okay but you don't know very much okay what I've done here and I look back at the expression for the time scale and you see that if from it if I then this is only it's independent of the process but if I assume that the stream power law is the process you can look simply at the relationship between these two and the power between the two should give me the value of n so if n equals one for example how should not depend on you yeah we know that so I've done that and you can do a fit and what you find is that best fit values too this assumes somehow that you know there's a global value for n which we know is kind of not true I'm not saying that you know I can tell you that the value m is two in all the mountains well but what you can do with this is actually provide to the modelers a tectonic modelers with a low okay an erosion low with numbers in them that would give them the right mountain height or an average mountain height on earth okay and but also it does it provides them with a with a good time scale for the evolution of the system as a function of the uplift and the isostasia you can do a little bit better even by trying to fit both k and n assuming that there are kind of absolute values for k and n and in that case you get slightly lower value n so the question you could ask are you should have told the one because you know n equals one is much easier in your role and then you figure out a k that that kind of fits there and they shouldn't worry because n the power there only really matters about you know if you are interested in the shape of nickpoints in rivers which a guy in tectonic doesn't care well it's not that simple because what happened I'm just showing you here two model runs okay in which I've simply changed n equals one and n equals two and I've scaled the case so that I know topography is the same as they said okay so you run the models they get to steady state and then after a few million years well you stop but you and then you go and decrease what you notice actually is that depending on n the rate at which the topography disappears much different and the higher n the longer your you know rendment of topography and you can show actually that because you turn you off you comes off then you don't have any more a typical exponential increase or decrease so you get actually a power load and you can show that that power load actually becomes an exponential when n equals one but is strongly dependent on n and you can think about if you want to recast that into any kind of you know erosion model doesn't have to be this extreme power load what really matters is how topography all the slope all the curvature which power it appears in the ocean and that will really determine this asymmetry between the growth phase and the decay phase and this may be explained in part with isostasis because isostasis tends also to increase the length of both the growth and the decay phase why we have ancient mountain belt like the southeastern islands in australia or the Pyrenees that can survive for tens of millions of years whereas the evidence is that the growth stage is much shorter okay now as i said the cool thing about linking erosion to tectonics is that in fact it links climate came up with this things saying like you know a mountain will look completely differently if it's on and the equator or at the pole so i want to talk now and go back to this idea of what happened now when we change climate and how does the earth the surface processes here react and potentially and you know there's a lot of debate you know whether this is actually happening when you for example look at the cooling of the climate in the late senozoic did it really change the erosion of the surface of the earth maybe the type of erosion from fluvially dominated and also did it change the efficiency of erosion so i'm going to take the opportunity to very briefly introduce our best models for the glacial erosion it's in a way much simpler first because it assumes that erosion here so again the change of height with time is uplift minus erosion erosion by glaciers is really proportional to sliding velocity of the ice the way how the ice slides and unlike the k and the n of the m of the stream paulo kg and l i actually quite well and i won't go through the detail of how it's done but several methods quite different methods came up with very very similar numbers or l roughly two and kg you know i don't remember the number but two is about you know whatever okay it's a value of k which i can't remember if i have so what we have to do the difficult part of the glacial erosion is that you have to model the glaciers that requires a mass balance between accumulation and deformation and accumulation we kind of have a good idea that it depends on altitude and we all use this concept of equilibrium line it which is the point where accumulation equals and the problem is that then we have to predict the flow of the ice we know kind of rheology the ice quite well against depending on another n and then slow it's about three but what we don't really know especially are these factors that determine how the ice is sliding basically a question of friction to the ice that's the least well constrained problem so i'm going to take that and i'm going to look at well what happens now if you assume climate changes and we are going to assume cyclic variations in climate and i'm going to model them by mapping that into elay variation elacrylamide it varies up and down in response to climate change and so what we can do is model solving these equations the shape of a glacier on a mountain profile and then we can change the elay and we can change the accumulation rate in this case and you can show how the the shape of the glacier changes and the effect it has on erosion what's really interesting is in this case again we can pull timescales out of the equation i showed you earlier and there are actually here two timescales when you change the elay when you change the climate first you change the ice there's a response time of the ice through the climate change then you change the ice with ultimately start to change the erosion change the shape of the mountain on which the ice you get to a new state so that is another timescale what i will call the erosion elay ocean erosion and you can show because we know k g quite well that the response time of the ice is of the order of a few hundred to a few thousand years whereas the erosional response time is much longer from ten thousand years to ten million and we also know don't don't remember that if you want to remember anything remember this focus on the erosional timescales that it must scales as the accumulation rates in minus two first of one over technology right and in the case of where elay calls to which is the most likely it also scales with the inverse of the so with frederick herman you know we've done this work together and eric is here we deal as well we've actually run many models and tried to find out what we call the shape of this gain well i should tell you very simply what the gain is we force the system with a periodic signal in elay altitude at various periods and for each period we look how much does the sedimentary flux coming out of the mountain changes with time if you have no gain it means that i'm changing the climate the flux doesn't change if the gain is one it means that if i change the climate by 10 percent the sedimentary flux is going to change by 10 percent the gain is high so what you see then in all cases you get no gain at short periods high gain in middle range period and low gain again at high a long period the reason is simple here you are going faster than the response time of the ice so the ice doesn't have the time to change and cannot transmit the signal to the erosion at the other end here if you go very very very slowly in changing the climate the ice is the time to change the glaciers the time to change the way it erodes and it remains always a steady state with the uplift and again you don't see anything happening to the flux and you can show according to you know the rules that i've given you from the various timescale that different uplift rates will have different gain function and assuming different mean accumulation rate will have different gain function so in other words if you force the signal let's say milankovic cycles okay we produce milankovic cycle different mountains depending on how fast there are plifting and what is the mean accumulation rate will have totally different very different response to the same forcing in the climate and you can go in from deeper and see if we now feed our model with the known cyclicity in climate which could arrive here from the limo cooling curve of the lacy nodoic and can transform that into variation in elay and put that in our model and what we see so what what is shown here are powers okay so this is power versus period and you can see this is the forcing so this is actually a spectrum of this the forcing and you can see the 40 000 100 000 year cycles but you can also see in the blue here this is the long-term cooling of the and then we apply that and we look at the gain that produces or the power in terms of sediment flux coming out of the mountain and you can see that depending on the uplift rate or the accumulation rate you can amplify some periods of some field will disappear so for example in low accumulating area so the ice rain or precipitation are low you really get a big response to the long-term cooling whereas if you are in a very high uplift area you really amplify the 100 000 year cycle it's really important to do and then what you can do you can do a bit more math if you want and take other processes like back to the stream power law you can look at the hill slope diffusion or propagation of a weathering front and you can derive the value or the stream power law of the time scales that are of interest and you can also in those cases do it for the glacial erosion get analytical functions for this gain function the key though is that they vary on they depend on many things the mean precipitation the mean accumulation they also depend in some cases on the length of the system all that to say that if I look at this is one of these gain functions that I'm plotting we have little hope in my opinion to actually see a global signal of the effect of climate change on erosion because basically if you sit in the middle of the ocean and look at the sedimentary core and see oh do I see actually the Mirankovic cycle the flux of sediment to the ocean with some proxy you are unlikely to see it because every part of the system is going to respond differently to the same forcing and I haven't said that but there's also there's a gain function there's also a phase function and not only some of them will amplify or not amplify the signal some of them will actually offset big question we discussed today is also look we try to couple surface processes with tectonics there's a big difference in scale I'm going to get down to it where there's a huge difference in time scale because as we kind of know in geomorphology surface surface processes rated with your efficiency not only depend on mean precipitation or mean you know slow accumulation but also on its variability on the weather so you know and ideally you would want to run your models with a time step like of a day rainfall changes from day to day of course especially if you want to couple them to 3d geodynamic models it's not going to be really helpful to do that and and what we do obviously is we use statistics I want to show you that and before I do that I want to show you that what I said earlier that you know climate really matters we've actually had difficulties not the modelers do that easily but the guys do the observations actually show us then when you change the climate erosion gets better when it rains more does it erode better that's data and this is now precipitation demutation rate and you know let's judge of that now that's a bit biased because obviously I said it's not only precipitation uplift rate matters slope matters and this says oh no it doesn't matter okay but what I want to explore with you is also how does variability in climate matters why would it be important but it is important in the original processes because there are thresholds so if you have this boulder in the middle of a river this is a slide from Eric nice you can realize that it's not going to move until until the discharge in the river reach some critical value and so clearly many processes in the surface of the earth are characterized by a threshold and therefore the variability of the flow becomes really important because what will really matter are the large event that will go above the threshold move the material from rivers or in glaciers or on tip so we have to have a statistical approach to do that as I said and the typical thing that we do and Greg he has been one of the people doing this first try to characterize the discharge in rivers is a statistical with the statistical properties is to look at distribution of discharge given point and so you know this distribution that gives us the frequency of a given discharge in a river there to other discharge and you see what I'm showing again this is an animation from Eric cool animation that shows different sets of discharge probability density function that are all characterized by the same mean but obviously have different variability white one is much more variable with more event of very large size and small size the narrow one have much less probability and you see now if I have a threshold in discharge to move that rock you can see that high variability implies high erosion rate low variability so I've summarized this here erosion efficiency slightly must be a function of the threshold but it should also be a question a function of the shape of that so the variability and you can show also on the slope of what's called the pale though more math and this is about 10 seconds on the on the Eric for his PhD actually shown that you can do that like others have done it for him but he's also shown that you can do it using a family one of the solution functions of discharge in rivers that fits much better observe distribution and it involves a couple of parameters that you can actually extract from looking at recession curves in river sit in a river and you look at what the discharge is doing after a big flooding is that you end up with variability and another parameter B and you can show that these depend mostly on the response timescale of a catchman which must be related to evapotranspiration or maybe vegetation and properties of soil properties of the catchman the frequency of storm and the strength of the storm so the cool thing about this too is that you can relate not only can get to discharge and related to climatic run you can go from precipitation to discharge and after having done that you can fall back onto a pure stream parallel okay a stream probably that means that erosion rate is propelled to slope area and you've started to untangle what you put in the K when you put in the K depends only on moments of that and in the case of beef especially beef we have an analytical solution so you can actually almost do a simple stream parallel make build with build into it the link between variability of climate and time is running um does it matter okay does it matter really I told you what you need a threshold and it is very uncertain type of distribution this is something we just submitted with Eric and you know feedback is really welcome on this great idea that Eric had but what we do here is actually um run millions of models okay assuming all sorts of thresholds all sorts of mean value for the the discharge all sorts of value for the variability in discharge so this parameter u the variability also the parameter b and what is plotted here for each of these models is here is the relative magnitude of the threshold compared to the mean discharge okay so if the mean is if the threshold is at the discharge this is one here and if the threshold is higher you go that way if the threshold is lower than the mean discharge what you see is that the conclusion to this is that variability only matters if the threshold is equal or larger than the mean discharge and you can actually this plot other one is going to detail this is true regardless of what you assume for an equation for erosion you can assume that the relationship between discharge and erosion rate specific discharge is super linear sub linear or even inverted you can have a this is the two and power effect it always predicts the variability here doesn't matter whereas it does matter a lot for these very large variation in erosion rate for the same mean discharge okay it doesn't matter here the threshold is below the mean or it does matter a lot of consequences for the important you know of the erosional system response to changing climate and I want to go through all of them simply to say that you can show with this that low threshold system will be really responding to mean precipitation whereas high threshold system will be proportional to mean and the variability and you can show also that the variability is most likely to be real temperature finally because as you increase the slope of rivers the threshold decreases easier for given discharge if there's already a lot of slope you can show that steep landscapes are less sensitive mountain ranges steep mountain ranges are less sensitive to variability so now I'll finish by something that maybe even more exciting than linking climate electronics but no power fetch but I think a lot of us are moving in that area to actually now look at the link between the evolution of life you know the formation the evolution of biodiversity for example given place on earth and the evolution of the landscape and I'd like to go one step beyond you know and actually link what happens totally it's serious but so we're doing this with two postdoc Catherine Kravitz who's actually here in Boulder and Jaurev Luong and we are looking in specifically at Madagascar which you may know is actually characterized by a very purely biota and especially this concept of microendemic what it means if you look for example here that you see that all of these catchmen what you have is one species of Lemieux occupies every other same ecology so you think you know that would have for a long time ago decided that one better than the others and would take over the whole island no they never even spread over the whole island but they live each of them specifically one catchman but there are exceptions to that and that's the inland catchman where you don't have this microendemic so they all live together but two things are interesting first what I just described the other thing is also the distribution of catchmen in Madagascar quite unusual actually for a spontaneous area a lot of photographer in Madagascar who have this high purge catchmen that are connected to the base level by this narrow valley whereas you have these other catchmen that are more natural although they are very elongated and there's also the reasons that people ecologists have used to explain the microendemic but one thing and I'm going to go into detail of that they relate it to the shape of the of the of the catchmen but also one thing to keep in mind is that if you try to time when this microendemism actually happened so when this variation happened potentially the catchmen wait for example you actually realize this is late late cretaceous I was talking about 16 years ago present this is a very long process it didn't happen overnight and it's mostly related first from like the Madagascar and the arrows I've put there which seem to correspond to this branching events happen are actually relatively well documented relief event in Madagascar uplift but also when it's reorganization that's the work of a so let me now get my crazy explanation of this and it relates to lecture okay so to the strength of the underlying why well an island is creating by rifting or by uplift of an area that's rounded by water okay so when you create the relief by rifting you create a skarpment by the side of the island and these skarmans are going to be rode away and it's going to create unloading at the surface and by isostasis now flexure will distribute that rebound and if you have local isostasis the the thing will remain basically like this sorry you have no isostasis or very strong lithosphere it will remain like this whereas if you have a very weak lithosphere you'll create this this you know very peaky skarpment top in the middle of a saucepan okay towards with all the drainage from so what I've done here to express that in a more clearer fashion that run full model of you know a rectangular Madagascar island okay it's roughly right and I've just changed the elastic technique I'll just let you appreciate so this is a view from the top I'm showing you the catchment they evolved through time and you'll see roughly where the rivers rivers are that really increase the photographic the duration and what do you see is that you know completely different behaviors in terms of catchment and catchment capture in the case of a very high elastic thickness you basically produce something that's more typical of an island with very linear catchment that go all the way all to the main divide if you have a very weak elastic thickness the rebound you get on the normally make the system such that there's actually only one catchment and this is you know sorry that we never see this on earth or we do see that quite often actually for example the orange river in South Africa which trains the whole of the South African cretin is surrounded by these very you know elevated regions that are most likely created by a lecture in the middle here we have these bizarre catchment that really look like Madagascar Madagascar for reference and also as you may have seen during the evolution they are extremely dynamic much more dynamic there's much more capture in this situation compared to what happened either or yet so here we go it's the thickness of the elastic lithosphere kermans microendemism in Madagascar I won't go through I'm already on the time thank you for your attention thank you Jean any questions so great talk Jean you reviewed how different transfer processes have different eigen frequencies and so they have different eigen frequencies right there's a resonance you have an eigen frequency depends on some parameters in some complicated way but in the end there's a frequency now you know if you have something like that right physicists things of course well we just can look at absorption bands and then we know what processes you made the case that that's not possible so I wonder if you could elaborate a little bit on that because why not look at you know forcing we understand like you did in your synthetic models and then look at the sediment record and then divvy out what is the dominant process so I guess you indicated that these eigen frequencies are too close together the overlap there's a wide spectrum why why is that not possible now the point that I said is that it's not possible if you look at the globally integrated signal but if you go at the bottom of any mountain belt and look at the sedimentary flux from the out of it in response to the climate then then yes yes you you will be able I mean that was the purpose of meeting like that and now I published a paper in 2014 where we looked at two geochemical proxies for climate change and erosion we tried to make a point that constrained us to empower law very nice talk thank you so I actually have two questions but first one is kind of more technology so you mentioned you use a game function so what type in function you are using do you think different type in function will influence your conclusions kind of things and the second question is you talk about the variability and mean do you think do you consider like an extreme once like a landslide and debris flow will influence your configuration about was it in condition so the first one the the gain function that we derive that we use are derived from the process if we go back to these gain functions okay are simply derived from the differential equation we have assumed for the function so that's in your stability analysis and the second question is on variability uh yeah uh Eric could respond but I can if you want okay so so the the the key yes you you write a thing conclusion we come to should be and we haven't really done it for each process but should be applicable to any process in which think of it in which there is a threshold and the forcing forcing mechanism when it's painful whether it's you know fall or other things it has a strong variability but it will only matter as I said earlier if the mean of your forcing is close to the question so you can think yeah of landsliding or any process that has a shun killin here um thanks for a dynamic fun talk I want to there's an opportunity maybe to connect to the um the way that coupling might happen one of the things you explored with the gain functions was looking at the el a varying and could you get an increased sediment flux out so now it seems to me you can't sustain that over long term unless for some reason there's a tectonic response and increase the uplift in other words the increase in sediment flux would just be while the morphology adjusts from a fluvial one to a glacial one and it's going to be over so maybe there's an opportunity for testing if there is a dynamic response by looking to see if that is sustained over long yeah so so thing but I fully agree with you that any response step-wise climate change is going to last only a period of time that is coming back to the delivery uh that timescale if you can measure it I think I think telling us a lot about the process I think for personally I think we had a discussion doesn't really really matter how you put things in equation what really matters is that timescale and from a tectonic point of view if you want to really model the shape of the landscape before but if you're really interested only as you say that there are going to be a tectonic feedback tectonics going to change that that's really old even climate change is going to change the topography by that much and over that time step you could even dream of highly simplified original model I believe it was quite a good job if you want to connect them to a really complexion model I have a comment to make as you may know I have three positions more interested in doing modeling surface points linked with tectonics climate life contacting okay we do thank you again because it was great okay the the beer should come I think


Hey, what's going on guys? Welcome back to another video. So in this one, I want to show you a Shopify store that is currently making approximately $25,000 profit every single year. In the grand scheme of things and modest achievement, however, I feel like it's really important to show and feature stores like this, so you guys can see exactly what it takes to get those level of results. I put out a poll on my channel a few weeks ago, asking people why they got started in dropshipping, how much money they wanted to make. Over 70% of the people that voted wanted to make over $10,000 every single month in clear profit and that was the reason they wanted to get started in this business. So by featuring stores like this, it can help kind of give you realistic expectations. So what I'm gonna show you today then essentially is draw back the curtain and show you behind the scenes of a store that's making 25 grand profit per month. I'm gonna show you the products they're selling. I'm gonna show you the ads they're using to sell that products. I'm gonna show you the Shopify store and also gonna be taking a look on Flipper as well so we can see the background contextual information and what sort of platforms they're using to sell their products on and essentially so you can see everything that's involved in a business that makes 25 grand profit every year. That way, depending on what your goals are, you will then know what it will take to get to that next level. So if you want a business that makes 50 grand profit per year, then you need to go above and beyond what you're about to see today. Okay, so here we are then on to Flipper.com. The company is called the SoundPod. The site age is 11 months with a monthly profit of $2,200 and it has a profit margin of 28%. Given my experience in the drop shipping space now since 2016, I can tell you 20, 30% is about average. I also think the fact that they've only been in business for 11 months and they already got a very profitable business is a great achievement. It's something not to be underappreciated. I don't know of any single job out there that would give you a two grand per month pay rise in the space of 11 months. And even while 25 grand might not be a massive amount of money, given the return on investment, time-wise it is a massive amount of money because to run a business of this size, you're literally talking three to five hours per week. Okay, so some background contextual information. Customers, this brand acquires customers with TikTok ads. They target the United States. They target a male and female ages 18 plus. So pretty broad in the grand scheme of things. If you're based in the UK, if you're based in Australia watching this, then essentially what you have in front of you now is a China-tested product, potentially presenting an opportunity to take it to a different country and replicate the sort of success that you're seeing here. However, given the fact that they've only made 25 grand profit, that still pretty much says it's a very unsaturated product. If we take a look at their revenue and profit then from the past 12 months, we can see obviously they had some really good months in those Q4 months. Typically that's when every e-commerce business will do their best months. If I actually get rid of this tool here, we can see in December they did 22K with a profit of 10 grand, which obviously is a nice amount of money to come through the door during those months. And what's actually really interesting is in January they actually replicated that success. So typically you'd see a massive drop off, but they actually replicated that success with a profit of just under 7,000 US dollars. And then after that, it really dipped down. It'd be interesting if I was somebody who was interested in buying this business, I would be asking why there was a dip here. Why didn't they continue to scale if sales were still strong after the Q4 months? That being said, it's still an active business that's regularly turning over as we can see. So 4 grand, 10 grand, 7 grand profit of 1, 7 and 2, 4. So there definitely is potential for this business. Once I show you the Shopify store, I'll also point out some of the things they could probably do differently to build on top of what they already have. So here's a screenshot taken from their TikTok ads manager. So we can clearly see the proof here of what they've been doing. Their total cost off the ads that ran up to this point is 29,000 US dollars. Their CPM is in and around $7, which is very good actually for the US. They have a cost per click of 0.43, which again is pretty good. It's better than average for sure. 3,000, 3.7 million impressions I should say. 67,000 clicks and a 1.8% average CTR, which again is about right. What's really interesting here is their best performing in terms of CTR campaign wise, they're only spent $30 on it. And the most money they spent here was 22,000 and that was on a 1.78% CTR, even though they have this one here, which is 2.1 and they have this one, which is almost triple this one on 4.39. However, they only spent $30 on that one. So again, that being another question I ask is why didn't they pursue the clearly better performing ad here? Now you might be thinking, because it had zero conversions, after $30 that is not enough money spent, come to any kind of definitive conclusion of whether something's gonna work out or not. So again, that would be another question I would ask if I was interested in buying this business. Before we move on guys, very, very quickly, 30 seconds of your time, I just wanna make you aware of the free training that I've recently created. It's one hour long, so when you've got the time to put aside, bring a notepad and pen, it's on demand so you can watch it any time you like and it will literally take you through every single step you need to take to take you from day one as a complete beginner with no idea what product to sell and it will show you every step you need to take to launch that business and make your very first sales in less than 21 days. Like I mentioned, it's 100% free, won't cost you any money. All I ask is for your email address and that is so I can send you this. This is my personal hand-picked selection of 194 profitable product ideas for 2023. So if any of that sounds good to you and you wanna grab yourself a copy of that, after this video just check out the top link in the description down below, cheers. Let's jump on to their Shopify store and then see then the setup behind a 25K profit business. So the first thing that stands out is there's no branding. Obviously they have the logo here and the logo at the top of their store, but there's no custom packaging, there's no logo on the product at all. So again, if I was somebody considering buying this business, that would be my first and next step. So they clearly have a validated idea here that's working. So why not reinvest that money into the business into a branded product and branded packaging? So you come across as more professional, more legitimate and ultimately convert more people. As we take a look at this section here, it's quite busy, there's a lot going on. I'm a really big fan actually of product descriptions or the initial what you see on a product page being like this because it's less typical to a default Shopify theme and it kind of helps set you apart from any kind of typical dropshipping business. What it also allows you to do is answer all of those questions that a customer needs to know about a product before they buy it. So for those people who are in a rush or those people perhaps are prone to being distracted or those people who pretty much come to the website already to make a purchase, they can see what it does, they can see the kind of benefits or why they would use it, they can see some more features and benefits of it, the sorts of things they would want to know and then just hit the add to cart button there and then without having to try and get them to read a ton of information about the product and sell them on a product because all the information and kind of FAQs if you like have already been answered at the top here. I'm gonna get rid of my tool here so what I wanna do is play you this video. This is actually a really nice video. In fact, I'll turn up the volume so hopefully you guys can hear this. So as you can see, hopefully you heard that. It does a really good job of showing exactly what the product is and it's kind of like before and after the effect which really helps sell the product. As we move down then one thing you might notice is there's an add to cart button there, add to cart button there, add to cart button here, add to cart button here. It's not a typical default Shopify theme. This will be an app which they have installed to allow people to do that and it's a fact. The more add to cart buttons, the more buttons that are on a page, the more clicks you will get which is only gonna help increase your conversion rate. So by having certain things like this on your Shopify store can definitely help. So we have a review, a social proof section here. We have the different features and benefits laid out quite nicely. We have again, more features and benefits, more social proof here. You may have noticed this lady from the flipper.com listing as well. So this has to be the main ad they're using which I'll show you in a second. They have kind of like why you would choose sound pod versus anybody else. I really like this feature. This is something I use myself quite a lot actually to kind of help differentiate my businesses from anybody else or any other competition selling the same products. Another add to cart button and FAQ and then of course they have reviews at the bottom. We can see then they're selling the products for $40. If we jump on to Aliexpress, we can see this is a slightly different version. This one looks like the same. You can buy it for five pounds plus five pounds shipping. So 10, what's that $15 delivered? So $25 of profit there. Looking at about $10 for this one. So $30 profit there. Again, about $10 there. So in and around sort of $25 to $30 profit depending on who your supplier is. Again, another reason to private label and bulk source these things because you could drive down that price probably by about 50% making your business a lot more profitable. If we come onto their TikTok page then we can see they've got lots and lots of different creatives. The one I'm going to show you today though you guys can obviously go and check these out yourself. So don't have to make this video like two hours long. This is the one with the most views. This is the one with the girl from this image. It's also the one that's advertised mainly in their product page as well. So let's take a look at what creative this is. And again, I'll put the volume up here so you can hear it. Guys, I just turned on the sound card. It turns anything into this speaker, we'll do this. So a really nice video there. Obviously someone who's very confident probably does this for a living on screen. I'm talking about the product showing the before and after and really demonstrating how effective the product is. Does a really good job. And again, she's doing it multiple times just to really kind of drive home that point of what it does. She quoted the brand there as well, original content, which is nice. And also something else that she does which won't be random either is the fact that she shows how to use it in many different settings. How you can use it during the laundry, how you can do it when you're doing your makeup, how you can do it when you're in the kitchen, cooking something. It does a really good job of showing the many different applications for it. So they have in its own right a very successful business making 25 grand profit per year. To summarize their setup then, so they obviously have a Shopify dropship in store. They are selling a product which they're sourcing from a supplier in China and they're using UGC TikTok ads to get their customers. So generally speaking, a relatively simple setup that everybody watching this video can certainly replicate. So definitely some food for thought in this video. If there's anything it could do to help you get started just leave a comment down below. I read every single one so I will get back to you. And of course if you want that extra help in hand make sure you check out that free training in the top link in the description down below. Thanks for watching guys. I hope you enjoyed it. I hope you took away some value. Don't forget to comment, like and subscribe. And I'll see you in my next video on Wednesday. Cheers.