You get like five seconds of hot pulsing techno music when you come up on the stage. It's great Hello, everyone. This is after death. My name is Sam Fippin I'm a member of the aspect core team and a architect and interim manager at digital ocean Let's get started if you've worked in software engineering For any time at all. You'll have heard tales of eventually encountering something that looks like this or This or Where the service is trying and trying and trying and then you do a deploy and it gives up the ghost Earlier today not one hour ago this status page went up on digital ocean and There was a non-zero likelihood that I would be doing incident response right now instead of being on this stage Which is fun? It happens The things we build fail. It's inevitable the systems that we work on aren't perfect and neither are the people and if you hold a Pager for any production system. I'm sure you know the feeling of sadness and frustration and What's going on? Clickers doing two inputs at once and for an anger at being woken up at three o'clock in the morning Why did my team not care enough to not make that bad deploy? Why is my company not protecting me from failing infrastructure? What on earth has gone wrong? As I mentioned I'm a manager now and I have a covenant with my people that says that this will be as rare as possible And I will empower them to prevent it from happening as often as we possibly can and the thing about digital ocean is It's not a rails monolith. It's a big Complicated services world and I can tell you from experience that those services contribute just as much to the downtime As any like other factor that we have in our infrastructure and this happens, right? There's really nothing we can do about it no matter how hard we try computers will Break the disks on your servers will fill up a human will write a bug and play it to production and Well in the worst case the underlying hardware literally catches fire So what can we do about this? What tools do we have at our disposal to make sure that as often as possible? We are not being affected by these incidents that we're getting better and just spoke to this very eloquently this morning And I'd like to sort of throw my ring in the hat many of us when these things happen do a post incident review or a learning review or Something else, but the truth of the matter is a digital ocean We call them post mortems, and I like this metal font So that's that's what we're doing and the thing about doing these reviews is that it gives us a Literal system from learning from the failures that we undergo and with that system We're able to really patch up and find underlying causes and fix them We're able to work with our organizations to understand what risks we might be accepting when we build the systems That we work on Everyone here who has the ability to deploy to their production Environment is accepting a risk every single time they do it but we're all asked to ship features basically as quickly as We can and while that's a really valuable and important thing It's important to note that ultimately if your product is not available You can't have any customers and so this is the thing that they care about To and by no means am I here to tell you that I think I have all the answers I used to be terrible at this and Working on a cloud scale system for a couple of years has taught me a lot Really, we only started getting good at this a year ago And so I'm just gonna share some reckons here if you disagree Feel free to have a chat with me afterwards. I love talking about this stuff. So To talk about this I'd actually like to dig into how we post-mortem at digital ocean to give you some understanding of How I think about the process and to do that I'm actually gonna walk us through digital oceans post-mortem template every time we have any kind of severe production incident One of these gets kicked off and an engineer starts filling it out We give the incident a name we take the date of when the incident commenced and then we assign it a severity Severities are useful because they tell you how to respond. What kind of response is necessary and who should be Involved and a digital ocean. We have a five-point scale that starts at Sev zero The Sev zero incident a digital ocean means that there is a Critical impact to the business that likely means that everything will end if engineers don't work quickly to fix it These are the company-ending events that you all dread and fortunately they're incredibly rare I think we've had two since the scale was defined but in a Sev zero incident Every engineer in the company multiple directors Infrastructure people a coordinating response literally everyone bands together to try and make the problem go away It's the worst thing that can happen. And so it gets the highest form of response as a result a Sev one is a major global outage or an entire single product not functioning if a data center is gone that's a Sev one and Again at this point we get executives involved There are usually multiple teams coordinating But we don't believe the business is going to end if we don't deal with it immediately and that's sort of the core distinction Sev two is the lowest severity of incident that we wake people up in the middle of the night for and that's basically when a Single product has stopped working or something else has gone wrong that's severe but not severe enough to wake everyone up Usually there's one engineer here doing response coordinating with our support and communications teams And then we have three and four which are basically just bugs and defects that go into Jira and get fixed at some point In the future hopefully If your organization doesn't yet have a mature incident response practice having a severity scale is a really good place to start It allows you to communicate with your company about What the incident classes that you have actually are how bad they are and what you can do to fix them? Who needs to be involved what the procedures should be and that sort of thing? This is pretty standard across all incident response practice And so this is a really good thing that you can add to your tools if you don't have one already the next thing that I want to focus on in our post-mortem is the timeline and the timeline is really one of the most core and critical pieces of These documents it's the facts of everything that happened that caused the incident to occur from the very beginning to the very end Everyone who is involved all of the systems that went wrong logging monitoring and those sorts of things But there are a couple of rules Timelines must be hard facts you shouldn't include any analysis or emotion What people did is valid how you thought about what you were doing is not this is entirely just to get an understanding of what happened The sort of pro tip that I have here is if you're dealing with an incident like this You should complete your timeline as close to the incident as possible And the reason for that is that logs rotate out graphs expire met memories get fuzzy And so depending on how severe your incident is we typically either complete this as soon as response is done or the next morning afterwards To steal a line from Jess Document everything no matter how small what the thing may seem it may actually be a critical underpinning of a later piece Of analysis and so the timeline is a really valuable tool This is what a typical incident timeline looks like it includes timestamps. We use UTC some people use their local time zone that's up to you and This is kind of a straw time timeline This is not a real incident timeline because if I told you what we did in real incidents that might be problematic And you might want to stop using us This first line here is a deploy was made which introduced a bug and here I'm really trying to make the point that your incident starts before you notice it a Deploy happens a server fails a database row gets corrupted at that point No human has actually noticed that something has gone wrong But something has gone wrong and so when we think about incident response We need to document when our incident actually started which is a separate point to when we notice and when incident response begins Next line is saying that our graphs indicate something went wrong And you'll see here. I have a like a little one Wikipedia like citation mark in all of these lines You should add some kind of evidence that explains what the That explains how this timeline was created be it a screenshot of a graph screenshot of logs a link to a slack line a link to page duty That's all really good and again remember log systems and graph systems expire Make sure you capture it permanently the other piece of advice that I have here is Observability is really good and you should have some the rails community as a whole I think is not great at building good observable services and thinking about what you can do to add Graphs metrics logs to your system is a really good idea a digital ocean We make very heavy use of Prometheus and Rafaana, but if you're hosting on Heroku They actually have a really great dashboard built in and this is probably good enough for most smaller rails applications This is really important the actual impact of your incident what happened how many customers how many tickets what data was lost Can usually be reconstructed from this information? And so thinking carefully about the observability is useful both for incident response And is just a generally healthy thing for you to do as well as helping you fill out your post mortems The next line you'll see is that Sam got paid for bizops internal tools Which is the name of my team at digital ocean with a link out to pager duty and pager duty is great If you have another paging system, that's cool I but I'm really only familiar with pager duty and here you'll see this is what a pager duty incident Looks like and the thing we're trying to capture with this line is When did we actually get a human involved and that's really important because that's the earliest point in your incident That you could possibly begin clearing it up if it's bad enough that it's not going to heal itself And so Documenting not just when the incident started and when we first noticed it But also when response began begins to inform on a wider basis how we're dealing with incidents that occur at our company Another factor that's really important is who alerted this person a digital ocean We're fortunate enough to have a 24 by 7 support and operations team that can wake us up in the middle of the night And is able to make technical Investigations if you don't have that you need to have some kind of machine Alerting and a thing that we have generally observed is that if a machine alerts us our incident Closure time is much faster than if a human alerts us And so having automated alerts is a generally healthy operations practice And it's something that you can build once you've got that observability in your application So We then come to our next timeline, which is the human acknowledging the page and that's another sort of useful and important point Because that's when this person that we reached out to actually began responding If you're paging someone at three o'clock in the morning, and they're in deep sleep It might take them 40 50 minutes to actually wake up and realize what is going on But understanding that time and that factor is really important But as well as this in our timelines where we have multiple responders We document where everyone got pulled in and what they were doing and so the idea here is that This part of the timeline really scales to everyone that is involved in incident response and should document who was involved and is doing what There's one thing that I have here as a super mega pro tip and that's announcing presence in the slack If you're doing incident response make sure people know you're doing incident response What you're doing why you're doing it when you're doing it and communicating frequently as a general rule I ask primary incident responders that I work with to update in slack at least once every five minutes with exactly what they're doing And that's not to sort of micromanage and continuously check in on them But that's sort of as a heartbeat to make sure that we're still fighting the incident even if they don't have a status update It's just useful and healthy good practice And then finally you have whatever you do to fix links to GitHub or whatever it might be and Grass that show resolution and here. We're really asking the question. How did we fix it? Can we prove that we fixed it? Do we have confirmation of that? So that's our timeline and that may all seem fairly obvious But getting that documentation exactly correct is really really valuable And if you have this ongoing log of everything that happened in all of your incident response You as an engineer will be able to build up patterns of what happened why it happened and how your team is fixing things So let's move on to the next section and this is the single most important part of Every one of these post-mortems and learning reviews the root cause section Now the thing about the term root cause is that it's inherently Misleading it implies that it's singular and it also implies that it's the closest thing to the incident That's how most people I see naively fill these out and I'm guilty of having done this in the past myself This is one of the earliest root causes. I wrote a digital ocean I absolutely don't expect you to read all of this text You basically need to get the idea that a deploy happened bad queries too much scale and That was a sort of outdated deploy load This caused the incident. I deployed something that was too much scale stuff went wrong That seems perfectly reasonable at a first glance But this doesn't actually help us solve any problem and to think about how we can Let's dig into exactly what's written there a little bit more There was an Atlantis deployment with a scaling factor of 10 where each instance had a default worker thread of 25 Select statements being issued to prod my sequel 1a Didn't have an index in place and an outdated version of the worker These four things are what should actually be called proximate causes These are all close to the incident that happened But not actually causal if you really really dig And so let's do a real root cause analysis on these four factors and have a better understanding of what is really going on To start with let's take a look at these select statements to prod my SQL 1a Quote to understand this you need to understand a little bit about how database architecture works at distortion We have one ginormous my sequel cluster Called alpha and it's made up of a bunch of servers My SQL 1a is the primary and accepts reads and writes and all the other ones are read replicas they only accept read queries and The point of doing this is that we can alleviate Traffic from our primary by moving those queries on to the followers and it's a generally established best practice That if you know a transaction is only going to issue selects to use a follower instead of the primary database If I was going to write this today I would probably say that Atlantis was issuing unnecessary reads to the production primary But also leave a training note that says words the effect that we should be careful to ensure to use the read replicas Wherever possible this reinforces that best practice and also creates documentation for other people to follow Going forward as a way to avoid this problem We had this thing about Issuing queries that didn't have an index and this should be fairly self-explanatory Have indexes database indexes are really good. They make everything faster. You almost always want to have them and So well, let's think about we have this missing index But but a query made it all the way to production without us having any ability to detect it was going to cause a problem You could argue absence of testing for full table scans. You could argue unrealistic testing data You could argue and this is a really common one staging environment doesn't reflect production How many of you have staging environments that you genuinely believe are a good reflection of your production environment? That can be one of the most useful silver bullets against operational failure. Try building one. It's a really hard problem Running an outdated version of the worker may have contributed to load Um Complex production environments when you run the internet are hard Basically, we used to have this production environment Where these servers their sidekick workers were deployed with chef and Capistrano and there were four of them then the great Kubernetes migration of 2016 came along and We dockerized everything and then we deployed 10 10 of these pods to our new shiny Kubernetes environment The problem is we didn't burn these chef nodes initially because we were worried Kubernetes wasn't gonna work and The deploys weren't in sync So we had like old revisions of the software in production at the same time as new revisions of the software and like That was a nightmare. Don't don't do it. It's a really bad idea. I would note this and root cause it now as Running parallel chef and Kubernetes environments exacerbated the load problem lack of procedure around decommissioning chef nodes and in general Having a little bit more maturity around thinking about big complex production transitions like that Which is not something that many of the people involved were experienced at doing at the time And then finally this like scaling factor one This is really tricky because the truth of the matter is if those other three things hadn't gone wrong This wouldn't have been a problem. And so here we have a really good example of like An exacerbating factor that if we'd fixed all of our other problems wouldn't even have been exposed But it's worth looking here. We didn't have a great reason to scale this application up It was just like Kubernetes infinite scale. Hooray Oh, no productions on fire So I might just note a sort of training procedure issue about when you change scale of production applications and move on So let's talk about the kind of thinking that we did that because it's a very specific kind of thinking We're not looking at what directly close to this incident caused it We're asking the questions what in our system made this incident possible The way I phrase this to newer developers is look for procedural policy systems and training problems That are affecting your organization and macro ways that you can fix them It's never ever a single person doing a bad thing that causes a major production incident like this Not one time everyone that you work with is trying really hard and Assuming Lee that's not a word Is doing their best? And so I always ask questions. Well, given that this person was trying as hard as they could What safeguards were missing? Why didn't they know that this was a dangerous operation to do? It's it's never a person. It's always a system and I'm just gonna Shamelessly directly quote Jess from this morning. If you show me a team that punishes people that makes mistakes I'll show you a team that makes a lot of mistakes. I think that's really true And it's like sort of the perfect summary of my talk This takes time getting really good at postmortems getting really good at healing the problems in your organization That cause these issues to occur takes time and practice and care and attention The truth of the matter is when I look at the people who write postmortems around me The ones that produce the best ones are the people who give the most of a damn about working with their colleagues the ones who have empathy with everyone that they work with and the ones who understand the pain of not being up to sleep Through the night because the computers aren't working So that's most of what I got um if you want to steal our postmortem template I will be sharing it on the internet because that seems like a good thing to do I am hiring literally for my team. So if you want to work with exactly me That's that's a thing that's happening That's yeah, that's the least chilly way I could possibly try and hire you is if you like me come work with me That's all I got. Thank you


Okay, as Irina said, so I've been working more at a sequence stratigraphic timescale, sort of the hundreds of thousands to millions of years, which means you have to sort of very much abstract some of the processes that are going on and all of the detailed process work into what's the long-term version of that. And so, as Irina said, I've been working on this with particularly with Eric and Greg here, trying to build this into into LandLab. And I've also been helped by David Olegon, who at the time was a Bronx High School of Science student to help me with Python programming. And he's now a, he's finishing his freshman year at MIT. So the original, and I should say this project actually got its its genesis the last time I came to the systems meeting and met with Eric and Greg. And we created a proposal to work on an area in in western Turkey. So in western Turkey, Wems is Arabia is hitting Turkey, which is is moving out west. And then you have a subduction zone here that's pulling this way. And in here in this kink, what's happening is Turkey is undergoing north-south extension. So this ragged edge, western edge of Turkey, is a result of north-south extension producing all these bays and promontories. And the collaborators that I have in Izmir collected some beautiful high-resolution seismic data there, where I hope you can you can see where there are various sequences that we believe are the 100,000-year cycle mapped out. And some of them, like if you follow this tier one, you can see it going down due to the subsidence because there's a riff basin just out of the page here. But then it comes back up and here's the clinaforms up on the shelf and then it's dropping down. So the sequences have obviously been warped by the tectonics. And so, I mean, obviously this is subsiding, but is this part simply stable with other parts subsiding around it? Or is this actively uplifting? This is certainly related to a transpressional fault across here. This is the location of the line. And so, the idea was to use sequence stratigraphic models to model the stratigraphy and try to back out the tectonics and relative subsidence rates. And, you know, I previously, as part of Stratiform and EuroStratiform, created a sequence stratigraphic model that was the basis for this one. And it is, you know, a completely undergoing bit rot. You know, it was built with a GUI that no longer works on any modern computer. But some of the basic stuff underlying it, you know, it produced models that at least made pretty pictures where you can see things like how the shore face is prograting with the sea level fall, as well as the depositional shelf edge. And you get interesting effects like, you know, even as sea level rises and the shoreline starts regressing, the depositional shelf edge can still be prograting, you know, being driven by the sediments that are being eroded by the transgression. The pictures and it like many other models can produce sort of the three main erosion surfaces that you see in stuccointigraphy is the subarial erosion surface, which is the sequence boundary, but it also produces a marine erosion surface that's usually mostly bypassed, not very much erosion, across the shelf when sea level falls and a transgressive ravinement surface as the shoreline advances and erodes across the top of it. And we're able to use that in a couple of places while it still worked. So in the Yellow River on, you know, sort of the tens to hundreds of thousand year timescale and some models and the millions year timescales, they said it, you know, is is gone in terms of effect will be able to run. And so the idea was to create this now within LandLab and take advantage of, you know, Python programming and and all the capabilities that are built into LandLab to hopefully make it easier to port it to something that's that's usable. Now LandLab is generally a plan form based model. I'm at the moment still doing a two-dimensional cross section so the way we do that in LandLab is by making a grid that's a three by a large number. And we set, very simply set the closed boundaries on either side so all the sediment has to simply go through the the center line. So that's what we we make it and yeah advantage in LandLab is I can simply tell it these are closed boundaries these are open boundaries and it worries about all the boundary conditions and making sure that everything's correct at them and saves a lot of of very painful time and effort in the programming. So here's for instance an example that's produced by the the modern version where on this I've just covered colored the fluvial, short face, shelf and slope. It's done with a slightly asymmetric sinusoidal sea level curve where you can see the again the shoreline prograding out the shelf edge moving back and forth not quite in sync with the with the shoreline. In this I'm still not completely happy with the algorithm I have for picking what the shelf edge is. It's not quite the easiest thing to to do simply and this is so this is a running with 100,000 year cycles so there they're actually 6,000 layers of sediment in here that it's kept track of here. I can also produce wheeler diagrams so this is looking at distance versus time and the blue is showing you where there are sediments that are being produced with the darker colors being the thicker sediments and the gray represent representing sediments that were deposited and then were subsequently eroded. And the one on the right is is similar but unlike the older version we have now a very simple but working two lithologies so we just have sand and mud yeah producing a very fairly simple variation in lithology across the model okay here for instance is a model where instead of a relatively sinusoidal I put in the oxygen isotope curve from bisecti and ramo where I've just converted it to a directly to a sea level curve even though I know their temperature effects and it's not the actual sea level curve and you can see now that it you know has these tongues of of prognation that are being produced with each cycle because generally each cycle is asymmetric with a very slow long slow fall and a very rapid transgression and so you get these long tongues that come out and all these little wiggles in the sea level curve are producing a lot of back and forth within the prognation with all these erosion surfaces that you may or may not be able to see from from a distance and again you can produce a wheeler diagrams going to shoreline and the and the shelf edge with the shelf edge again not quite being in sync with the with the shoreline but obviously a lot more a lot more complicated with all the variations in in in sea level so the the basics of the model includes well the the big thing is is how to represent the sediment transport and deposition across the model which I'll go into in a in a few minutes you know there are sediments coming in the landward side where you specify the volume of sediments percentage of sand and then that sediment influx can vary with with sea level for instance you read in subsidence through a file sea level either through a simple sinusoid or or or a file and we have flexural isostasy compaction we have the routines already exists but we haven't connected it up I was afraid to break the model before this meeting because I'm sure once I do there will be all kinds of things that will have to be repaired and so there it's still in a state of of of flux where there's a few more things that we have to you know add in adjust and debug but I hopefully soon it will be you know publicly available through through systems so the input parameters come in through a yamo file which is yet another mock-up language you know so there are various sets that talk about how to time step through the model the shape of the grid out at where to output the file which come out as net cdf files you know where to get the the symmetry and the sea level the subsidence various parameters for the sediment transport and the properties of the of the sediments probably more parameters will get added as time goes on so breaking this down that's real awesome essentially on the coastal plain I follow Chris Payola's long-term model of essentially a linear diffusion with as you you know increase the river flow from precipitation towards the coast the diffusivity increases towards the coast and you can set a basin scale for how much and then I use alanita rotors model or a non-linear diffusion for the shelf so the transportation on the in the marine regime is obviously some kind of non-linear advection diffusion scheme and people like john swenson used a more effective model I've used a more diffusive model I think both of the models produce a lot of very similar things because I think a lot of the features that you see at the sequence or at a graphic scale are relatively robust relative to the details of the model but this model does have some nice features that I'll show in a moment the you know failure on the slope we haven't yet added in but we'll probably add in some kind of simple scheme for for collapse of the slope when it when it gets too steep and I'll show right now we have just a almost a more of a placeholder for the for the hemipelagic mud deposition that I really like to improve and I welcome anyone who wants to help me improve it to join me in this so so onshore they said it's simple a non-linear diffusion with a linearly increasing diffusivity when I go and and forgetting the lithology onshore we it's a this is a scheme that was produced by Juan Fidelis and Chris Paola we're essentially as we you see earlier today you know channels tend to accumulate the sands and grow faster than the floodplain and so based on the model parameters it calculates a channel boat aggregation and a floodplain aggregation relative to the mean sedimentation rate and you know and assuming that channels evulse when they become super elevated it calculates then the number of evolutions per per time skip to figure out how the sand and mud are distributed and calculates a an average sand fraction you know this is a two-dimensional model so this is essentially averaging across this width of the of the of the basin offshore it's a said it's a this non-linear diffusion that alanita rotor devise we are basically diffusivity is a function of both distance and and water depth with some you know with small terms added in to help keep things from blowing up near the near the shelf edge and then I've added in an exponential decrease in the diffusivity below a wave base the one of the nice things about this essentially makes a dynamic diffusivity that varies as the as you prograde and and and and regress so what you find is that when the system is prograding the shorefaces tend to be taller and steeper when they're transgressing they tend to be shorter and flatter and so when you transgress you often just a road off the top of the shoreface and the lowest shoreface tends to be preserved in the upper shoreface eroded which you know fits with what you see in a lot of outcrops so i've been so you know i don't know if there's a better schemes out there but this certainly produces things that that seem realistic to me um and then that exponential decrease in diffusivity once we pass um a wave base that you that you set um enters the sediment transport and gives you this rollover of the depositional shelf edge or the the clina forms or pro deltas or whatever you want to call them um the only problem with the scheme as it's implemented is that then um very little sediment goes past the clina form um and so the idea is that we will take the amount of mud coming out of the river mouth and use plume to calculate an initial distribution and then need some kind of advection diffusion scheme to distribute that properly um and i could use some help from people who know about those kinds of processes right now as a placeholder what we've done is just a simple scheme where um the mud um so it's getting deposited from wave base and increases and then feeders out as you go farther from shore which makes a nice tail onto the onto the system but clearly could be you know very much improved to produce something a lot more more realistic so here's um that same example again and here's a uh just showing it in a in a movie let's see what's going up so you can see silver going down and the shelf prograting out in front of it um when the shoreline turns around you actually continue to probe for a little bit and then it and then it comes back in you can see the shelf is having going back and going back and forth um so this is just taking that that file um the output of what the final stratigraphy and since compaction isn't in here i'm able to invert and calculate the stratigraphy at every at every time step and run through it you can see the the sink in here that's due to flexure um so we have flexural isostasy where you can set the flexural rigidity where if it's stiffer the deflection from the weight of the sediments is broader and if it's weaker it tends to be larger and more closely focused this is a fairly low one but one thing is at this time scale flexure is not instantaneous right so you know at best it would um go on at sort of a glacial isostatic time scale you know of sort of thousands of of years but you know the the actual gia response is also wavelength dependent so the time scale should be longer when you're going down to a basin scale instead of a instead of a you know ice sheet scale um and that has the potential to um have have influence and i think you know estimates of what the time scale should be is not very very good so initially we put on the sediment load um right now we have nothing initially you could put on just a pure elastic load uh deflection which would be uh relatively small and then it sinks into it and so like here's something where i have a relatively short time scale which is what i've been using in most of my models um but if for instance the response time becomes very long becomes long um comparable to the um sea level cycle then what happens is you know this is programing out and the area behind here is still subsiding you know when when sea level was up here and you were depositing up here it wasn't um getting isostatic load and so you can see more fluvial deposits being accumulated because it's subsiding after sea level has after the shoreline has gone by and you get sort of the opposite effect um in here where it doesn't subside and then it's um it's subsiding during the transgression and so it's getting more of of deeper waterfaces instead of instead of shelf so this is something where the the response time and of the isoscecy and sea level cycle can interact when they start getting close in in time here's a another couple of experiments i did where this one is with constant a constant sediment input and on the top and bottom i have ones where over here sea level uh the sediment input decreases as sea level falls so that might be something like in the Ganges Barma Putra Delta where i'm working where when the monsoon shuts off the sediment supply they are probably decreases by an order of magnitude and so you know there's more sediment at the at the high sea level and you can see the larger more fluvial and you can see the shorefaces becoming thinner and much less of the of the shelf because there's less sediment coming in then here's the opposite where sea level is where sediment is increasing as sea level falls it might be more of like a Mediterranean climate you know and so we're seeing a lot less fluvial and a lot more you know sediment prograting things at the at the low stand so again that's a user a user-based parameter and so here is another movie showing a 600,000 year run with you can follow the little bouncing red ball where you can see there's a lot of jiggling back and forth with the with all the twists and turns of the oxygen isotope curve and you can see you know coming in and you know every time it tends to go back and forth you can see lots of erosion surfaces and and more complicated pattern of advance and retreat it went back and forth you can see the little tongue from the first part of stage seven versus the main part of stage seven against some some kinks and then here comes the big final Holocene transgression across the whole thing and so you can see again these these multiple tongues related to the long-term sea level curve if i take the same model and run it not with the last 600,000 years but 600,000 years earlier in the Pleistocene when the 40k cycle was dominant i get something like this where it's lots of very fast patterns and not as much variation in sea level and you're tending to get much more vertically stacked fluvio shore and shelf deposits so i can actually yeah so you can compare the two you can see overall the over the last 600,000 years sea level has on average been lower and so the entire package is a little bit farther seaward in here and then for another variation i did this these models are run with a fairly rapid subsidence so that these nice tongues of sea level are are fairly are visible and not well you know i'm off off to a new jersey where you know there's almost no subsidence and a lot of them are just eroding and and repeating each other and amalgamated so here's one where i i cut down and got rid of the the shoreline and smoothed out the initial photography and you can see this much more of an overall progradation of the system as it as it progrades across this this ramp so there's enough of a ramp to to give you this accumulation but you can see the the difference in fact i had to move the sea level curve to the other side because it was covering covering it in terms of of tectonics we haven't yet started to model the the Gulf of Cusadesi field area i'm actually going there in in in two weeks to visit the area and collect and get a kingdom sweet version of the of the project with all the interpreted seismic lines but meanwhile i also in that area been working in the Marmara Sea where what we found is in this rift basin is that there's a series of beautiful prograding low stand deltas and what happens is most of the main rift basins are here but there's an extra one here and so this area here in the southern shelf is marine now but during low stand this becomes exposed and the rivers come and dump the sediment over this fault and so when that jump over the fault is not too high we've seen a bunch of a series of low stand deltas prograding off during low stand and then it transgresses back so we just tried making a model where i put that kind of backward subsidence in here to produce these tongues of of shallow prograding shruff face it's it's not yet tuned to to match this but was more of just a proof of concept to make sure that i can make it work i also the the marmara sea becomes disconnected from the world ocean when sea level fall when the last time when the sea level fell below 85 meters and became a fresh water lake so i cut off the sea level curve at 85 meters so these are a little bit flatter in front but you can see these series of of prograding deltas which when we when we collected this data was you know a really useful discovery because that gave us these nice hundred thousand-year timelines that we can tie and get age control over the stratigraphy within the entire marmara sea so the you know in any way the you know future plans is you know whoops is is now to go ahead and add compaction i also want to make some improvements to the ability to change parameters during model run so you can you know start a model change parameters around and continue the continue the run you know for instance in my field area i don't know if that uplift you know perhaps started partway through the time period i'm i'm looking for you know or if you know there's a river capture and sediment influx increases you know i want the ability to be able to to pressing the wrong button reason and then i hope once we you know put in the last few extra things like this to release it the sds dms and github and make it publicly available you know on a slightly longer term i i'd love some help to improve the the mud transport across the the shelf in deeper water so i want to add the some kind of slope failure and and simple turbidite and i'd be happy if anyone else wants to volunteer to help join our group to you know take part in in future improvements on the longer term we actually have a proposal to couple mock persons fluid flow modeling through this so that as sea level fall sea level changes go on we can pump fluids through the through the system to look at where there's both you know fresh and saline groundwater entrained in in in in the systems was for instance off of new jersey there is fresh to brackish water up to 70 kilometers offshore in the sediments in bungalow that's where i do a lot of work the hollow scene section in in the southwest is is primarily saline but underneath in the Pleistocene there is fresh groundwater so we'd like to try to adapt this to model those those places i talked about this for at Levant and there was a suggestion well what about you know coastal deposits can you get more detail in that and perhaps an estuarine you know module or organic matter of preservation and you know the the system is written relatively modularly so that i hope you know if people want to help develop new modules that we can plug them in relatively painlessly you know and of course then you know perhaps down the road if people want to work on on carbonates and you know greg has certainly suggested 3d i think 3d will be a lot bigger job and probably require you know a whole new proposal to to go that way but you know landlab is inherently 3d and so you know since it's built in landlab it should be you know at least possible to turn this into a 3d 3d publicly available system because there are some 3d sequence stratigraphy models or models but you know some are for shorter time and longer time ones a lot of them are proprietary so i wanted to present you know where we are you know right now and so we welcome anybody who wants to come in and help us improve this or just wants to use it to yeah talk to me later thank you thank you


This research project aims to evaluate a strategy of point-of-care testing for six sexually transmitted infections, STIs, in antenatal care, ANC, in Zimbabwe. 1,000 pregnant women will be recruited from ANC clinics in Harare Province, Zimbabwe. Routine HIV and syphilis testing will be performed alongside an integrated screening package, including tests for chlamydia, neoceregonorrhea, trichomoniasis, hepatitis B, and other STIs. Individuals diagnosed with any STIs will receive treatment, partner notification services, risk reduction counseling, and referrals if necessary. Gonorrhea samples will be cultured and tested for antimicrobial resistance according to World Health Organization, WHO, Enhanced Gonococcal Antimicrobial Surveillance Program guidelines. The primary outcome measure is the prevalence of chlamydia, neoceregonorrhea, trichomoniasis, syphilis, and hepatitis B. A mixed method process evaluation and economic evaluation will be conducted to assess the except. This article was authored by Michael Marks, Victoria Sins, Cheeto Chivichiquari, and others. We are article.tv, links in the description below.


I'm delighted to be here as usual after several years and you alluded to the number of years I've been here and it's been quite a few and I should say I started in 1978. So if you do the math, this is probably 43 field days I've been at. They're all delightful. They're all good. I love the breakfast touch with Carl and the Hoarsen group here. That makes all the difference in the world. We didn't used to have that when it was the Red River Valley Potatoes and Growers Association but now it's the Northern Plains Potato Growers Association and it's what, the 75th anniversary? Is that what I saw? Is that right? And well, we won't go there but there's a coincidence in there somewhere. Anyway, I was on my way up here this morning and it was raining and it was dark and so I turned off the radio and I said, I'm going to let my mind float for a while and just see what I'm going to say when I get up here. And I was telling that to Donovan this morning, okay? He said, oh, that's interesting. He probably ran out of miles before you had a good thought, right? I'm not sure where to go with that but I thought it was a good reaction on his part. Anyway, I'm a pathologist. I usually have bad news this year. There's good news. Let's give a little update on what we've seen this year. No late blight. Of course, that's a da, intuitive aviosity. It's been too hot and too dry. The only light blight's been on organic farm in Ontario and then there's been in one field in Wisconsin and then maybe down in South Carolina or Georgia on tomatoes, okay? So no light blight. That's good news. Nothing in our area. This rain isn't going to bring the late blight to us because there's no inoculum around. That's a good news. We also have been doing some testing for soft-trop bacteria, blackleg soft-trop bacteria. We've had several samples this year, several, maybe a dozen from North Dakota, Minnesota. They've all been pectobacterium. No dickia at all and the other samples we've gotten in from other locations, mostly west of the Mississippi, have all been pectobacterium as well. So it looks like the dickia is kind of subsiding a little bit. We're going back to the pecto for the gods of the soft-trop bacteria. The other thing I want to mention, and you'll probably hear this a thousand times during all of these presentations, is that because of the heat there's going to be a lot of physiological disorders in potatoes.  So there's going to be, that heat is really going to cause a problem with potatoes, so I think you all need to be aware of that. Of course there are disorders, not diseases.


Live from Gillette Stadium in Foxboro, Massachusetts, extracting the signal from the noise, it's theCUBE, covering VTUG's New England Winter Warmer 2016. Now your host, Stu Miniman. Hi, welcome back to theCUBE. I'm Stu Miniman with Wikibon, and this is the VTUG, the Virtualization Technology User Group here 2016. It's the 10th year of the show, third year that we've had theCUBE here. Happy to have on the program a first time guest, Scott Martin, who's the Systems Engineering Manager in SBU, which means the networking group is kind of at the core of that, and security is part of that business unit. Scott, welcome to the program. Oh, thanks for having us, Stu. All right, so you gave one of the keynotes this morning, talking about VMware NSX, some of the big changes before we kind of get into some of the technology. Tell us a little bit about your background, what brought you to VMware and the VTUG. Yeah, great. So I probably saw the better part of 20 plus years working for a multitude of different network and security vendors. In fact, my last role was working for the office of the CTO of a pretty predominant hardware vendor, where essentially it was tasked with trying to figure out how do you make hardware relevant in a software-defined world. Hence, I had this aha moment, if you will, and came on board here with Nice Arrow. Yeah, so how does hardware stay relevant in a software-defined world? You know, if you're putting more and more intelligence and you're abstracting that layer of hardware, I think hardware becomes more or less irrelevant, right? Look at me wrong, there's some great technologies to maximize and manage hardware-based infrastructure, but true software-defined networking in its nature is software-based, right? Yeah, it's interesting. I mean, I think back, you know, I studied mechanical engineering as an undergrad, and it's a software world, so I've tried writing that article a few times as to the role of hardware in a software-defined world, and a lot of times it's, you know, integration matters a lot. You know, reliability matters a lot. Hardware is the platform that everything sits under. It's not like the cloud doesn't have hardware. It's not like your network and software-defined world doesn't have hardware, but there are shifts, and especially as the application and the software layer can control a lot more. I get away, what I say is there's the difference between kind of the old enterprise mindset, which was let's harden everything, let's have triple redundancy, let's, you know, build reliance into the hardware when the software can take out of that. We can have, you know, failures are okay a little bit more if I've planned and managed and done that, so that's my take.   Yeah, fully agree. I couldn't agree more, actually. So Scott, tell us a little bit about, you know, the keynote, you know, what are you talking about and how does that resonate with kind of the community here at this user group? Sure, so for, been a member of the VTUG now for a number of years, probably like five or six years now overall, and so the keynote we gave this morning was actually about helping customers understand the practicality of actually rolling out NSX within their environments, right? I think there's this myth out there that essentially to run NSX, you essentially have to rip and replace your existing infrastructure, it's just not true, right? A number of our customers that actually deployed NSX simply for micro segmentation, or they may start off with micro segmentation from there moving on to extending the reach, right? Going beyond traditional IP boundaries or even limitations within the context of their data centers. And then the other thing is around application continuity, right? So with this year with the release of vSphere 6, we changed a number of things, right? Prior to vSphere 6.0, with ESX 5.5 and below, round trip latency was tied to about five to 10 seconds around trip latency 4V motion. Now we've extended that to 150 milliseconds, now we're talking the ability to potentially span across very large geographical regions, or potentially across the country as well too if you have a decent fiber footprint. So now we couple that with our NSX 6.2 release, and now we can actually reach much further, right? So NSX 6.2 now gives us the capability of tying it to up to eight vCenters overall. So Scott, you're in the field, you're working with a lot of customers. I guess one of the challenges I've had with kind of the SDN wave in general is a lot of it, if you're not a networking person, it sounds like, oh, the networking guys have a problem, they've got all these protocols that they can't sort out and they're putting something else in there to help them fix a networking issue. Talk to us about kind of the business challenges, how this isn't just a networking issue, but what it means for the business from that standpoint. Right, I think the reason why we've gotten so much trash over the past couple of years has been right along that line is that we're actually fixing business use cases, right? So customer problems. So I'll give you a couple of examples, right? Within the context of a data center, segmentation to begin with, right? We've seen over the past two years or so, that it's not a traditional attack vectors, if you will, it's not folks coming in from the internet, it's folks coming in from secondary, tertiary, portal, gaining access to the data center. Our data centers are highly interconnected. Next thing we know, that hack has breached the whole entire data center. So we have to be more conscious about reducing that attack vector overall, right? And within a sector that granularity is as small and that risk profile is essentially as small as an individual virtual machine. And we take that as step forward. So microsegmentation is probably one of the more predominant use cases, but it's not the sole use case. And in fact, I think a lot of our customers are actually pushing us towards new and interesting use cases that we even haven't even thought of for that matter. Yeah, so I guess right, so security, who's one of those kind of tip of the spear use cases, one that I heard a lot of, we did a bunch of interviews at VMworld last year, and that seems to be a great use case there. You mentioned some other applications, anything you can point to? I guess, there are many users that said what I'd love in the networking space is to have what I have in general, which is I can just choose applications, regardless of hardware, kind of from a northbound standpoint at applications as need be. Is that coming to reality yet? Oh, absolutely. In fact, we have a number of customers that are actually leveraging, not only the context of being able to stand up everything an application stack needs from the layer two components, the routing functionality, the layer three routing OSPF or BGP, the stateful firewall services or layer four through seven services for load balancing. So everything an application stack actually requires can essentially be instantiated within a multi-machine blueprint and then put up to a self-service catalog. A number of customers are actually rolling out multi-tenant environments right along that line and they have been doing so now for a number of years. And that was the other thing is that, we look back at 2015, I think we've actually crossed the chasm. What I mean by crossing the chasm was that we've actually seen an asexical mainstream. So we've seen deployments in not only traditional very large IT organizations, but financial industries, healthcare industries, even smaller commercial opportunities as well too. Banking, healthcare has been another great vertical as well too. So we're seeing the adoption rate and it's not tied to a specific customer size either. We've seen very small customer organizations adopt NSX and very large ones as well too. Scott, can you quantify for me because your small might not be kind of small that they're there. What is kind of that entry level that you say below this? Maybe it might not be something that you want. I guess, isn't it a good fit for you guys today? Right, that's a great question. And I can only speak from my customer base here in New England. I'll give you a great example. I have a relatively small financial service provider. And essentially what they do is they provide financial services for banks. So they check the processing, credit card processing and the like. And they also provide IT as a service for those same banks that they provide services for as well too. And throughout this whole entire organization, we're talking about a rather small IT organization that comprises less than five folks total. So we're talking networking, security, infrastructure and automation and across five individuals. So they're a really small team and they were actually one of the earlier adopters because they saw the need to be able to abstain, skate and build out a true multi-tenant environment without having that complexity of having very large teams to support that environment. So that's probably on the smaller side. Yeah, but once again, they're really acting as a service provider. So there's the use case that says they might be smaller but they have the value for the services there as opposed to, if I was just kind of a standard mid-range company and I might have five IT guys but the network's not as critical to my business. Might be a little bit tougher sell. I guess, do you guys see this? What about the service providers there? Are you working with them? We are working with service providers as well too. But for the most part, I think given the fact that we're so well entrenched in the enterprise space, that's probably where we're spending most of our time right now. In fact, we've seen such a rapid growth of customers that we've actually started to verticalize. So now we actually have a team that specializes in the healthcare space and that's actually the team I'm heading out. We have another team that's completely specialized on the commercial space and our traditional enterprise and select customers as well. Yeah, so when I think back to one of the first VMworlds, Martín Casado got up on stage after the, like I think it was a year after the acquisition and they tried to make the case that the virtualization admins actually managed networking because they have all of their V switches and VNICs and the like. And the average virtualization admin was like, I got a networking team. I got a guy that handles that. I don't want to know it. I don't want to touch it. You know, where are we with kind of the roles and is the typical virtualization or cloud person? You know, how much are they touching the networking? How much does this cross boundaries? And that's a great question, right? And this is a question that our customers often ask of us as they're exploring in a sex overall. And I think what we've seen is this massive shift within the industry overall, right? Networking folks are starting to learn about cloud. Cloud folks are starting to learn about the networking context, security context and like, and we're starting to see tiger teams across organizations, right? Because essentially if you're going to essentially try to enable your business and your IT infrastructure to move at the speed of business, you have to enable those teams, right? Those days of having silos of excellence, they're kind of, they're no longer silos, right? They're becoming more or less tiger teams overall. Yeah, one of the other keynotes this morning talked about AWS VPC. If you look at public cloud, you know, the network is really one of the first points that you put in, customers understand their VPNs, they understand how they can kind of create private resources in the public cloud. How does VMware and the NSX solution tie into public cloud deployment? That's a great scenario, right? So if you look at the underpinnings behind VMware's VCloud Air, for instance, or EMC's Hybrid Cloud, the underpinnings behind that are NSX overall. So we can maintain the same security and networking context, whether we're talking about on-prem or off-prem as well too. And even at VMworld this year, we actually talked and Guido did a talk and actually did a demonstration of how we actually can apply security context in AWS workloads as well too. So, and this has been the number one question that our customers have asked as far as, how do we move our workloads to the cloud? How do we actually secure those workloads? How do we know that those workloads are secure and maintain my compliance, regulatory compliance as well?  I know you support more than just VMware as the hypervisor, but maybe you can kind of give us a little bit of support on the low. Right, so the whole foundation around NSX first and foremost start off with NYSERA, right? And NYSERA with Martin Casada, as you well know, spent the majority of his time on the open stack endeavor, right? So OVS and, you know, OVSDB as well too. So our first platform was actually multi hypervisor platform. And now we've actually released our NSXV, which is essentially the NSX platform optimized for vSphere as well too. So that will continue to evolve as we continue to grow out our market overall, right? So we're not just essentially solely focused on one specific setting, one specific area. Now I think you'll see more to come on that in the very near future. Okay, maybe can you share some customer stories?  What problems are they solving? You know, what are some of the top case cases? So actually this morning, one of the use cases I talked about within my keynote was one of my customers based at Health, right? And we were actually tasked or challenged by the CIO of the organization. Essentially, their goals when they went about to build their infrastructure 2.0 infrastructure was that they wanted to have the ability to have any application exist anywhere and be available at any time. And to build it out, so I had done active-active type deployment scenarios, data center deployments in the past, in hardware, right? And it takes a level of complexity behind that as well too. You have to have shared storage volumes, you have to have, you know, layer two stretch it in between the infrastructure as well too, latency is a factor. What was nice about this infrastructure is we were essentially able to leverage NSX to provide all the networking context, all the routing functionality, and essentially leverage vSAN now as a shared storage component. So now an application could essentially exist across any of the three data centers in that environment. And then by using simple fault tolerance and fault domains, right? Any assets we created in data center A automatically had an HA pair in data center B. Anything we created in data center B had an HA pair ready to go in data center C and so on and so forth. So you can think about the ability now, it didn't matter where workload was spent up, where workload egressed or ingress from, you had that full capability, right? So you can think of it instead of doing traditional DR, just having pure application continuity. So beginning of the conversation, we talked a little bit about hardware. So there are any limitations that we need to worry about, you know, what environments can you go into? I know when the solution first came out, you know, one of the larger hardware providers out there, you know, caused some noise as to what would or wouldn't be supported. Where do we stand? You know, what environment should go in? What does a customer's network need to look like to support NSX? So obviously if you have a network infrastructure that's highly oversubscribed and very old and legacy in nature, that's going to be very difficult to actually abstain from, you know, multitude of workloads on top of that. So what we're seeing now is this pretty much started off with the hyperscale data centers, right? With the folks like Google and what we saw with AWS and the like too, is essentially start building other data centers in the spying leaf architecture. And that's pretty much what we're starting to see a lot of customers adopt as well too, because you essentially have the capability of having full cross functional or cross sectional bandwidth and your latency profile is guaranteed more or less because it's two switch ops across any environment overall. And now we're starting to see that resonate even further. Right, so as I mentioned before, we're completely decoupled from the underlying hardware. That doesn't mean we don't work well with hardware platforms that have been optimized for NSX overall. You know, whether we're talking about a Cisco-based infrastructure and a Rista-based infrastructure, Cisco, Juniper, it really doesn't matter, right? As long as the network infrastructure is built correctly and is highly resilient, then you have the great, you know, pinnings for an interlay. And then we can ride on top of that with our overlay technology. Okay, so Scott, you said you've been coming to this event for a number of years. Give us a flavor as to, you know, why you keep coming back, the interaction with the users and the value of the user groups. Yeah, I love this, because this is a great user community and it's New England, right? New Englanders are always known to be very self-reliant to begin with. So it's a great exchange of ideas that we see over and over again, right? So during the course of a conversation, you may find out how one person specifically solves a problem in their environment. They're willing to share that, right? So it's a great, great avenue to share resources as well as, you know, technical successes. All right, well, Scott Martin with the VMware NSB, you really appreciate you joining us. Thank you, Steve. Thanks for sharing with, you know, the people at this event and with our community. We'll be back with more coverage here from the VTUG. This is theCUBE. Thanks for watching.


This study investigates the clinical validity and reliability of an acute pain scale, UPAPS, adapted for newborn piglets undergoing castration. The study found that the UPAPS scale is a valid and reliable clinical tool to assess acute pain and castrated pre-weaned piglets. This article was offered by iRobles, SPL Luna, PHE Trindade, and others.


This is the other side. Kambua here, and also I have Ford over here, okay? The patient has cleaned themselves here, and then when they get here, they're kind of naked, right? What do you do? So at this point, we have all our supplies over here, right? Any questions? Good? All righty.


Everybody forest here right in the box of 2022 Phoenix football hobby hybrid for Michael G. Thanks Michael and good luck I don't know what exactly is that ghost fork those are fork ball anymore There's a sock this is gross. He's it's not fair Him with like more development is gonna be not fair. That's been read her base Purple K. Dot and rookie whose number is 73 of 99 the Derek car flame govers Jordan Davis hot routes Kelsey and orange rookie quay Walker This one's 22 out of 75 with an Eli Mitchell gyro fork.  That's we're gonna watch it tomorrow Jermaine Johnson Jalen Tolbert red that Crestcott 62 of 125 Hot routes Dalton Schultz Otten Michi flamethrowers fields Troy Anderson Romeo Dobbs. It is a fire and ice of Channing Tyndall 29 to 50 Carson Strong Weidermeyer Pierce purple terror Terry McLauren To 99 contours Garrett Wilson Dulcich Sam Howell rookie and a rookie rising Breeze Hall David Bell Sam Williams a Red Isaiah likely this one's numbered one twenty three of two of 150 hot routes McLauren Kenneth Walker the third rookie Louis scene Clyde Edwards a layer orange That one's to 75 Ezekiel Elliott Austin Jalen Petrie and flame goers Malik Willis That'll do it. Thanks again Michael. We'll get these out to you