Hello, everyone. I am delighted to introduce Oliver Kopp who is here to talk to us about protecting your machine learning against drift Hello Oliver and welcome to Europe. Hi, then. Hi. Thank you for having me. Where are you calling in from today? London and not so far from me. I'm in Cambridge All right, I'm very excited for this talk. So without any further ado, I will let you take it away Great. Thanks a lot. Thank you all for tuning in. Hope you've been enjoying the conference so far I'm Oliver Kopp. I'm an applied machine learning researcher at Seldom, which is a machine learning deployment and monitoring company Here at Seldom, we'll spend a lot of time thinking about how we can best enable our users to deploy their machine learning models in a manner that's Responsible and robust so they can be trusted to automate processes of real-world consequence One way in which trust can be instilled is by having systems in place That can detect when the model is being asked to operate on data from a distribution that differs to that on which it was trained These systems are called drift detectors and they'll be the focus of this talk In the data science and research team here at Seldom We develop and maintain an open source Python library alibi detect Which provides implementations of state-of-the-art algorithms for drift detection as well as outlier detection and adversarial detection But this presentation will focus solely on drift detection In particular, we will look at what drift is and why it pays to detector The different ways in which drift can manifest itself How just can be detected in a principled manner where we're able to differentiate systemic change from natural fluctuations The anatomy of a drift detector in the various components of which they're made up And hopefully along the way we can demystify some concepts such as online detectors permutation tests and MD tests etc Which might appear a little abstract or theoretical when you stumble across them in the alibi detect docs For example, but at least in the context of drift detection are relatively intuitive And yeah, we'll finish up with a practical demonstration showing how you can use the alibi detects Python library Okay, so let's dive in and set the scene into which we will introduce drift detectors This is the sort of standard supervised machine learning context, which I'm sure many of you will be familiar So I suppose there's some quantity why that we'd like to use as part of some decision-making process be that manual or automated Sadly, we can't observe why but we can instead observe some features that are related to why in some probabilistic manner So what we do is we fit a model to predict the target from the features and then use the model's predictions in place of the unobservable quantity So here is a simple example We've got a binary classification problem where we'd like to be able to predict a category north or one given the features x1 and x2 To make this a little bit more concrete, we're going to think of each data point as representing a person x1, an indication of their economic conservatism x2, an indication of their social conservatism And then the category is their voting intention with the blue data points Representing those who are intending to vote for the UK Conservative Party and the red data points those intending to vote for the UK Labour Party And you see here that we've got two sort of clusters of voters One of particularly economically and socially conservative individuals and other economically and socially liberal There's obviously a bit of a simplification that's the example that we're going to roll with And then supposedly fit some model to predict their voting intention from these features And the decision boundary is given by this black dotted line It might turn out that say on held out And then we wish to use this model as part of some downstream application for example Target ad campaign for example where we send voters ads depending on which way we predict that they'll vote The way this works is that we get an indication of how well we expect our model to perform on future data points By looking at how well it performs on held out training instances And that then comforts us in using our model in deployment However, this is only a good indication assuming the process underlying the data remains constant over time There's various reasons why in practice this might not necessarily hold And we'll come on to those in a moment So yeah, this sets up nicely what precisely drift is And it is simply when the process underlying the data that arrives during deployment differs from that which underlies the training data And this is problematic because when such a change occurs We can no longer expect the model's performance during deployment to match that observed on held out training data So in effect we're flying blind and we don't know how well our model is performing Many of you are probably thinking that you really don't care if our targeted ad campaign suffers So before we proceed I'm going to introduce this example from medical imaging which will hopefully help better motivate the problem So suppose that we would like to train a model to predict whether these tissue scans are benign or cancerous For the training data we collect scans for a mixture of three different hospitals We train our model and we observed on held out training instances issues a classification accuracy of 93% Suppose this is better than some human baseline, whether they suppose it's 90% for example We then might decide it's worth deploying our model to use on future patients But then once more deployed it might work well initially if it's been queried with scans the same underlying distribution For example if it was deployed from scans from the same hospitals Then suppose the new hospital started using the model and the distribution underlying their scans was subtly different in some way So for example maybe due to a different demographic of patient or for example hair looks a bit like Potentially the stain used in the microscope might have been slightly different When such a change occurs we can no longer rely on our model to perform at the level it did on held out training instances And therefore it might end up misdiagnosing patients and putting them at unnecessary risk So that's a very concrete example of why these drift detection functionality can be needed Okay and yeah by the process underlying the data we're of course referring to this joint probability distribution And this is useful because it helps us to categorize the different ways in which drift can occur And we do that by noting that the joint distribution can be decomposed in two different ways And returning to our voting intention example we'll now see what the different types of drift correspond to So we call this is what the undrifted data looks like so the training data One way that drift can occur is with the distribution of the features remaining the same But their relationship to the target changes so applied to our example There might for example be some new tax policy that means that voters no longer intend to vote along economic lines But purely along social lines and then our model might end up sort of misclassifying these individuals Who are particularly sort of economically conservative but socially liberal or socially conservative and economically liberal And the model's performance will suffer Another type of drift that can occur is what we call prior probability drift Where given an individual's voting intention Their features keep the same underlying distribution There's a proportion of voters that fall into each category that has changed as you can see here When this happens what was previously a suitable model decision boundary no longer looks suitable The individuals falling close to the decision boundary now overwhelmingly conservative whereas before it was 50-50 And therefore the model is going to misclassify a lot more individuals and the model performance will suffer Another way in which drift can occur is what we call covariate drift Where it might be that the relationship between the target and the features remains the same But the distribution of the features changes So suppose that there's some change in society which means that individuals are pushed towards conservative and liberal extremes Or alternatively they're sort of brought together into more of a central cluster Where the relationship between the features and the target has remained the same And therefore it makes sense to keep the same model decision boundary It now turns out that the model performance could still change dramatically So if they're pushed away from each other the model actually might perform better Because there's less data points falling into this region on which the model is uncertain When they're brought together suddenly there's a much harder problem for the model to solve And it ends up misclassifying far more points the model performance will drop So these types of drift where the model performance that causes the model performance to decrease Is referred to as a malicious drift You might be thinking that this looks like a relatively sort of straightforward problem That as the data arrives we can just monitor some metric of model performance If it looks like it's decreasing we can say oh well malicious drift is occurring let's take corrective action But it's actually not that simple because in practice the data points usually The deployment data usually looks more like this where we don't observe the labels Our data doesn't arrive labeled and this forces us to look for the change directly in the feature space And this is far more challenging because the features can often be very high dimensional and complex So whereas in this example there are only 2D you can sort of eyeball it and say oh well there's clearly been drift hair or drift hair If you instead imagine that these data points are like collections of images or text Suddenly it's a much more challenging problem to know when a change has occurred And this is the unsupervised drift detection context which is typically much harder So given that we're looking for change in the feature space How can we actually go about detecting drift because we don't expect the deployment data to look identical to training data Because we expect there to be some sort of natural fluctuations So for example if this is our training data and this data arrived during deployment We might look at this and say oh well the clusters do look like they're potentially slightly closer together But is that a systemic change or could that have just been a natural fluctuation? But if this batch of data arrived where it certainly looks like there's more of one big central cluster We'd probably be quite comfortable identifying that with drift and we'd probably take some sort of corrective action The problem is where do we draw the line? How much deviation do we allow before we say well that's been drift we need to correct for it The way that we can approach this in a principled manner is via statistical hypothesis testing Which I'll now quickly review for those of you who didn't take stats at school or are desperately trying to forget the experience The way it works so leaving the drift detection context for a moment so more generally Before observing some data we specify some null hypothesis An alternative hypothesis about the data generating process We also specify a test statistic which we expect to be small under the null hypothesis and large under the alternative hypothesis We then observe the data and compute the actual value that the test statistic takes And we also compute what's called a p-value which is a probability that such an extreme value of the test statistic would have been observed if the null hypothesis is true The idea here is a low p-value discredits the null hypothesis It's like saying we could continue to believe the null hypothesis But then we'd also have to believe that we just happen to observe some extremely unlikely data So yeah the way it works is we normally specify a threshold And if the p-value falls below that threshold we reject the null hypothesis And even if there's been nature even if the null hypothesis is true that occurs with probability equal to the threshold But at least this means it allows us to reject the null hypothesis in a manner where the false positive rate is named Okay so returning to the drift detection context the way in which we apply this is by denoting by q0 Distribution underlying the training data in which our model is fit q1 distribution underlying a batch of deployment data And we take as a null hypothesis that the two distributions are the same in the alternative hypothesis that the two distributions differ in some way And then we try and specify a test statistic so function of the training and deployment data that we expect to be small If the two distributions are the same we expect to be large if they're different We then compute the p-value and the flag drift if it falls below some false positive rate that we desire And the hard part here is in specifying this test statistic So this is a function of potential high dimensional complex data that we expect to be small under no drift large under drift And that's a difficult problem specifying that test statistic and even if we are able to do it How do we then associate it with a p-value that determines how extreme it is This is the sort of problem that we're looking to solve Unless you think that's not challenging enough in practice we observe the deployment data as like one big batch that we can then compare to the reference data It actually arrives sequentially over time like this it evolves When drift occurs we'd like to detect it as soon as possible So the way that we formalize this problem is by saying by assuming that the data follows the pre-chain so the training data distribution until some change point T star And then there's a change and it then follows some alternative distribution But each time step we'd like to perform a hypothesis test of whether or not drift has already occurred So every time say has drift occurred yet Yeah, so Designed properties of algorithms designed to tackle this problem is that we'd like them to be fast to respond when drift does occur So we formalize this expected detection delay which is expected difference between the time T dash at which we flag drift And the time T star at which drift occurs would like that expected detection delay to be small However, we'd only like it to be small subject to the constraint of a known frequency of false detections when there has been no change You know we can formalize this as the expected run time which is the average time which drift is detected even when there hasn't been a change So we can think of the change point being at infinity because it's never occurred Yeah, you might have noticed here there's sort of an inherent trade-off where we'd like the detectors to have this low detection delay But that requires sort of sensitivity which means the detector is also more susceptible to natural fluctuations that might trigger a false detection And therefore the expected run time is also low which we don't want So yeah, we have to balance that trade-off And yeah, this point is often overlooked actually and the ability to specify the frequency of false detections in the absence of change Often, algorithms allow you to know abound this expected run time But that's not particularly useful because the action that we're taking in response to sort of a 1 in 100 event is entirely different to the action that we're taking in response to say a 1 in 10,000 event It's important to be able to know accurately how significant a detection is when it occurs so that we can respond appropriately Okay, so you might have been wondering how we can apply the sort of statistical hypothesis testing framework to the data that's now arriving sequentially And the way that we do this is by collecting the deployment instances together into test windows And these can then be compared to the reference data or window using test statistics in the way that we described before Yeah, there's actually various strategies you can adopt collecting the instances into windows and the different ways of doing this Actually the characteristics and properties of drift detectors often derive from how they approach this window And so we now describe some of the main strategies for doing this So here, if you consider the data here arriving sequentially, so you've got t equals 1, 2, 3, etc. These are the features spotted here One way we can window the data is using disjoint windows where we specify windows say at a size of say 5 Click the first 5 instances together, perform a test, consider whether or not the test statistic is below some threshold Above some threshold, if it is we flag drift, if not the window slides along to the next 5 instances, we perform another test, etc And the window progresses like that An alternative is using overlapping windows where again we fix a window size But this time after performing a test, you only slide the window along by 1 rather than by 5 And therefore the window contains much of the same data Perform another test and it progresses like that Third way we can window data and algorithms that implicitly adopt this approach Often don't formulate their approach as a windowing strategy But I find it particularly useful to think of it in this way Is by instantiating the window with just a single instance Not only do we perform a test whether or not there's evidence of drift But we consider whether or not we should allow the window to grow If there's no evidence for drift, we reset the window to size 1 and just start again and perform another test If there is then a data point that's indicative of drift But not so much say that it causes a detection, we keep it in the window and allow the window to grow Perform another test, more evidence, allow the window to grow And this allows the window to grow and accumulate evidence over time Which when drift is slight, this can be particularly useful Because we don't just cut the window short and move on, we allow it to accumulate if the evidence is there And all these different windowing strategies call for different test statistics And different mechanisms for determining the thresholds above which the test statistics should fall Each of those strategies also have their own pros and cons which we'll dive into a little now Firstly, the Disjoint Window Detectors Distinguishing characteristic is that the tests are independent because the window is a disjoint And that perform infrequently because you only perform them every say 5 or a bit more realistically say 500 instances that arrive And this is good because we can then take any test statistic and then it's actually possible to accurately estimate the p-value That says how extreme that test statistic is because we can perform a what's called a permutation test This would be too expensive to perform every time step say for overlapping windows But in this Disjoint Window Framework, we have time to perform these And these are actually dead intuitive how they work so I'm quickly just going to overview them So where they work is we take the reference window and the test window and we define a function that is able to shuffle all of the data Such that we then have two new windows that contains a mix of reference and test data points that completely shuffle together We then compute the value of the test statistic on a large number of these shuffled windows And then we look at the value of the test statistic on the unshuffled windows relative to all of the values on the shuffled windows And we see how extreme it is. So for example it was at the 99th percentile Then it would be in the top 1% of extremeness and that gives us a p-value that says how extreme the test statistic was The idea here is that if there's been no change then it doesn't matter whether or not we shuffle the data or not Because they all have the same underlying distribution and therefore we don't expect the shuffled, the unshuffled test statistic to lie at an extreme end So if it does line at the top 1% then yeah that's only one of the 100 event for example and that gives us the p-value Another key advantage of the disjoint window detectors is that we don't actually necessarily have to specify the test statistic Which as mentioned before can be a huge pain. We can actually use a portion of data to learn the test statistic So we can train a function to be small under no change and large under change And that completely gets around this problem of specifying the statistic Cons of disjoint window detectors is that they're very sensitive to the choice of window size Because if you use a small window size then there likely won't be enough evidence in any given window to allow drift to be detected But if you use a very large window size then suddenly we're only performing a test and we say 500 or 1000 instances And therefore the detection delay is necessarily going to be large because well we have to wait until the next test point regardless And that means that the detectors are particularly slay to respond to severe drift because it doesn't matter how severely drifted the instances are We still have to wait until the next test point before they can cause drift to be flagged And yes then looking at overlapping window detectors which are much trickier actually because consecutive test statistics are highly correlated This is because test statistics are successive time steps of functions of much the same data and therefore they're going to be correlated And this makes specifying the relationship between like thresholds and expected run times much trickier Because even if we can identify the threshold above which the first test statistic resides with only probability say one in a thousand Given that we make it to the second time step the probability that the second test statistic then exceeds that threshold Is then some unknown value much rarer than one in a thousand because we already know that it didn't cause a detection at the first time step So you sort of have this correlation which makes things much trickier And yeah this is something that we've spent a lot of time sort of researching and selling how we can capture this relationship between thresholds and run times for the overlapping window detectors It turns out that there is actually a way in which we can estimate sort of time varying thresholds that allow a desired expected run time to be targeted However we then have these sort of additional constraints on the test statistic that we didn't have for the disjoint window detectors Where it must be possible to update them incrementally because we're performing tests every time step It's important that we don't have to compute them from scratch every time because that would be too costly And also we can't learn the test statistics like we did for the disjoint window detectors We have to be able to specify them at the outset which as I mentioned is a bit of a pain But if we can do that they're computationally light because of the incremental manner in which they operate And they can be very fast to respond to severe drift because the severely drifted instances filter into the test window immediately We don't have to wait until the next test point we'll perform tests every time step and therefore we're faster to respond when severe drift occurs Okay a final way of wondering the data is these adaptive windows and these introduce additional complexities These typically work by accumulating some notion of outliners At least approaches for which the page change distribution is unknown and they operate at fixed cost this is how they normally work The way that this works is you sort of score each instance independently as they arrive Where they receive a negative score if they look relatively inline and a positive score if they look outlying The way this works is that prior to a change if we accumulate these scores we expect to solve downward trajectory that keeps decreasing Then if there's a change then suddenly the data points are going to start appearing more outlying And the trajectory will start to rise And then the challenge here is specifying the threshold for how much of a rise we allow for we detect drift And that's again a very challenging problem, specifying this threshold to correspond to a desired expected run time And we did spend some time looking at this that's seldom because we thought these adaptive windows were very nice In the way that they allow the testimony to grow However it turns out that this idea of accumulating outliners is actually not great for drift detection It doesn't result in good in low delays It's a little bit surprising because it was quite an intuitive idea we thought But we're now going to look at why that's the case because I think it helps illustrate the difference between drift detection and outlier detection So if we suppose our free change distribution is this sort of isotropic Gaussian distribution With the brighter region in the middle representing region of high probability density and the darker regions lower probability density And to think of these pink points as representing our reference set or training data points Then suppose during deployment we observe data that's much more tightly clustered Say these seven data points here which we can eyeball and say they're from a different distribution that's much more tightly clustered Some sort of drift has occurred The problem with the outlier based methods is that if you can consider the points in isolation you treat these points exactly the same as you treat these points And suddenly they're probably not data points that you'd have looked at and said oh there's been drift So by considering them in isolation we're not able to capture that And that's why looking solely at how outlying data points are is insufficient for drift detection Okay so having ruled out outlier based test statistics is insufficient How do we propose that the test statistics should be formulated And the ones that work particularly well are those which estimate the distance between the underlying distributions So these distributions are unknown but we aim to estimate the distance using the data that we observe One way that you might think of doing this is by using the data to sort of model the two distributions And then plugging them into some notion of distance between the distributions However with few data points that we normally have in practice this is an inefficient way to approach the problem What we instead do is try and estimate some notion of distance between the distributions directly And one distance between distributions that appears particularly prominently in our library alibi detect is this maximum mean discrepancy distance Which we find particularly useful for various reasons so we're now going to quickly overview how that distance works Okay so reason this is useful is because it transforms the problem of specifying the distance between the underlying distributions To simply specifying a similarity between data points which is a sort of much simpler more intuitive problem Typically we do this by projecting raw unstructured potentially high dimensional data points onto lower dimensional more structured representations using some projection phi And then evaluating some simple notion of similarity such as an inner product And this is great because this is possible across all data modalities Say for images, for the projection we might use some pre-trained convolutional network for example For text, some pre-trained transformer or whatever See this is particularly straightforward The maximum mean discrepancy can then be defined in various ways But in this context probably most simply there's the expectation of the similarity between reference instances added to the similarity between deployment instances Take away two times similarity between reference and deployment instances And this distance can be estimated from the data simply by taking the average similarities This is particularly convenient way distance can be estimated very simply But regardless of what we're trying to estimate here we can look at this as our test statistic which satisfies the properties that we want of our test statistic Because if the two distributions are the same then we expect the average similarity between reference instances To be the same as the similarity between test instances to be the same as the distance between the two And therefore these terms all cancel out and the test statistic is low But if there has been some change then one or more of these terms will be different and they don't cancel out and it will be large So this allows us to specify our test statistic to have that property of low and no change, high and the change which is what we want And often we'll see it written like this which is a bit more intimidating But yeah just it is a very simple concept in terms of average similarities So yeah so if we return to the example which outlier based test statistics were insufficient to tackle Here we're able to see that the deployment instances are much more similar much closer together than the reference instances And the sort of distance between the two and therefore when we add up these terms they don't cancel out and we get a relatively large test statistic value And then this is great but then we need to determine precisely how extreme is that value in order to associate a p-value And decide whether or not to flag drift and the way that we can do that is using the permutation test that we described before So here on all of the shuffled reference and test windows the similarities between the shuffled reference instances The shuffled deployment instances and the cross similarity between the two are all the same in expectation and on average they cancel out So for example we get very small values and therefore we expect the MMD estimate to be much higher on the unshuffled version And on the shuffled versions it ends up in the top say 1% and then we're able to use that as a flag drift And yeah that's pretty much all of this theory I was planning to go through So in future when you're trying to recall how drift detectors work I highly recommend that you think of this sort of funky looking fellow with alibi detect glasses And know that there are three main components that define a drift detector There's the strategy for collecting the deployment instances into test windows A test statistic which is then able to compare those test windows to the reference window or training data And this might involve specifying some projection as we described before And additionally we then have some strategy for setting fresh holes which those test statistics must remain below in order for drift not to be detected And it's important that these three components are able to work together in a manner that allows the expected run time to be specified So that when drift occurs we know precisely how significant it is and therefore can take appropriate action So yeah this is a sort of dial that we can tune and as we do under the hood this expected detection delay will vary But this isn't what we directly manipulate this is just sort of a downstream effect And yeah as I say we've got various drift detectors implemented in our alibi detect library And if I've got time yeah I've got time to just give you a quick demonstration of how the library can be used So hopefully this will work as put together pretty short Jupiter notebook to demonstrate how the library can be used I won't have time to go through all of the code line by line but I'll try and pick out the most relevant points There's a load of imports here which aren't particularly important But for the data we're going to return to the Chameleon 17 medical imaging data set that we described earlier Where we wanted to train a model to classify tissue scans as benign or cancerous But during deployment we start getting instances from an alternative hospital with a slightly different distribution So yeah this data set is particularly convenient to use because there's this python library So the wild python library which is put together by a team from Stanford that allows you to load and manipulate this data particularly easily Using these sorts of Chameleon or wild dot get the data set functions By just create this sort of wrapper around this to convert the data into a stream in order to simulate a sort of live deployment environment We then define a reference set of n equals 2,500 instances And then we also define two streams, one representing no change So the instances are from the same distribution as training data, so this one And one stream from the drifted hospital where there has been a change This is slightly simplified the problem and kind of put the change point at zero But by considering these streams and looking at how the detector, how long the detectors run for on these two streams Will allow us to investigate the average run time and the average delay and see how that compares to the expected run time that we desired So quickly just plot here some instances from the no change stream Looks like this, find the drift detector We're going to use a strategy of overlapping windows and we're going to use the test statistic, an estimator of the MMD that can be updated incrementally And therefore is suitable for the overlapping window context As I mentioned before because these images are quite high dimensional and structured We first perform a projection down onto a lower dimensional and more structured representation In order for the kernel to be more meaningful in the MMD to be a useful test statistic And to do this we train an auto encoder that passes the images through a lower dimensional representation Such that they're able to be reconstructed and therefore it captures the sort of structure of the images in that lower dimensional representation And here is just a convolutional auto encoder defined in PyTorch Here you can see the auto encoder being defined This is a sequential PyTorch model and we just pass the reference data to a PyTorch data loader module And then alibi.tech provides a function that allows you to train all these components particularly easily So we train the auto encoder and you sort of see the loss getting down here Such that it's been trained And then we define a function that uses the auto encoders encoder to map the images So alibi.tech expects functions mapping numpy arrays to numpy arrays We just wrap the encoder in this function that just does a bit of a conversions for us Having now defined the projection that we're going to use for our kernel Defining the detector has actually been very straightforward Simply specify the expected runtime we desire The size of the test windows we'd like to use The number of simulations we'd like to use to configure thresholds Then we simply import from alibi.tech What's called the MMD drift online detector So just overlapping windows using MMD Tector at which the detector flags drift Just to be expected runtime I'm only averaging over 15 runs this time So it's a very noisy estimate and it's not that close to the expected runtime of 150 But if we averaged even more run times we would gradually converge in on 150 So this is the sort of false positive rate On average it takes a false positive Every one in the 122 time steps We can sort of look at the detectors Test statistics and thresholds and sort of see what was happening under the hood The test statistics sort of fluctuated around zero But then eventually there's a fluctuation large enough to send it over the threshold And that happened at time step 163 as you can see here But then when we apply the detector to stream on which There has been a change The detector then only runs on average 14 time steps This is a sort of estimate of the expected detection delay This is great because it shows that the drift detector is working And that it flags drift much more quickly when there has been a change Than when there hasn't Which is obviously in the property that we'd like In this case you can see that the test statistics Started rising very quickly from the start And very quickly exceeded the test threshold that the detector had calibrated So that's pretty much all there is to it I hope that was useful I'll return to the slides And if you found this interesting Do check out our open source piping library That you can find here And also you can follow us on Twitter Set up a Twitter account this week So if you'd like to see Updates Okay Looks like there's been a problem Okay so it looks like I Break up a bit there Where did I get to And if someone could help me out a little I think I was pretty much wrapped up anyway You can follow us on Twitter If you'd like to receive updates on our research And our Python libraries Personally I'm ocselden.io I hope that there wasn't too much of a disconnect there But yeah that's all about to it Thank you for listening Thank you so much for that Oliver I was definitely thrown back to my stats A level For a room in there We do have a couple of questions from chat We have just a minute to go through them The first and probably quickest to answer is Is the alibi example Jupiter notebook available somewhere For people to download I think I was saying about this when I probably disconnected I'll probably tweet that from our Twitter account later So people can work through that more slowly If they desire So yeah do give us a follow I'll get that tweeted out Alright and a question from earlier Which was given as you said That after deployment we don't often have access to Why being true Can these methods only detect covariate drifts And then But you also said that if PX only PX changes Then the original model is still valid Yeah so that's a good question If the drift is purely Sort of Concept drift where only the relationship Between the targets and the features has changed Then sure we can't detect that By looking solely at the features But normally When a change occurs more than one Component changes and therefore You're able to sort of detect that the changes Happened in general Yes the model is potentially still valid If only the feature distribution has changed As I said The performance might have really degraded And it's important to be aware of that Because the model may no longer be suitable For the problem that you're trying to solve So although it may still have The best decision boundary given the context It may then no longer be suitable to deploy For example But yeah that's a good question Alright we've got a couple more coming in From the chat And we are about to run into lunch Okay For your windowing strategies Have you tried using time lagged approaches Where window sizes is ever increasing Yeah so that So I guess that sounds pretty similar to these Adaptive windows That I was describing I'm not sure if They meant something slightly different But they allow the window to grow And that's great But that means that the cost Of operating these detectors can then also grow And that's not really something you want And also it makes setting the thresholds Much, much more trickier So I was talking a lot about targeting An expected run time And that's tricky to do Using these windows that grow So there is that sort of that compromise Of using overlapping windows In order to allow A fixed size So that the desired run time can be targeted So there's pros and cons All over the place for these strategies It's always the way isn't it And one final question from chat Which is Would you see an issue in using an embedding model For the distance which is very close to the model For the predictions For example using a base BERT B-E-R-T for embedding In the case of a fine tuned BERT For the predictions Yeah so there is This is a good point that you shouldn't use For the projection A component that's been trained on The same reference instance For the detector Because then The component then sort of hugs these instances And puts them into a different sort of region So it does unseen instances And that sort of causes a discrepancy It isn't due to changing the underlying distribution But it's just being Because they've been seen before So it is important that you Don't train the component using the same data And you'll see in the notebook That actually split off a bit of the data To train the projection The separate split of the data To the detector And yeah that's a very good point That's an important consideration to make I have a follow-up question about that Is there any rhyme or reason to how you would split the data set For those two partitions Yeah so obviously again It's a compromise And the more data used to fit the projection The more potentially meaningful That projection would be The better the test it is equal than B But then the less data you then have To the detector And therefore The sort of less statistical power you have With which to detect drift So there is a sort of that compromise And I recommend sort of Using the smallest amount of data That you believe is sufficient to train Whatever component you're trying to train So that does require potentially a bit of domain knowledge But yeah It's not something I worry too much about At the exact split point All right thank you so much for that answer I don't really have any more questions coming in from chat So I'll just say thank you So very much for this talk It was really really interesting And thank you so much for giving your time to your reply Great thanks a lot for having me Enjoy the rest of the conference everyone


Our next presenter is the FAA's Deputy Federal Air Surgeon. He's retired from the Navy as a Captain after 30 years of service. He holds several degrees, including one Doctor of Medicine. He is also the Emeritus Member of the Society of U.S. Naval Flight Surgeons. And he was also President of the Naval Flight Surgeons from 2002 to 2003. His topic today is maintaining that important document for us, maintaining your medical. Well, let's welcome Dr. James Frazer. Well, good morning. Today I would like to talk about a couple of things. First of all, I'll start off by just briefly telling you a little bit about my background just so you know where I'm coming from. And then I want to talk a little bit about what the FAA's Office of Aerospace Medicine is all about. And lastly, and of course most importantly, I want to talk about some of the current issues regarding maintenance of your medical and of course leave time for questions at the end. As Walt said, I am indeed former Navy. I was a happy sailor some four and a half years ago when the Navy sent me one of those letters telling me that 30 years was all the fun a sailor was allowed to have. So I was very fortunate to be able to apply for a position with the FAA and came aboard four years ago and I really enjoy my job. I really enjoy what I'm doing. My background in the Navy did not start as a flight surgeon. I started as a family physician and after doing residency training in Charleston, South Carolina and sticking around as family practice teaching staff, I went overseas with my family and served as a Navy family physician in first the Philippines and then in Scotland. I had some wonderful times overseas and certainly I enjoyed family medicine but I had always wanted to learn to fly an airplane and to be a naval flight surgeon. So I came to Pensacola, Florida and Pensacola is the birthplace of all things involving naval aviation and naval flight surgeons. So just before I turned 40 and got too old, I reported for flight surgeon school and they taught me how to fly an airplane. I no longer had call every second or third night because I got flight pay. I got a pay raise. So that was my start into aerospace medicine and that's where I finished my naval career. I did a second residency in aerospace medicine there at Pensacola and if you do an aerospace medicine residency in the Navy, it's basically a one-way ticket to one of our national treasurers, one of our nuclear aircraft carriers. So I was proud to serve as the senior medical officer on board the USS Theodore Roosevelt as such was in charge of a 67-bed hospital and a 50-man department but truly looked back with great honor at that time as the senior medical officer on a carrier. Following that, I joined the Commander Naval Air Force's Atlantic Fleet where I had oversight of what were then eight carriers on the East Coast and all the flight surgeons and physicians that were a part of Naval Air Force's Atlantic Fleet. That was a wonderful job and I finished my career in the Navy at the Naval Safety Center in Norfolk where I had the opportunity to teach all naval flight surgeons. I had the opportunity to be involved at the very highest level of naval aviation in terms of mitigating some of the mishaps that we would have and I was in the chop chain and became intimately familiar with all the Navy Marine Corps accidents that finally got closed out at the Safety Center. And then I just in the last couple of months when you're supposed to be gearing down and figuring out what you do after you retire, the last thing I did in the Navy was serve on the Columbia Accident Investigation Board for a couple of three months. So that's my background and that's what I did before joining the FAA. Let me tell you a bit now about the Office of Aerospace Medicine. Aerospace Medicine is comprised of some 330 people spread out across nine regions and of course, CAMI, the Civil Aerospace Medical Institute in Oklahoma City that probably all of you are familiar with. We reside within aviation safety. We are the third largest service, a distant third behind flight standards and aircraft certification. However, we do have the largest number of designees because we have some 3,500 aviation medical examiners that we have trained and that we keep up to date. In the Office of Aerospace Medicine, our main job is the development of policy and standards, not only for airmen but for air traffic controllers. The FAA is basically the world's gold standard in terms of medical certification and I think most of the world's certificating authorities look to the FAA for guidance in terms of what we allow and don't allow with medical standards. We at the FAA are certainly in the forefront of looking at new conditions, looking at new procedures, looking at new medications and I'm very proud of the fact that we use the very best of evidence-based medicine to look at these new changes in medicine and we are willing to certificate those things that we feel an airmen can safely fly with. We certainly don't say no, that would certainly be the easiest thing to do in the interest of fairness to the airmen. At the same time we keep the national airspace safe, we certainly lead the way. There's very few countries in the world where a diabetic could fly while taking insulin and yet we the FAA allow that for our third class pilots. Whereas our primary interest in aerospace medicine is keeping the national airspace safe, a secondary mission of ours is really getting every airman up that we think can safely fly and I'm proud of the fact that of those airmen that work with this and give us the procedures or the consultations or testing that we require, ultimately only about one-tenth of one percent of all the airmen that we certificate are finally denied. So we'll talk a little bit later about reasons you should never falsify on your medical, but first and foremost the best reason is only one-tenth of one percent of all airmen that one medical are ultimately denied and of course the way we're able to do this for people that have had heart attacks or cancer or one thing and another is through special issuances and special issuances are just our way of giving airmen that have conditions that would formerly be disqualifying, they give them a time-limited certificate and we specify what kind of follow-up and what kind of testing they need and by working with the airmen in this manner we're able to certificate just about everybody that's interested in a medical. So at headquarters in aerospace medicine we oversee the issuance of 450,000 medicals every year at headquarters we serve as the appellate level or the last level of appeal for airmen that have been denied first perhaps by their AME or by the Aeromedical Certification Department at CAMI. So by the time the cases get to headquarters they're usually about that thick and they're typically very interesting medical or psychiatric issues that I certainly enjoy getting involved with. The Office of Aerospace Medicine has a role in occupational health. We work with NIH and CDC and Department of Homeland Security. We meet with those folks and we talk about how we the FAA would collaborate in cases of communicable disease like TB or SARS, certainly both very topical issues of late. We talk to these other agencies in terms of how we would interact with emerging diseases like avian flu and we also work with them in terms of what we would do for a biological agent such as anthrax. We also have roles in occupational health doing more mundane things like figuring out how big of commercial aircraft an automatic external defibrillator should be on what should go into a medical kit, how big a plane needs, how many medical kits and occupational medicine issues of that nature. The Substance Abuse Substance Abatement Division resides within Aerospace Medicine. We have a very robust division made up for the most part of our industry substance abatement folks who regularly test our 121 and our 135 pilots to make sure that they're not using drugs or under the influence of alcohol. But we also have our own internal program whereby we test our own FAA employees that are involved in safety or security sensitive positions and for the most part that's the air traffic controllers. One of the things that I'm really proud of is that we at the FAA are very active in what's known as the HEMS program and HEMS is Human Intervention Motivational Study but basically it's a program that's been around for 20 years now whereby air transport pilots that are substance dependent have a method to be identified, treated, rehabilitated and brought back into the professional world as meaningful contributors to society. We work very closely with the unions and the air carriers to be able to treat these air transport pilots and to bring them back to be a useful member of society. We have trained enough aviation medical examiners and their role as a medical sponsor in this process that we're now able to also work with 135 pilots and Part 91 private pilots in terms of getting pilots that have been successfully treated and successfully rehabilitated back into the cockpit sooner rather than later. Substance dependence is not a kiss of death for a medical certificate. We have a very robust role in air medical education. One of the primary reasons we exist is to train aviation medical examiners and as I mentioned there's some 3,500 aviation medical examiners that are spread throughout the world. We not only do basic training but all of these AMEs are required to go through recurrent training. So many of us in the FAA take a very active part in training these AMEs. And of course we also train pilots, if you step outside you can go through the spin and puke which we typically take to the air shows to show all the allusions to airmen but we also bring airmen to CAMI and are able to put them in the hypobaric chamber or are able to put them in the pool to deal with evacuation issues and things of that nature. Aerospace medicine is also very much involved in research. We do research throughout aerospace medicine. CAMI is the biggest place we do research and there we have sleds and we have hypobaric chambers and we have pools to drop people into. We look into things like emergency evacuation. We do testing to determine how you could better survive an airplane crash. We have a big human factors division. We look at things like airmen, flight attendant and air traffic controller fatigue. We look at how you select and how you train air traffic controllers. We look at color vision and other aspects of certificating both air traffic controllers and airmen and do a number of research activities involving the toxicology laboratory that we have at CAMI in Oklahoma City. We also do airliner cabin research. We are doing active research to look into how you decontaminate an aircraft following a communicable disease, emerging disease or even a bioterrorism agent. We look at airliner cabin issues such as the petroleum products that come out of bleed air that may or may not cause health problems for people in commercial aircraft that they work in those aircraft for many years. We also look at the effect of pesticides and what effect pesticides might have on these people. And we look at the more natural environmental issues that all airmen are subject to. Hypoxia, low humidity, low barometric pressure, ozone and radiation. So we have a very active role in research in aerospace medicine. So let me move on to some recent decisions and some current issues that will start to get into the maintenance of your medical issues. In terms of the headquarters FAA perspective, the biggest recent change was the Fair Treatment for Experience Pilots Act. And many people know this as the age 60 rule or the age 65 rule and basically it's the rule that goes back to 1959 whereby the FAA declared that for an air transport pilot the day they turned age 60 they could no longer transport ticket-paying passengers. And of course there was great controversy regarding the age 60 rule and our last administrator Marion Blakey basically went online and said we were starting rulemaking. But fortunately the president on the 13th of December of 2007 with the stroke of a pin can do what presidents do and change the law so that now air transport pilots can fly up until the age of 65. That was great news, welcomed by most of us that I know of in the FAA. Those that are involved in rulemaking know that rulemaking is a long, arduous process and it would have been probably two years to go through all of the rulemaking that would be involved in changing that by the typical procedure. But thank goodness we have people like the president that can just do that with a stroke of a pin. So certainly that has made my life a little bit easier and I look forward to following this process because we, the FAA and Aerospace Medicine and CAMI are going to be responsible to make sure that we are equally safe or safer with a pilot that's between the age of 60 and 65. In Aerospace Medicine we heard from pilots like you at venues like this and at venues like Oshkosh and via Congressional and other means of communication.  But for those of you that have had a health issue that need a special issuance a couple of years back there was a delay. There were times that it took several months to get an airman authorized for a special issuance and you weren't particularly happy about that. So we did a lot of things that I won't go into to speed the process up. In particular we made the process electronic so that all of our regional flight surgeons and deputy regional flight surgeons could play in the airman certification process. But one of the biggest things we did was create this class of AME assisted special issuances. We now have 35 medical conditions such as myocardial infarction, heart attacks, cancer, atrial fibrillation, I could go on. But basically 35 of the most common medical conditions whereby after a first time issuance by either AMCD at CAME or by the regional flight surgeon we then specify given a current status report on usually an annual basis and specified testing depending on what the medical condition is that you go to your AME and your AME can give you your medical every year usually and that doesn't have to get sent to CAME. Right now the average processing time for a special issuance is 16 days. And I'm very proud of the fact that we've been able to whittle that time down and no longer have the several month delay that we had a couple of years ago. As I mentioned I like to think we're on the cutting edge of allowing different medical conditions as opposed to our certificating authorities throughout the rest of the world and a good case in point is heart transplants. We now have certificated to heart transplants and when we do something of this nature we typically have to look at a subset of heart transplants and basically you can look at heart transplants and if you have angiographic evidence that there's no allograph basculopathy or left systolic ventricular dysfunction you can certifcate someone and you guys know all that so I won't go on. But the important thing is we have used the evidence based medicine and we take something as serious as a heart transplant and we identify that subset of those that we can safely certifcate and then we do so through the special issuance process. FAA MedExpress. Some of you in here have probably used FAA MedExpress. FAA MedExpress is our internet process whereby those of you that want to complete the front of your 8500-TAC-8 at 2 in the morning while in your pajamas can now do so. It's our effort to be more efficient and basically all you need is an internet address. You get onto FAA MedExpress, you fill out the front of the 8500-TAC-8 if you need to find out information or if you don't complete it in one sitting it will save whatever information that you've put in thus far.  So some 25,000 airmen have now used FAA MedExpress and I encourage all of you to give it a try if you are so interested. It's not required for airmen and it's not required that AMEs participate but certainly you can ask your AME if he does participate or is willing to participate in FAA MedExpress and we're always seeking to make the application process more customer friendly. Oops, finger finger. Alright, some more good news. We are currently in the very last couple of weeks of what's called team concurrence for final rule on changing the periodicity for medical certificates for airmen under the age of 40. There is ample medical evidence that if you're younger, particularly if you're under 40, you don't need to have physicals as frequently as those of us that are over 40. So we have proposed and soon will have finalized the rule whereby if you are first class and you're under 40, you'll need a medical once a year by six months and if you are third class under the age of 40, you will need your physical every five years and I haven't found too many airmen that are disappointed about the fact that they can have physicals a little less frequently. Now we did, when we published our notice for proposed rulemaking, we did discover one small hiccup and I'll bring that up just to be complete. Many of you are aware that when you fill out that 8500 TAC-8, you sign a statement that gives the FAA the authority to go to the National Driver Registry and look and see if you have DUIs. When we created this new rule, we didn't realize that this was based on a statute that we tag on to that belongs to the Federal Highway Administration that basically says we only have the authority to look back three years. So even though we are going ahead with the final rule, we are going to have to work on getting our own statute whereby we have the authority when you sign that 8500 TAC-8 to look back five years just to close that potential crack that an airman could potentially slip through. And of course I know most airmen dutifully report their DUIs. We occasionally find some that forget they've had a DUI. So we want to close that crack and we have five years to do so so I feel very confident that this is good news for all of those airmen under the age of 40. I also want to mention the antidepressant working group. We have, I talked about this last year. I've talked about this at meetings for the last four years. But I can tell you we are no kidding very serious and very close to identifying a subset of airmen that are on antidepressants that we're willing to special issue and follow very closely. 10% of all of us are going to be clinically depressed at some time in our lives and that pertains to airmen just like the rest of the general population. Right now if you have a diagnosis of clinical depression or you're taking an antidepressant that's disqualifying, certainly depression in and of itself has some disqualifying features that are not compatible with being a pilot and then the medicine, the selective serotonin reuptake inhibitors, SSRIs those have been shown to have cognitive effects such that you're not as sharp as you otherwise would be. However, we have identified some of the SSRIs that are less sedating than others and we have a cadre of psychiatrists that are willing to follow airmen on a probably a biannual basis and we are probably going to be using the neuropsychological testing which is just a very sensitive test that can determine that you have the appropriate cognition, memory and attention that you need to be a pilot. So very soon you will be hearing more if you are interested about how you can be a pilot and fly on antidepressants.  Well, we know that. We know there are airmen out there that are taking antidepressants that aren't being forthright and truthful and even worse than that, we know that there are airmen that realize that antidepressants are disqualifying so they're going untreated and that's probably even worse. So we look forward to being able to certificate this subset of depressed pilots that are appropriately treated. Our lives in Washington have been interesting of late. If you've watched the news you know that my boss Nick Sabatini has had more fun than any one man should be allowed to have. First the house beat him up last week and the Senate beat him up yesterday quite soundly. According to the paper. But we have these, I shouldn't say nasty things, that's part of our democratic process. We have hearings in Washington D.C. which are not always fun to go to. And the hearings that have affected aerospace medicine of late in a specific sense have been Mr. Oberstarr's house transportation and infrastructure subcommittee. And this is regarding airmen falsification and it affects you guys just like it affects me. And it all stems out of a project that was initiated by the Social Security IG a couple of years ago that got in cahoots with the FAA IG and they looked at a group of pilots in Northern California with over 60,000 odd pilots in a database match called Operation Safe Pilot. And the premise was if you're on Social Security disability unlike military disability, Social Security disability pretty much means that you're not supposed to be able to do any kind of work. You're unemployable. And as the IGs from our various agencies match the database, lo and behold, there were a number of folks that held both the pilot medical as well as collected Social Security disability. So the IGs found the 60 odd most agrarious cases, those that had told Social Security they couldn't work because of cardiac or neurological or psychiatric reasons and yet they had not told us the FAA that they had any such problem and these people were all taken to trial and some of those are ongoing but certainly the airmen had anything between loss of medical certificate, significant penalties or even face potential prison times. So Congressman Oberstarr picking up on Operation Safe Pilot has called Mr. Sabatini and my boss, the federal air surgeon Dr. Fred Tilton up to the hill and has basically told them that we, the FAA and specifically Aerospace Medicine must have majors in place to nip this airmen falsification problem in the bud.  We've told them that there are serious consequences to falsification.  And those of you that follow the news know that last month there was a pilot that was sentenced in Massachusetts for falsifying. This was a diabetic pilot flying a 135 operation when he went into a diabetic coma. Fortunately there was a young lady on board that had had some 10 hours of flight time or something of that nature that was able to safely land the airplane. I think it was a wheels-up landing but everyone walked away so she was a hero. But nonetheless the diabetic 135 pilot has now been sentenced to a couple of years in jail. So it's my job to tell airmen like you and to tell AMEs that there's very good reasons that you should be honest and forthright on your medical. Not the least of which is the fact that less than one tenth of one percent of you would fail to get your medical if you would just work with us and get the specified follow-up and testing that we agreed to. It's also fair to point out that there can be serious consequences with falsification if you were so unfortunate as to be the pilot in the fatal mishap. You know all of our fatalities in aviation mishaps get investigated by the NTSB and by law we, the FAA, Aerospace Medicine have a world-class toxicology laboratory at CAMI and part of the post-mortem findings involves shipping specimens to CAMI where you are tested by very sophisticated equipment that looks for any unusual drugs and unfortunately there have been cases where there have been antidepressants or cardiac medicines or other post-mortem findings that indicate something different than what the airman reported in his 8,500 TAC-8 and when that turns out to be the case insurance companies not uncommonly refuse to pay life insurance so it's important, another important reason that airmen should be honest and forthright when they fill out that 8,500 TAC-8. As a result of these hearings a couple other changes that you'll be seeing is a new box on the front of the 8,500 TAC-8 whereby you will be asked if you receive medical disability and also if you turn the front page of the 8,500 TAC-8 over and actually read the instructions there will be a new line that will make you aware that by filling out the medical certificate or by the application your information could be matched with other federal databases in much the way that Operation Safe Pilot was operated in Northern California. So as long as you're honest and forthright there's nothing to worry about but certainly Congressman Oberstar is interested in this falsification issue and we are working with him to do what we can to educate and tell airmen and AMEs to be honest and forthright. I want to close before I open it up to questions just talking about current issues with Sport Pilot because in a venue like this Sport Pilot is certainly topical. These are some of the operational limits that many of you that fly Sport Pilot are aware of basically low and slow. The most important part from my perspective is the medical issue regarding Sport Pilot. All of you probably know that you can fly a light sport aircraft with a valid driver's license unless for any reason you ever applied well not ever, if for any reason you applied for an FAA medical certificate and you did not receive that on your last application or if you were revoked, suspended or denied or if you held a special issuance and that was withdrawn. If for any reason on your last application if you had applied for your FAA medical and not received it you cannot fly with your driver's license. The good news is that many people that were disappointed with this were people that had been denied medicals 10, 20 years ago and 10, 20, even 5 years ago it was a different world. We the FAA were acting much like certificating authorities across most of the world today. It was much easier to say no to heart disease or cancer or any number of medical conditions. We have certainly become much kinder and gentler since then and we know by the evidence that you can safely fly with many of these conditions under specified follow up and testing. So there are many airmen that got denied 20 years ago that would have no problem getting a medical certificate if they were to reapply today. And then of course even if they have had something significant we are willing to work with them and offer them a special issuance and of course once you get a special issuance you're good to go. You're good forever more and you can use your driver's license you just have to get that medical ticket you have to get your medical certificate and we're willing to work with you to help you do that. And I will say right now that many people have pointed out that this rule which was not medicals making but this rule that I get asked about frequently is potentially unfair. I admit that. Two brothers that both have the identical seizure disorder. One brother applied for a medical certificate with the FAA and was told no. Seizures and flying airplanes don't go together. Well the other guy never applied for an FAA medical and he wants to buy his light sport aircraft he goes to his private physician his private physician says well Joe you know I know you've got this seizure disorder you're owned to Latin and you seem to drive your car and you know driving a car must be like flying an airplane and this guy is legally entitled to use his driver's license and fly his light sport aircraft. Not fair. But we at the FAA are government employees and if indeed we are aware of honest to goodness medical reasons why you should not get a medical certificate in flying an airplane we are obligated ethically obligated to act and then work with you to get a special issuance or whatever process need be to do it the right way by the rules through the proper channels and with that ladies and gentlemen I consumed 41 minutes so we have 19 minutes for questions. Hi Jim thanks very much I'm not going to pin you down on the dates for the SSRI changes but can you pin down the time frame on the final rule for the increased duration of medical certificates? You know Gary it should be a matter of a couple of weeks which in Washington speak might mean a couple of months but we are in final rule and even though you have to have these meetings with the economist and the lawyers the enforcement lawyers and the statutory lawyers and they can always throw you a curve at the last moment I really, really am optimistic that we'll have that nailed down in a couple of months. So definitely in 2008? Definitely probably maybe in 2008. Thanks. I have a question what do you feel about extending the status of the sport pilot to the level of private pilot certificates? I mean you will not be out of business you still have the commercial part 21 and 135 but I say that because I understand in UK is going in that direction so what's your comments? So you're talking about using a driver's license for a third class medical certificate? Yeah and that's certainly a logical extension that has been brought up before I will tell you that in talking to my counterparts in the UK that they have a medical system that is much different than ours with their national health care system certainly many of their private physicians have have a better idea about the patients that they follow and though are not to get into the pros and the cons of our different medical systems but certainly you can get away with a lot more in terms of medical diseases and eyesight issues and other problems in the United States than you can in the UK so I for one know that there are enough third class pilots that have such significant medical conditions that I think it's in the best interest of our national airspace to keep that a medical certificate you know it would certainly help with FAA resources if we were able to do that but if you look at the medical causals if you look at mishaps that have been because someone has had a heart attack or a seizure or a medical problem in flight that has led to a mishap there are few and far between but those that have happened are all in the part 91 in the third class variety so they're out there and I I think from my parochial perspective we would be on safer sounder ground if we were to keep the system as it was just for a clarification as some of us age a little bit we may elect to go from private pilot to the sport and without any you are allowed to simply allow your medical to expire okay and then that doesn't raise any issues you know I'm assuming you're healthy and everything else expire okay at the end of its term okay and then you go into the sport aircraft category and from there on and that doesn't raise any issues you are correct sir you just let it expire and as long as you weren't denied or suspended or revoked you're good to go it's an instructor pilot who failed to renew his medical after having an MI still able to conduct a biennial flight reviews you know that that would be a question that I would need to get help with from my flight standards brethren I think the answer is yes but I'm really not sure you're going outside of my territory there that's as long as the other guy has a valid medical you see there's a lot of people here that know more than I do which is not uncommon could you give some general information on the diabetic condition with pilots and what the limits are what they consider a diabetic pilot what number and what ramifications and so forth should we be aware of we certificate a lot of diabetic pilots first of all many you know you can clinically have diabetes that is controlled by diet our guide that we provide to aviation medical examiners of what we teach to aviation medical examiners tells them that if you have diet controlled diabetes they can issue the certificate they need to get some testing done but once their testing is done and their blood sugar is under control the AMEs can issue a regular certificate and then there's the category of the diabetic that needs the oral medication used to be called oral hypo glycemic but now there's anti anti hyper glycemic and there's all category of diabetes medicine and basically this is one of those AME assisted special issuances whereby if you show us with a hemoglobin A1C which is the test that looks at your control over time if you get a current status report and the diabetes isn't affecting your vision or your neurological status or your renal status and your A1C is reasonable with you on oral medicines will give you a six year authorization whereby once we give you the medical certificate you get current status reports and testing and your AME can then issue you for the next six years lastly there is the category of diabetics on insulin and here we have very specific protocols whereby third class pilots third class only even though we take grief from some first and second class pilots that are diabetic and would like to see the rules apply to them right now it only applies to third class pilots because of the risk of hypoglycemia but we have some very specific rules about how you test yourself taking snacks along when you'd be required to land and I can't rattle those off the top of my head but they've been used for over a dozen years now and we've had great success with the safety of our diabetic pilots that use insulin that have been able to safely fly you were talking about the blood test what is the FAA sealing for that blood test on sugar content do you know? you know it depends on the laboratory every laboratory has different normals the AC1 test or whatever I thought the FAA had a sealing that if it was .9 or something and above you're considered are you familiar with that? I am familiar with what you're talking about but I'm not going to stick my neck out on a number but it's it's very conservative it's very reasonable we do not expect our diabetics to be as tightly controlled as some diabetologist some medical doctors would like because the real risk of diabetes is not the hyperglycemia the real risk of diabetes is using insulin or one of the oral medications and being hypoglycemic whereby you could actually lose consciousness, have problems concentrating so we like our pilots to fly sweet I had prostate cancer in 2005 had prostate removed in December of 05 and through your special issuance program I received my medical back about 6 months later which is wonderful but I continue to participate in the special issuance program with a PSA of 0 how long do I have to participate and is there a method for early out out of the special issuance? you know that varies that's a real good question that varies on a case by case basis this question comes up all the time in people that have had cancers like melanoma cancers that may have been lung cancers or liver cancers, the kind of cancers that are more likely to metastasize to the brain and then they have to get an annual MRI which is not cheap so lots of them want to know when is this annual testing going to end and by and large we look at 5 years as the time that we would expect someone to get either an MRI or a PSA and of course a PSA is a whole lot cheaper than an MRI but yes there is a way that you can resume special issuance and typically 5 years would be the rule of thumb ladies and gentlemen thank you very much thanks a lot very interesting stay right here we're still on camera


Regis what do you think is going on with this team right now it doesn't seem like you're that far removed from that stretch where you've won 13 in a row and it seems to have been really a struggle over the last 10 games yeah I mean you know if I knew exactly what was going on I think we would you know have an idea a better idea of how to address it and put a stop to it but you know unfortunately we just haven't been playing good baseball this last week or so I know for a few weeks and even a month or two we were playing much better baseball and you know everybody talks about the winning streak and obviously we played pretty well during those two weeks but um you know since then it's just been hard to get things going you know not doing a good job of picking each other up throwing that bats away from time to time and we just need to be better what's the level of concern when you see a guy like Derek Cole exit the game early you know pretty high I mean he's obviously a guy that you know is very very important to our team and to our chances going forward you know of getting into the postseason and make them some noise once we get there so you know he's a guy that we need that we need healthy obviously you know I think you know hopefully hopefully it's not something that causes him to miss any if not much time so you know hopefully he'll be back out there in five or six days taking the ball for us because we all know how good he is and and how important he is to our team


section one scenes of madness one Tony Perkins wrote that out for me he actually wrote out to he wrote out the number two Corinthians I took exactly what Tony said and I said well Tony has to know better than anybody to I don't want to get into it it's very personal you know when I talk about the Bible it's very personal so I don't want to get into verses I don't want to get into it the Bible means a lot to me but I don't want to get into specifics three proverbs the chapter never been to envy I've had that thing all my all of my life where people are bending to envy for well I think many I mean when we get into the Bible I think many so many and some people look an eye for an eye you can almost say that that's not a particularly nice thing but you know if you look at what's happening to our country I mean when you see what's going on with our country how people are taking advantage of us we have to be firm and have to be very strong and we can learn a lot from the Bible that I can tell you and notes this is partially polemical section to introduction it is utterly predictable that an American who does work partially in biblical studies would point to the inverse relation between a boobish president in biblical Texas after all one is more likely to defend what they spend their time working on preferring to note an inverse relation between acceptable use and political evils of course then it is a tantalizing prospects attempting to gesture to a certain type of biblical irrelevancy that the Trump predestive presidency points toward but by a certain type of biblical a relevancy I do not mean to suggest that the Bible has disappeared in the political sphere nor something equally silly like that theology or religion are no longer politically relevant we saw too often theological apologies for the actions and words of Trump by people with popular religious authority popular preachers and people like that for instance in my own home state of Texas Robert Jeffries of first Baptist preacher Trump as well consistently appeals to the religious persuasions and affections of his audience he saturates his audience with concerns for the evangelicals for the favorings of Christianity for the freedom of pastors and church people to be able to do and say things that many of them believe have been eroded during the near decade of Obama's term there is clear religious tension that Trump plays into a third section shifts I mean to suggest that the Bible's relevance can be read as having shifted though even noting this does not enter into bold claims about the intransigence of the shift a shift implies context in the context of this shift is narrow in scope and point of fact it is quite obvious that the Bible has not left political discourse in this sense one can note a subsection of shift one which proceeds the contextual shift to be kind of sort of flesh out later the Bible in one sense was more often than not at the forefront of political discourse but it appeared in otherwise strange places places that would not have been obvious or interesting in the last few elections in the US regularly images of Jesus are used in political left discourse particularly and memes that attempt to either make a serious point about how a particular understanding of for instance Jesus's ethic would work out if transposed into the 24th century nothing too novel as this is par for the course and public discuss discussions of ethics in the West or in means that parody a conservative Jesus who for instance withholds health care because a lame man's inability to work or who asked for money before feeding the 5000 this shift perhaps an example of the radical Bible that James points to in British politics is only a relative shift seen as an intensification because of conservative atrophy and the ease of media dissemination in the world of social media of course the radical Bible tradition hasn't been utilized effectively in any major US elections that I'm aware of except maybe recently much less presidential elections despite some gestures by people like Bernie Sanders to the ethos of Jesus he spoke to Liberty University for instance and yet he cannot be ignored that presidents in the US from Obama to Bush to Clinton and beyond have used biblical materials in support of principles especially broadly American principles Bush frequently employs biblical materials not to mention theological ideas to supplement his speeches he frequently mixes in biblical motifs and imagery talking of a light on a hill or utilizing the familiar Samaritan parable Clinton Bill Clinton as well was quite open about his relationship to the Bible in his own faith even employing if sometimes in a bit of a convoluted way versus from the Psalms or Proverbs and Obama's well often quoted from the Bible whether from Matthew or famously from 2 Timothy during I believe maybe the National Day of Prayer I believe it was or maybe it was the National Breakfast Prayer or whatever it's called too many prayer breakfast national things going on in the US however Trump's relationship to the Bible mirrors his own or his use of other props for public image quotations and illusions are few and far between and often egregious his use is often a misuse much like his use of relational props family familial parading boasting of properties and wealth bravado and machismo we can recall then his early recollection of a favorite Bible verse one which didn't exist there is no despite his confidence verse about not bending to envy in Proverbs and it should be particularly egregious that this is said to an evangelical interviewer someone who should kind of pick up on that but these incidents are not isolating I isolated they occur over and over again in the course of a year and a half long presidential election something that's much too long section for evangelicals it must be remember remembered that Elizabeth Diaz wrote an article that appeared in Time magazine on September 26 2016 called quote the pastor who prays with Trump and quote it is hard to not categorize this as a puff piece despite the fact that I can't really point to any determinate political agenda behind Diaz's behind her persona that justifies a pro Trump slant this comes at a time that isn't the end of a long year and a half battle one with numerous gaps but also a lengthy war that seems to justify over and over to observers that Trump doesn't have much of a relation with biblical materials much less with Christianity in an obvious hands-on sense which would be a necessity one would think for an evangelical voter here in this piece Diaz points to a near decade and a half friendship or at least an acquaintance ship between a well-known televangelist Paula White and Trump the two have allegedly had a long and fruitful relationship one which included Trump attending numerous Bible studies with white Diaz admits of course that Trump spirituality has been a mystery during the campaign and again this is coming right before you know the election happens this is a mystery despite his public announcements of his Christian religiosity but we have to note the ambiguity in this whole mess Diaz in this piece points to the 15-year history of Trump's relationship with the Pentecostal leaning non-denominational televangelist his prayer sessions with her his attendance of Bible studies and other things of this sort you seem to fly in the face of his numerous public biblical gaffes and also it's strange that these things would be pointed out while also noting the mystery behind Trump but about these gaps we recall his not knowing the usual name way of naming some biblical texts and at the quote at the beginning you know of course he defers blame onto someone else his fabrication of texts his refusal to name his favorite texts and in almost absolute lack of biblical references much less relevant or carefully picked ones though they do occur some after the campaign they can be heard when he stays on track sticking to prompts in some recent speeches but they do not come natural as they seem to with Bush Clinton or Obama despite all this Trump is baptized here there is a subtle shift a shift from the Bible being a political tool that occurs naturally to the politician one of the politician wields as a weapon of persuasion to a political tool utilized by the supporters of the politician Christianity imposes its imperial substance baptizing the hapless fool while also fooling itself thinking it is truly converted to savage. We recall the often used evangelical defense of Trump he is only a baby Christian he is just learning how to be like Jesus and how to love the scriptures but we have always read what we want to into our politicians after all Obama was the paragon of leftist virtues despite being a raging neoliberal who continued usual American foreign policy. Section five what shift does it matter to continue on with this comparative game to try to gesture to a shift that I can't quite name my finger on I think requires noting that the Trump with Trump it is precisely because he doesn't proof text the Bible that he doesn't have a Bible in the same way that someone like Thatcher or Corbin or Obama or Bush to name a few did we find a peculiar political authority in glances towards texts even when they don't show up in the places we would expect to politicize text to Bush's biblical authority come from his public religious life his autobiography his continual use of biblical figures like Moses or books like acts to underwrite his story no matter how self affirming and gratuitous the use of these texts may be Trump however uses the Bible as a political piece but is largely evacuated of similar uses the Bible is a sort of hollow prop to be held and thrust in the air as he often did he issues a stronger oral dimension that one sees or one hears in other leaders who have utilized the Bible the words the phrases the stories that remind catalyze emotion and call forth an imaginative aspect for the especially evangelical listener or absent but at this late stage I have to admit the ambiguity and in fact the contradiction between my early inclination put quite strongly about shifts in the Trump era it isn't proper to speak of a shift in the Trump era because this era has been in flux since June 2015 when he bans race straight out of the gate with an eye fixed on the evangelical contingent a subset that he overwhelmingly won despite the overall lack of comparatively biblical references illusions and motifs in his campaign he has deferred to the safer route finding time to quote verbatim some passages during very specific post campaign moments most notably his inauguration in the national prayer breakfast back in February where he quoted greater love hath no man than this that a man lay down his life for his friends um it's only now that we that he begins to make tendentious connection between perhaps starting a formation of a Trump Bible where values are connected to texts such as in the national prayer breakfast where this quote that he references is you're gesturing to a soldier who was martyred for the cause of American freedom as he might put it well he wouldn't put it that way but I'm not sure then what we can actually say about Trump's Bible but it seems clear that if it materializes if it ceases being a specter it does so in a very public way and it does so over time it isn't thrust upon the scene in a mature state like some of his people that came before him but instead is like Trump himself quite clear in its own strange imperfections that seem to call to the initiated like a moth to the flame that's it


Welcome back to my channel and welcome if you're new here I have a recipe video for you today, and this just isn't anybody's recipe This is a zero smart point clean eating whole food Dinner or lunch recipe and it is absolutely Delicious now. I don't recommend always eating zero point meals I think that zero point food should be the foundation of your meals So I would recommend adding something with points to tonight's dinner like maybe an additional vegetable side Or even a starch side potatoes rice may be a slice of your favorite bread But regardless this is a zero smart point recipe on both the blue and purple plan and Extremely low smart points on the green plan and again de Delicious, so let's jump right in to tonight's zero smart point dinner So for tonight's zero point dinner, we are making Italian turkey burgers. I've made these before they're absolutely Delicious, we're gonna pair this with some spaghetti squash some zero point marinara And if you do want to add some points to this meal, you can do what I'm gonna do Which is just taught mine with a little bit of Parmesan cheese So let me show you first what is in the turkey burgers and then how we're gonna make the spaghetti squash mix First you're going to need one pound of 99% ground turkey Definitely save the points and get the 99% because by adding the zucchini it makes the burgers nice and moist anyways You'll also need some salt and pepper and some seasoning. So we're gonna start with some basil oregano onion powder Garlic powder and then you'll need some zucchini and a spaghetti squash. We're just gonna pop this in the microwave That's actually the first thing that will do and then I just have a jar of the skinny-ish dish Crock-pop marinara, which is my go-to at zero points no matter how much you have and it's delicious So we're gonna use that as the marinara sauce. So let's get started on tonight's dinner So the first thing I'm gonna do is go ahead and chop my spaghetti squash in half I'm going to place it on a plate with a wet paper towel and get this into the microwave It usually takes eight to ten minutes for the squash to cook and then we'll get started putting together our turkey patties Standing on the porch in the night thinking I will used to be way back in time In time They put on the grass open skies. I can feel it every time I close my eyes Next I'm gonna go ahead and take two Zucchinis and I'm going to grate them onto the plate here and that is what we're gonna add to the turkey burger meat So there's our zucchini. I like it grated really small. That's why I use my zester So we're gonna go ahead and add it directly to our bowl and literally these are the easiest patties to make We're then gonna go ahead and add our pound of 99% ground turkey And then we're gonna add some seasonings for seasonings. I'm going to add some garlic powder I will be putting the original recipe on my website, of course But I'm just gonna season it to my liking and we like our food pretty heavily seasoned and then I have some onion powder here I'm gonna go ahead and add the same of that I'm doing about maybe a teaspoon and a half of each of those seasonings Then I have some bay leaves here I'm going to do about a teaspoon of bay leaves and about a teaspoon of Oregano and again just because we like our food with a lot of flavor and Then lastly I have some salt and pepper here So I'm gonna do a pinch which is roughly a teaspoon of each of those as well And then we are going to dig in with our hands and mix this together so to mix this up the best way to do it is just dig your hands in and Go ahead and get everything combined the meat with all of the Grated zucchini and then we are going to just roll these into four equal patties So these are gonna be good size turkey burgers And I like to put them in the oven at about four twenty five Until they're cooked through and then you can always pop them under the broiler For just a couple of minutes to get them nice and brown and crispy And we just continue to let that spaghetti squash cook I put it on for another five minutes in the microwave and then in the meantime We'll make sure our marinara sauce is ready to go. So let's get these turkey burgers up and ready to go So here are our turkey burgers. These look so good So I'm just going to get these into the oven until they're cooked through and then once the spaghetti squash is done I'll show you what to do next. So I just pulled out the spaghetti squash We have a temporary very cheap not-high wattage microwave It took 20 minutes to cook the squash But normally in a regular microwave eight to ten minutes once it cools I'm going to shred it with a fork and put it into this bowl here And then we'll season it up and we'll be ready for when our turkey burgers come out of the oven All right, so we cleaned out the spaghetti squash here It is in our bowl And then what I like to do is just season it with some salt and pepper And then that way when we go to put our turkey burger and marinara on it on it The spaghetti squash is a little bit seasoned as well So I'm going to give it just a little pinch of salt and pepper Give it a quick stir and then we'll set this aside Our turkey burgers are just about done and in the meantime I'll warm up the marinara So I just pulled out the turkey burgers. They look great I'm just warming up the spaghetti squash with the sauce on it We'll put dinner together and I'll be back to show you my dinner and give you the smart points So here is my completed dinner. So I have half of that spaghetti squash. It was really small So I just went ahead and did that into two servings marinara We topped it with one of the turkey burgers And I went ahead and topped it with two points worth of the Kraft Parmesan and Romano cheese I thought I had fresh cheese and I did not so we went ahead and use that So this is actually zero smart points on both blue and purple which is awesome I would always recommend pairing a zero point dish with something else So I added the cheese for points It is three smart points on the green plan and then if you add any cheese you could add the points for that So this is my dinner and it is a total of two smart points Thank you for joining me on another WW recipe of video again zero smart points You can't go wrong and I'm telling you it's so good my husband loved tonight's dinner You can see that I went ahead and added two smart points worth of Parmesan cheese to mine to bump up the smart points just a little bit and Compensated for all of the zero point foods throughout the rest of the day But this is as clean and whole food as you can get and again super delicious by adding the zucchini to the ground turkey It made the burgers nice and moist and so good I will have both this recipe and the recipe for the skinniest dish marinara on my website So definitely check out my website to get today's recipe It is linked down in the description box below also in the description box is the link to head on over and join my Facebook group We'd love to have you be part of our community over there and the links and discounts to all of my favorite things So definitely check out the description box if you're new welcome welcome subscribe to my channel hit that bell So you're notified when new videos are uploaded again. We'd love to have you join our community We're always looking for some more wonderful people to be part of this amazing community Give this video a big thumbs up if you love these recipe videos leave your comments down below Let me know if you're gonna give this recipe a try highly recommend Thank you guys so much for watching and I'll see you all in my next video. Bye