This week's episode is sponsored by Change. Change is an online mentoring program that teaches people with no experience how to create a real profitable online business and e-commerce. I have been working with Ryan at Change for a few years now and attended many events and got to meet the amazing community of like-minded people. These guys are the best of the best. The support these guys offer is personal, no bots or employees. There's no experience needed but like anything in life it takes time as it's a real business with real results. For more information go check out Ryan on Instagram at RyanGyB and he will guide you through the steps to help build a successful business. You can now follow me on all my social media platforms to find out who my latest guest will be and don't forget to click the subscribe button and the notifications bell so you are notified for when my next podcast goes live. As I went to put the seat back I seen him coming round you know and I just opened the door and as I got one leg out he already had the gun there. I remember when I was honoring man Jesus Christ I was like there was all like the guns the harm response helicopters you know everything above you know it was like shut out the city off to take me to the court. I said you ain't got your bullet proof vest on. I said what do you need a bullet proof vest?  I was still heavily being looked into with the police when I was in prison you know I was still under constant investigation for the fall, full duration and as soon as I stepped out the gate it continued.  That switch of being a criminal to then being training people to be the best fighters on the planet? Even the people upstairs you know the two people you know there was gag bound and gag light on the bed stretching out you know.  All as I remember it was a was a lovely young fellow you know listen if it would have meant giving our lives for him to win that fight we'd have done it. I was in the cate again and I was looking multiple charges you know if I would have went wrong I was getting 30 plus years. And boom we're on in today's guest we've got big Peter Fury. Peter. Good to see you James. Absolute pleasure brother. Pleasure. Yeah listen a man is very well respected all around not just the UK but the world for the things that you've achieved. Listen some dark things a lot of positives like so many positives and listen that's life. I know a lot of people who's very well connected with you and they speak highly so highly of you let us. There's not many people who show respect but for me I don't I like a lot of people who don't respect a lot of people and you're one of those men who I do respect. Thank you James I really appreciate it. Yeah you try and be right with people that's the thing go for your life and that's it you know in in all parts of my life I've always been respectful to to the people that I've been around associated with and the friends I've got over the years. Yeah that's all you can do. That's all you can do yeah that's it and every try and better yourself every day don't you. Yeah before we get into everything Peter I always like to go back to the start of my guests get about understanding about you how you function kind of where you grew up and how it all began. What can I say I've probably began when I was about nine years of age so yeah gone way back so yeah just worked very hard as a young fella. What age? From about any age nine, ten, eleven we put to work very young so yeah. Is that the traveling nature straight away? That's the traveling culture yeah that's not just for me that's that's what your travels was you know. Yeah and your mum is at Hume and Sissy is that correct? That's right. How are their parents? Best parents in the world you know the best. I was very very lucky to have them you know he brought us up well and very proud of me mother and father what they've done for us. Yeah how about school? I know a lot of traveling community don't really go to school but what about yourself? Yeah I went to school till I was about the big age of eight. So you know I remember it well I went from five to eight so I spent about three years in different schools all around the country week a week there two weeks here two weeks there you know doing a bit of colouring and stuff but yeah but no I never I never learnt anything at school I wasn't interested in it. Did that become the norm for you? Like with did you understand what a traveler was then? Is it a very young age? Just the way we lived it was our culture you know so we always I don't know I never differentiated as being any different for anybody else but the people did in the school and stuff so I knew we was I knew we was a bit different but it was a bit different back then than what it is today. Were you ever like people know when you were different then? Did people try and bully you? Yeah the normal stuff in school play grounds we had it you know there you'd be being called names and stuff and singled out nobody would play with you and all that type of stuff you know but it was listen they done me a favour at the end of the day because it makes you stronger doesn't it?  What was your first ever job? First ever job was probably doing a bit of snow clearing of drives and stuff when it was snowing that's all I can remember you know little bits and pieces jobs for people and going around the houses where we lived and asking them if they needed any work and stuff like this you know any manual jobs so we do a little bit of that. What did your mom and dad do? My mother and father they was like into mainly a little bit of everything but mainly they'd be door-to-door sales people selling carpets selling selling bedding you know all this type of stuff that's what they were doing. Big family here Peter as well was it three four brothers? Yeah four brothers including myself yeah. How was it with everybody at that time? It was good we just all stuck together we played together and you know we just stuck together as a family was a good type family unit when we were younger. What was your first proper job? Proper job as in as in what? Legit. Legit well there was all legit you know when I was growing up you know you know done many things done many things put me hands to everything about probably the motor trade I got involved in that when I was shortly after I was married.   I met my missus when I was young 14 and we've seen each other for two and a half year and we got married uh just two months shy of being 17 you know so I was still 16 you know I just remember it went into a registrar in Bradford we just got married and uh that was it got in a van next day work that was it there was no there was no only moons then there was nothing because there's no money you know who was hurt there was that but you know look you know when um when there's two people I remember it you know it was when we started out you know we never anything so but we was on our own we was on our uh you know we didn't get any end doubts we've done our own thing and we struggled along how did you how do you manage to keep a relationship for so long Peter Hitler what's the what's the main ingredient do you think?   Very little you know look when you got no money you're happy to make anything that's the thing people can lose word if come from because when you've got no money you'll settle to do anything you're just very happy to make it to the next day you know that that's how it was you know you get a few quid and think oh you know thank god I can you know we can eat you know we can do things we can pay the rent or whatever you know that's that's the way it was back then so you're always uh you're always fighting to keep you head above water where were you living then? I was living on uh on uh travellers sites gypsy sites um mainly in uh Akrington Lancashire we was there for a bit. How was it there? It was good they enjoyed it you know we stayed on a place called Jim West Place in Ozil Twizzle and it was a good people you know travellers again they won the place you know and they've got on with the people very very well why why is they travellers why are they such tough bastards like how why is that blood that ingrained the fighting spirit like where does that come from?  Scousers are mad bastards aren't they?  Definitely you know one wasn't involved in crime at all you know I was only young uh where you know I got went a bit deaf and um and that was just that was just a neighbor you know that was someone like was mad you know so I was that I was lucky there to be honest with you because uh if hadn't pushed the gun up it went up I knew he was gonna do it but I was jammed because my feet was behind uh I was reversing the car out for me old man actually and I was uh and obviously he's a lot shorter than me so as I went to put the seat back I seen him coming round you know and I just opened the door and as I got one leg out he already had the gun there so as he put the gun there I just put my head down straight away just natural instinct anyway but it went off you know so I was lucky how do you think you're still here today Peter? I think uh maybe I have no idea I have no idea but I've been I was lucky or maybe the Blessed Lord had another plan for me you know and I found my faith later on in life as well um it was always brought up with faith anyway but I definitely didn't practice it you know you couldn't say there was no christian in me I wasn't I wasn't a good person when I was younger so seeing you came out then and straight back to business then just if you kind of try and provide for the family that was a more pressure on you over in spain or did you find that a little pressure of you were being in a different country?


Who do you trust more about COVID? A legendary rock star? Or the world's most popular podcaster? The biggest takeaway from the latest culture war battle is that you get to decide. Neil Young pulled his music from Spotify, but you can still listen to his discography easily enough. And even if the world's most popular streaming service had dropped Joe Rogan, he'd still have all sorts of ways to reach his massive audience. The scandal du jour reminds us that radical free speech is alive and well. In a profound way, it doesn't matter if somebody somewhere wants to block it, because it'll be available just a few clicks away, no matter what. Young fired the first shot by telling the streaming service Spotify that they had to choose between keeping his music on its platform or continuing to host Rogan's podcast. Young says Rogan spreads dangerous misinformation about COVID. Spotify updated its rules about publishing content that promotes dangerous false or deceptive medical information about COVID-19 and other diseases. But it stood squarely by its marquee star, whose contract is worth more than a reported $100 million. Young also immediately inked a deal with satellite radio provider Sirius, and his music remains widely available on YouTube, Apple Music and his own website. Same goes for Joni Mitchell, who quickly followed suit. Rogan also keeps rolling on, telling listeners via Instagram that the whole point of his show is to have interesting conversations and let people come to their own conclusions.  Young's call for Spotify to boot Rogan is visible, but his decision to pull his music is worth celebrating as an assertion of his right not to do business with an entity that offends him. Cancel culture is a real concern, but its impact is limited by the limitless nature of how easily stuff gets made and distributed these days. Our world of cultural plenty was simply unimaginable in 1966, when Neil Young cut his first album with Buffalo Springfield. Back then, there were a handful of major record labels, a few movie studios, three national TV networks, and only AM and FM radio. Gatekeepers actually had a lot of power to suppress speech they didn't approve of. Nowadays, even the mighty Spotify is just one streaming service among many. The problem isn't that our choices are too restricted. It's figuring out how to triage what to read, watch, and listen to because there just aren't enough hours in the day to take in all the amazing movies, shows, songs, books, and podcasts available. We get to decide what and when we're going to watch and who we're going to trust, which is a great outcome, but one we should never take for granted.


The purpose of the pretrial conference, as described in rule 17.1, is basically the same as in a civil case. That is, to get the parties, the lawyers, and the judge together in order to discuss matters that will promote, in the words of the rule, a fair and expeditious trial. That means simply, the conference should be used to address both case management considerations and substantive issues. Under rule 17.1, you may conduct one or more pretrial conferences, either on motion of any party or upon your own motion. At the conclusion of a conference, you should prepare and file a memorandum of the matters agreed upon. If time does not permit, then I have a transcript of the hearing prepared. If any admissions are made by the defendant or the defendant's attorney at the conference, they may not be used against the defendant unless they are reduced to writing and signed by the defendant and the defendant's attorney. An example of that would be a stipulation which would avoid the need for calling certain witnesses at the trial. In fact, one of your major goals at the pretrial conference should be to get the parties to agree to enter into stipulations. Why? Because stipulations help streamline the trial. For example, by making it unnecessary to call foundation witnesses. Of course, the foundation for a particular item of evidence may be hotly disputed. But when that's not the case, it's arguably in everyone's best interest to agree upon a stipulation that obviates the need for a parade of witnesses. You might also discuss with counsel the possibility of stipulating certain facts, such as the nature of a controlled substance or whose handwriting appears on a document if they aren't really at issue in the case. I'd like to suggest two main ways of using the pretrial conference to obtain a speedy and efficient trial. First, use the conference to ensure that there is good advanced preparation for trial by counsel. And second, resolve as many substantive issues as possible at the conference, so you don't wind up interrupting the trial to resolve them. At the pretrial conference, you should check to see if all motions have been filed as ordered, handle requests to allow late filing of motions, and set dates for hearing motions if you haven't set them previously. In my court, as in Judge Hatter's court, all of these deadlines are set at the arraignment. That eliminates the need for a separate pretrial conference prior to the hearing on motions. Instead, schedule a combined motions hearing and pretrial conference for the same day. And on that day, you rule on pending motions, then begin your pretrial conference. Of course, sometimes it's necessary to set separate hearing dates for special evidentiary matters. That's true. For example, there may be a need to set a separate hearing for a Miranda-related motion to suppress requiring the testimony of five or six witnesses. But I would strongly counsel against hearing any type of evidentiary motion on the date of trial. Doing so will inevitably undercut any previous ruling you've made setting a deadline for completion of plea bargaining. It will also have an adverse effect on counsel's trial preparation. When you come to the day of trial, you should be in a position to proceed immediately with jury selection, opening statements, and presentation of evidence. At the pretrial conference, you should also revisit the question of discovery in depth. You need to know whether or not discovery has been completed. If it hasn't, you should ensure that the remaining discovery is completed pursuant to a specific schedule, which you said, and which will not interfere with other deadlines or hearing dates you've said in the case. Now, as a new judge, you may find it helpful to use a checklist of some sort in making sure that all discovery-related matters have been completed. This is also the time to resolve any non-evidentiary discovery or Brady issues and set firm deadlines for compliance with your rulings. Now, in a civil case, the pretrial conference would naturally be an occasion to inquire into the status of settlement negotiations. But under Rule 11E1 of the Rules of Criminal Procedure, the court is specifically prohibited from participating in any discussions regarding plea negotiations. You simply cannot involve yourself in plea negotiations in a criminal case. What you can do is encourage the parties to engage in those negotiations and set a deadline for presentation of any negotiated plea bargain to the court. I will not consider any plea bargain presented to the court less than seven days prior to trial. Unless there are some exceptional circumstances. Whether or not you choose to use a deadline like this in your court is entirely up to you. Some judges have told me they would never consider using such a deadline because they're always happy to take a plea of guilty, whether it's a week before trial, the morning of trial, or a week into the trial. The reason that I have that rule and use it is because I believe lawyers will typically comply with that deadline if they know you're serious about it. Also, by following this rule, I keep to an absolute minimum the number of Monday afternoons where I find myself without a case to try because the lawyers did not give any advance notice of successful plea negotiations. Now let's consider another issue related to discovery. The production of statements of witnesses other than the defendant. Of course, we indicated earlier that prior statements of witnesses aren't discoverable until after the witnesses have testified on direct examination. So this may seem like a trial matter at first bludge, but as you'll soon see, there are strong reasons for making it a pretrial matter as well. Rule 26.2 of the Rules of Criminal Procedure, entitled 18, section 3500 of the U.S. Code, govern this particular area. They address what we commonly refer to as Jenks Act material. The basic requirements of the Jenks Act are set out in section 3500. Section A of section 3500 specifically provides that any criminal prosecution brought by the United States, no statement or report in the possession of the United States, which was made by a government witness or prospective government witness, other than the defendant, shall be the subject of subpoena, discovery, or inspection until said witness has testified on direct examination in the trial of the case. Section 3500 goes on to provide that at the conclusion of direct examination, you are to order the United States to produce any prior statement of the witness, which relates to the subject matter the witness has testified about. That is, if the defendant so moves, if the entire content of the statement relates to the subject matter the witness testified about, then you must order it to be delivered directly to the defendant for his examination and use. If the government claims that part of the statement does not relate to the subject matter of the testimony of the witness, then you must order the United States to deliver the statement to the court for inspection in camera. You are then to excise any portions of the statement which do not relate to the subject matter of the testimony of the witness. Now, once the material is excised, you must direct delivery of the statement to the defendant for his or her use. Under Section D of the Act, if the United States elects not to comply with your order to turn over the statement, either in its completed or redacted form, then you must strike the testimony of the witness from the record and determine whether or not a mistrial is appropriate. Rule 26-2 is substantially a restatement of what is contained in the Jenks Act. However, it should be noted that Rule 26-2 requires the disclosure of prior statements of defense witnesses, as well as government witnesses. The rule is designed to place the same disclosure requirements on the defense as the Jenks Act places on the government. Like the Jenks Act, however, Rule 26-2 does not apply to prior statements of the defendant. Now, here's where this becomes a pretrial matter. The Jenks Act provides that you may recess the trial to allow defense counsel a reasonable opportunity to examine prior statements of government witnesses prior to cross-examination. And so you should, if counsel has not previously seen those statements and had an opportunity to incorporate questions about them into his or her cross-examination. But I would suggest you don't want to be in the position of recessing your trial each time the direct examination of a government witness is completed. As we discussed earlier, you cannot require the prosecutor to disclose Jenks statements in advance of trial, but you can and should strongly urge the prosecutor to do so. Again, anything that can be done to encourage an open file policy should be done. The alternative is that the trial will be delayed in order to allow the defendant to examine the Jenks Act statements of each government witness before cross-examination. And frankly, I have never had this occur. Another important thing that should be discussed at the pretrial conference is the approximate length of trial. You may be thinking, why worry about this? Most criminal trials are short, two or three days in length. But when you're handling a complicated criminal case or a conspiracy case with multiple defendants, it's very important to get a handle on the shape of the trial at the pretrial conference. In my experience, lawyers aren't always prepared to give you a reasonable estimate on the length of trial. If you simply ask the prosecutor or defense attorney how long do you think this trial is going to take, you won't get a response that will be helpful to you in scheduling. You have to be more specific in your questioning. You need to press the attorneys to specify the number of witnesses they intend to call. Then ask them how long they estimate it will take to present those witnesses. In a case that's only going to take two or three days to try, perhaps you don't need to spend a lot of time on this. But in a complex criminal case, I've taken to going to somewhat unusual lengths to try and identify what the shape of the trial will be. It's important to remember that many of the things you'll be trying to accomplish pretrial can only be accomplished by agreement as opposed to order of the court. That means you should begin trying to shape those agreements during the early stages of the case. For example, if your U.S. attorney is willing to engage in an open file policy on the case, then it's reasonable to expect that the parties can reach an agreement to exchange witness lists 14 days prior to trial. That is, with the understanding that the witness lists are not written in stone, that names may be added or subtracted. But in cases involving hundreds or thousands of pages of discovery and potentially hundreds of witnesses, it's ultimately in everyone's best interest to know who is most probably going to testify or not testify. The same with exhibits. That's right. Try to get the attorneys to agree to exchange exhibit lists at a pretrial conference. If you succeed in that endeavor, you'll end up with something similar to the sort of final pretrial you'd have in a civil case. Some complex cases involve potentially hundreds if not thousands of exhibits. Again, it's usually in everyone's best interest to know what exhibits are probably going to be relied on at trial. Rule 12 of the Federal Rules of Criminal Procedure discusses pleadings and motions. The rule requires the following motions to be filed prior to trial. Motions raising defenses and objections based on defects in the institution of the prosecution. Motions raising defenses and objections based on defects in the indictment or information. Motions to suppress, requests for discovery under Rule 16, and requests for severance of charges or defendants under Rule 14. Most judges will tell you to let the defendant file whatever motions he or she deem necessary. Nevertheless, it's important to recognize that some motions filed by defense counsel are filed primarily for the record and commonly referred to as boilerplate. Basically, their file so the record reflects that defense counsel is diligently representing his or her client. It's important that you develop the ability to distinguish between motions filed for the record and those filed in a serious effort to prevail upon an important issue in the case. You'll need to develop that ability in order to allocate your time accordingly. Very often, an evidentiary hearing is required for suppression motions. It's important that you make clear to the attorneys when you set an evidentiary hearing that you expect each side to have all of its witnesses available to testify on that hearing date. You don't want to wind up having to continue the matter so that all the witnesses can testify. Let's talk for a minute now about the exclusionary rule. In essence, the rule requires that when a right of the defendant has been violated, the evidence obtained as a result of that violation must be suppressed. That is, unless an exception to the exclusionary rule applies, or the government is able to establish that it obtained the evidence from an independent source. When a defendant files a motion for suppression of evidence, alleging violation of fourth, fifth, or sixth amendment rights, the government usually has the burden of going forward and showing that the evidence was not obtained in violation of the defendant's rights. Once the government meets that burden, the defendant has the burden of showing a violation of the constitutional right. Consider, for example, a motion to suppress a statement given by the defendant. The motion alleges the statement was obtained in violation of the defendant's fifth amendment rights. The government would normally have the burden of going forward on such a motion. The government's witnesses would establish when and where the statement was taken, who was present during the statement, the events leading up to the taking of the statement, and the matters raised by the motion to suppress. The motion might allege, for example, that the defendant did not receive Miranda warnings, or that the defendant was incapable of giving a voluntary waiver because he or she was under the influence of drugs or medication. After the government has presented its witnesses, the defense would have the burden of presenting evidence showing that the defendant was not given the Miranda warnings, or that the defendant did not knowingly waive them. There is an exception to the general rule that the government goes first in presenting its evidence at a suppression hearing. That is, of course, when a judicial officer has issued a search warrant. That's correct. If a search warrant has been issued in the case, there has already been a finding of probable cause by a judicial officer. So the defendant has the burden of going forward and making a prima facie showing that his or her rights have been violated. A defendant who testifies at one of these hearings may do so without waiving his or her fifth amendment rights. However, it should be noted that a defendant's motion hearing testimony may later be used to impeach the defendant's trial testimony. Also, suppressed evidence may be used for impeachment purposes at the trial. Rules 8 and 14 cover Joinder and Severance. A Severance motion may allege that the defendants will present conflicting and irreconcilable defenses, that there is a disparity in the evidence as to the defendants, that the defendant desires to call a co-defendant as a witness, or that the defendant will be prejudiced by admission of a non-testifying co-defendant statement implicating the defendant. We refer to the last situation on that list as a Bruton situation. Bruton versus the United States held that Sixth Amendment requires severance when the government seeks to introduce a statement by a non-testifying co-defendant, which implicates that defendant. The rationale behind the rule is that instructing the jury that the statement is to be considered only as evidence against the non-testifying co-defendant is simply not an adequate substitute for cross-examination. Which means that severance is not required under Bruton if the co-defendant takes the stand and can be cross-examined about the statement. That's right, nor is it required if the statement is properly redacted. Now the case of Richardson versus Marsh holds an out-of-court statement by a non-testifying co-defendant may be admitted at a joint trial without violating the defendant's Sixth Amendment right to cross-examination if the statement is redacted to remove any reference to the co-defendant and if a proper limiting instruction is also given. When the defendant requests a severance to allow the calling of a co-defendant as a witness on his or her behalf, you must weigh the exculpatory nature of the desired testimony, the desire of the movement to present the testimony, the willingness of the co-defendant to testify and the demands of effective judicial administration. This type of motion for severance cannot be properly considered unless supported by an affidavit by the co-defendant indicating under oath not only that he or she is willing to testify but that he or she will testify if called to testify at a separate trial. This particular severance issue is discussed in depth in the Eccles case cited in your materials. Rule 11 governs guilty pleas and allows the prosecutor and defense counsel to attempt to negotiate a plea agreement to resolve the case. However, the terms of any such plea agreement must be disclosed to you in open court and you are the one who ultimately determines whether or not to accept the agreement. Rule 11 requires that before accepting a guilty plea you ensure that the defendant is entering the plea knowingly, intelligently, and voluntarily. The defendant must understand the charge to which he or she is pleading, what the possible penalties are, and what rights are waived if the plea is entered. You must also make certain that there is a factual basis for the plea. The bench book addresses the taking of pleas of guilty at section 1.06. I would suggest you follow the script set out in the bench book. You may want to make some minor changes but using the bench book ensures that you probe those areas which must be probed if the defendant's plea is to be valid. Rule 11 provides for three basic types of plea agreements. First, charge agreements, that is, where in return for the plea the government will move to dismiss other charges. Second, non-binding sentencing recommendations, where in return for the plea the government agrees to recommend a particular sentence or not to oppose the defendant's request for a particular sentence. And third, binding sentencing agreements, where again in return for the plea the government agrees with the defendant that a specific sentence is the appropriate disposition of the case. The court may accept or reject the plea agreement at the time the plea is offered or defer its decision until after considering the pre-sentence report. In a case where there is a non-binding agreement involving a recommended sentence the court must advise the defendant that if the court does not accept the recommended sentence in the plea agreement the defendant nevertheless has no right to withdraw the guilty plea. Now a number of factors are involved in deciding whether or not to accept or reject a proposed plea agreement. These factors are addressed more fully in the program on sentencing and other post trial matters. So I'll just mention them briefly here. First of all, the United States Sentencing Commission's policy statements make clear that you should only accept plea agreements reflecting the seriousness of the actual offense. Also under the Commission Sentencing Guidelines all relevant conduct that is all acts or omissions of the defendant that are part of the same course of conduct or common scheme or plan as the offensive conviction must be included in determining the sentencing range. This means the defendant's actual sentence may be greater than that contemplated by the plea agreement. In light of that possibility you might want to wait until you consider the pre-sentence report before accepting the plea agreement. Because the court must consider all relevant conduct in determining the proper guideline sentence non-binding sentencing agreements are particularly risky for defendants. Now let's discuss the situation where a defendant enters a plea of guilty and then files a motion to withdraw the plea. What should you do? The motion to withdraw the plea could be filed as early as the day the plea is offered or as late as the day of the sentencing hearing. I almost always take the view that if a defendant wishes to withdraw his or her plea at any time prior to the imposition of sentence I will allow it. Of course I never say never. There may be a circumstance where it is completely inappropriate but my concern is this. Whatever the reason for the request to withdraw the plea if the court denies the motion it raises questions about the fundamental fairness of the proceeding. Ordinarily the government is not substantially prejudiced by withdrawal of a guilty plea. Normally if the plea is withdrawn the resulting burden on the government is no more or less than the burden it carries in every criminal case that is the burden of going forward and proving the guilt of the defendant beyond a reasonable doubt at trial. So my feeling is that even if the reason for wanting to withdraw the plea of guilty is a weak one legally the interests of justice are best served by granting the motion. Of course there's another side of the coin here too. If the defendant has entered into a knowing and voluntary plea agreement and tendered that plea to the court then allowing the defendant to withdraw the plea without good reason is allowing the defendant to break his contract with the United States. The court goes to great lengths at the time that the plea is tendered to ensure that the plea is knowing and voluntary. The defendant has every opportunity during the time the plea is being presented to the court to make the court aware of any feelings of hesitancy or uncertainty he or she may have about entering the plea. Consequently there's also a strong argument to be made that allowing the defendant to withdraw a plea of guilty in the absence of a strong legal reason to do so is a bad precedent and encourages disorder in the courts. You should be especially careful in taking a plea or conducting any court proceeding involving a defendant who does not speak English fluently. In that situation the court must make a special effort to ensure because of the language problem that the defendant is making a knowing, intelligent and voluntary waiver. It is also incumbent upon the court to provide an interpreter. The question whether or not an interpreter is needed should be decided the first time a non-English speaking defendant comes into court. Your court should have a list of available interpreters. In dealing with these interpreters over a period of time you'll find that some of them do a much better job than others. You may need to weed out those interpreters who have a good technical command of the foreign language but seem to have difficulty in conveying to the defendant the real meaning of what is being said. I think you'll find that it takes a substantially longer period of time than usual to conduct a plea proceeding using an interpreter. In such cases you'll find it necessary to paraphrase and rephrase the standard bench book questions but you must take the time and effort to do so. Some of the words and phrases that come into play when a guilty plea is being entered don't translate too well and you have to be absolutely sure that the defendant fully understands what he or she is doing. Let's touch upon one other problem area in the taking of pleas. Conditional pleas. As codified in Rule 11A2 a defendant may enter a conditional plea of guilty or a no-lo contender with the consent of the government and the approval of the court. The defendant reserves in writing the right to have the circuit court review on appeal the adverse determination of any specified pre-trial motion. For example, a motion to suppress evidence which has been denied by the court. If the defendant prevails on appeal the guilty plea may be withdrawn. Judge Mem and I trust that our discussion has made clear just how important it is for you to have a good criminal pre-trial plan. As we said at the beginning of the program most cases actually plead out during the pre-trial process but there's no way of knowing at the beginning of the process which case will plead out and which case will actually go to trial. So it's a good business practice for you as a judge to see that efficient pre-trial procedures are followed in every case. Judge Mem. I agree with Judge Hatter's comments and I would end by stressing that active case management during the pre-trial stage can go a long way toward eliminating unnecessary delays both pre-trial and during trial itself. Thank you.


For more than 35 years, NATRAL has been committed to formulating high-quality nutritional supplements to better support overall health and wellness. Careful monitoring of all manufacturing processes ensures that products meet specific requirements for identity, strength, purity, and quality. NATRAL's own extensive and uncompromised quality assurance program ensures the purity, safety, and efficacy of their products. Diligent testing throughout the manufacturing process ensures truth in labeling throughout the shelf life of the product. Here's a sampling of high-quality products from NATRAL. NIASIN TR-TIME RELEASE This no-flush, steady-release NIASIN, also known as vitamin B3, helps to maintain proper metabolic function, supports cellular energy production, and promotes healthy cholesterol levels already within the normal range. PICNOGENAL Sourced from the bark of the French Maritime Pine Tree, this clinically tested, standardized product provides high-quality premium antioxidants in support of cellular and immune health and overall wellness. B-COMPLEX FAST DESOLVE WITH COCONUT This quick dissolving, energizing tablet offers B-COMPLEX vitamins, along with cofactors vital to protein, fat, and carbohydrate metabolism, red blood cell and cellular energy production, and immune health. Should you have any questions about these or any products from NATRAL, please call, visit, or email professional supplement center.


Hello everyone and welcome to another episode of Code Emporium where we are going to talk about some sequel Now sequel is something you use in your day-to-day job as a data scientist and From my experience for the past two years as being a full-time data scientist I want to show you the kind of sequel that we actually use and how it's really different from all the Theoretical stuff that you would potentially learn in school, too A lot of the sequel that we use is definitely overlapping with a traditional education system But I just wanted to make sure that you all know exactly how we use sequel in the industry And so if you want to know that this is the video for you I wanted to make this video a little different to where we're gonna be looking at a data set And I'm gonna be giving you a few like actual questions that you would potentially be solving with sequel where you could just take a pen or paper and also like maybe open a notebook or something like that and Just type away the sequel answers to this I'll give you some like five seconds to do it where you can just pause the video and then after that We can look at the solutions together So that we cover all of our bases from like select statements to joins to where clause is to even more advanced concepts like windowing functions and then at the end of the video I'm gonna show you a couple of applications Mostly from like previous videos of like how all of these concepts were tying together to actually You know how we would actually use sequel in creating Let's say a machine learning data set or just in machine learning in general and with that let's get started Alright, so first right here. We have this huge monster of General syntax of a sequel statement that I wanted to type out I did this because a select statement is literally what you will be doing for 99% of your querying in sequel and And this entire syntax kind of has about everything that you'll even need From selects with a from and you'll have a couple of joins Where clause is dealing with aggregations with group by and having and then we have like an order by to order your data And then limit to get a number of specified rows So this is kind of the gist of all you need to know But we're gonna take a day to set and actually work with some problems because this is a little hard to look at so first of all, I have the CSV of some Indian takeout food and They have like information here that I'm gonna scroll down to where each row kind of looks like this So it's an order technically each row of corresponds to an order But it's an it's more like a product for every order So you'll see that one order will span multiple rows with you know How many of each item we're getting the price of the items and whatnot, right? And I am just doing some Pandas manipulation right here just to get that data out and up to us and also like creating this extra date field so that It's so that the data is in a particular format that just easy to work with now I'm kind of like modifying date for for you know tips here I'm just kind of modifying this date to look like You know the year followed by the month followed by the date because in sequel you can kind of treat the data as a string and When strings are ordered and if you sort the strings this will actually sort the strings by date. So 2017 04 to 3 will actually come before even 2019 04 to 1 even as a string and not just like in a typical time series date format Just a little tid been letting you know So this is kind of one of the tables that we have in another table that we have in our data set is like an item table and a price So this is definitely the price of an individual item whereas this price I actually do not know what it is and What we could do is kind of use sequel to figure it out So if this is the orders table and this is the products table Let's jump to our first question of can we verify if the orders dot price that is this one over here Represents the item price itself or if it represents the item times the quantity price So there are situations where this quantity is two and the price is given So does this price represent the price of this entire row like two quantities of plain rice for example If it was there or would it just be one no matter what this quantity value is How can we do that by joining this product orders table to this products table? So just take out a piece of paper or just open a top of your notebook and try to write a sequel query for this in the meantime Though enjoy my pointless banter. Have you liked this video? Please like Have you subscribed to my channel? Please subscribe. I Hope that was enough time for you to write down a sequel query and right now We shall go through some results so first of all I'm taking the orders table and doing a join on the products table by Taking that the products item field is equal to the order item field So this is a typical just a typical join condition right here and in that case I also use kind of this as function, which is basically used for aliasing and Product start price is now called product underscore price, which you see appended to this table here And so What we can see is that the orders are actually the same and I think that if I were to even Just refresh all of these columns and just do some sampling Let's try to do some sampling and see what happens here So you can see that the plane the plane bottom is we have four of them here Where the price in the orders table is 80 cents whereas the price here the products is also 80 cents Which means that the order dot price actually represents just the price of a single item That's good information to know when you're kind of digging around your data set because a lot of the time You know you don't have You just don't know there's no description of some of these fields So you might have to like figure it out yourself a little bit But it's good to know very simple query, right? It's still with us great Now let's go to select and where clauses which items cost more than $10 Now I'm gonna try to try to write a query for this I'll give you some time and in the end. Let me just actually give you the let me let me scroll up right here Yeah, so that you can see kind of the table that we're dealing with right now In the meantime, have you joined our discord server because it's poppin It's really cool, and I think it would be an even better place if you joined Yeah, all right, so I hope you all have written that query out and we are now going to check it out So we want distinct items from just the product from just the products table where We want to see if the price is greater than $10, which is probably the easiest way to do it You can also do it from the orders table too But like the products table is definitely more simplified and just easier to work with in this case and So I've just displayed like the top 10 products in terms of pricing and so that is lamb to kabiryani chicken to kabalti lamb to kabalti prawn bati paneer to kabalti chicken to kabiryani Persian lamb biryani prawn balti Persian chicken biryani lamb Persian and the list will go on All right. Now, you know how selects and wears work and also the joins Let's take it up a notch and deal with aggregations of data Where we're going to be doing like the sum of values or the count of values or the max or min of certain groups of values So let's try this question now. How many orders were placed daily in 2019? So in this case, I'm looking for more of a time series chart where every day I just want to aggregate and count the number of orders that occurred. So how do we do this? Don't worry.  Are you ready? I hope you're ready Because right now Let's get into it So we have the select statement again and this time. Well, we have a date that's kind of given in Well, I've already kind of it was technically in a time stamp format with like hours minutes and seconds But I just wanted to convert it into a proper date format and since I'm just casting at his date here And then after that, we are going to count the total number of distinct order numbers that exist Right and we're doing this per group and the grouping here is every date So every date we are going to count the number of orders the distinct number of orders that occur Where of course the date is in 2019 only So because like we're dealing with these dates as strings now It actually kind of makes sense that we transformed it into this what year hyphen month hyphen day because even in a string format we could compare dates and That's how we would do aggregations here And if we were to plot this out in a chart over here, it would look something like this Looks really beautiful. Looks really neat and tidy And this is I guess kind of like how you would kind of structure your data even in a time series problem For example, if you were dealing with it foreshadowing of what's to come Now apart from this and aggregations, we also use something called case statements which are used for conditionals kind of like if else or if and then Situations so let's say right now I want to add another column where we are going to categorize data based on sales So if every date will compute the sales, right? And if the sale date has over 30 sales, then we'll say it's a high yield day Otherwise if it's between 10 30, it's a medium yield day and other than that if it's lower than 10 It's a low yield day. So can you add an extra column to reflect this categorization with sequel? We'll give you a minute Wow one minute passed so fast. That's so cool. Wow time just flies when you're having sequel fun So first of all, this is also introduces the concept of nested queries where I'm literally copying the query that we took before to get the daily order counts and I'm calling it T. This is essentially an alias in sequel and then from this table T We are going to get the date the total number of orders and then we're gonna also add our Categorization so when the number of orders is greater than 30, it's high if it's less than 10. It's low Otherwise, it's a medium yield day and when you execute this query you get this results where 22 is Definitely between 10 and 30. It's medium 9 less than 10 is low So it works out pretty well. I hope you're keeping up and I hope this is fun Now next is probably one of the most important concepts that is not taught in Traditional education, but we use all the time at in like actual work in the industry and it's common table expressions which is basically a better way to manage your queries and Essentially, like if we took this previous query right here It can be a little hard to see in some of these nested queries and in many situations you might have like nested queries within nested queries which become very difficult to manage and If we wanted to convert this into using common table expressions because it creates a logical component It's easier to understand. We can do that So I would just take this chunk of code over here and then we call it daily orders So it's essentially taking that same chunk of code and let's just call it daily order So it's a nice little logical partition there And then we can treat this as if we treated any other table where we take daily orders Which is over here and then we just write the query as we did before So in my mind, this is a lot easier to read than something like this Especially when the conditions get get a little more tricky as they would in in the future Right Okay, cool. Now, let's go on to an advanced concept, which is windowing functions So let's say that we have this particular question right now What are the top three most expensive orders? Every single day. So every day, I just want to get the top three orders in terms of pricing, right? How do we do that? A little hint here is that you can break the problem down into three steps Where first you would get like the total price of all orders every single day And then you would start to rank those prices and then you would only filter for the top three. So We're gonna walk through the step-by-step situation, but after you take some time to think about it All right, hope we're done here. So let's now talk about this problem The first of all is get the total price of all orders every day Which is kind of what we have done But in this case like for every single order, we are going in every single order on every single date We are going to determine the total price So we're using an aggregation right here and we're grouping by of course like the first two two columns here So what this will lead to is well, you have the date with an order number and the total price of all items in that order Now what you want to do is rank those orders from most expensive to least expensive And essentially you can do that with well The this is like the main thing that we're adding here. It's it's this is a window This is a window aggregation function where basically we can just start ordering We can order these rows or results just in any way we want. So in this case, I'm partitioning by date So for every single date, I am going to be creating an ordered list Where the total price or the height where the higher price is on top and lower one is on bottom And that's kind of exactly what we're doing here if we just look here So it's kind of sorted with the rankings in just that order and it restarts for the second date Right so cool We got a ranking going on now for every order every day and now we just need to filter for the top three And that's exactly what we do in this third chunk where we're just saying where ranking is less than or equal to three and And everything else kind of remains the same and so you'll get a list that kind of looks like this where Every single date will have three records and the record will be correspond to the most expensive food items on most expensive orders on that day the top three at least and So I hope this kind of helps you understand like intuitively now. We've we've builds like how Joins aggregations case statements and windowing functions can all come together to make reasonably complex queries with pretty cool decisions Now we're gonna jump into a couple of applications of like how this could be used in machine learning so Specifically like I have two previous videos One of them is on time series forecasting as a regression problem And then also fraud detection as a classification problem and we can look at like how sequel works with both of them So this time series forecasting is actually a part of a previous video That I kind of I made a video on so if you want to check that on you're interested in time series forecasting be sure to check it out in this case I I have like I created the same kind of time series data set because it's it is using the same data set So I created the same time series forecasting over here But I wanted to model this as a regression problem and this is the kind of rows This is kind of what I wanted here So I created two features which is like what is the total orders that happened the last week as well as the last month and Try to determine the total number of orders that will happen in the next week, right? And for that I created this data set So you have features you have labels and it's created in a data set called queries slash Query slash base dot sequel so we can actually go to that And when we do you'll see all of the concepts that we just talked about are kind of embedded in here So we have CTE's over here. We have a CT to get distinct orders then total number of orders every week We have that now this by the way is a strange all of this is kind of like a strange query You can say accent or dialect, which is known as sequelite There are many different variations of sequel that you could try You know programming the stuff in for example, like my sequel postgres Sequelite snowflake, there's a ton out there. Just choose one whatever you're comfortable with and just roll with that Right now here is to get like the first feature for seven days Recent orders in the last 30 days and then I'm getting total number of orders in the next coming week Right and I'm using a bunch of joins over here to Accommodate for all of them and just join them together and create the data set that I showed you before So essentially we're just using all of the same fundamental concepts that I just showed you right now in Order to you know use this for machine learning so very useful at work And it's actually not much that you would need to know Now the next one is fraud detection for classification problems. So let's just go through that real quick right now So in this case, let's go fighting fraud that I pine bee And in this case, I modeled fraud as a classification problem where We have essentially a label which would be like hey does this is this a fraudulent transaction Zero or one. It's a binary label And then we have a slew of like all these all these features that we're using and putting into our model, right? Now, how did we create this? Well, let's go to the query slash base dot sequel file As as we always do Let's go here And voila, we have a very similar format that we saw in the time series the time series query as well Where we first start with well just getting the basic stuff for the transaction that we need Then this is a CTE for just getting past orders This is just the same thing for like looking at the past how many fraudulent orders are made in the past by this particular user And then we do the same thing for merchants. I'm creating two CTEs for past merchant sales and past Merchant sales that happen to be fraudulent and then we just join all those CTEs together in this way, right? And essentially we're using the exact same concepts including here We're also using case ones, but I guess what's extra here is this cast and coalesce So casting is used to basically turn You know from one data type to another in this case I wanted to make sure that the location the category all of these fields are strings or variable character sequences and then coalesce It's a it's a fun a fun little function that says hey if this thing is null like number of past transactions null Then that means by default set it to true Right so overall my entire point of like even showing this right now was that we could use the fundamental SQL ideas that I just talked about in the beginning of this video for even more Complex, you know creating machine learning data sets for training altogether And that's pretty much all the sequel. You'll this is a good amount of the sequel that you'll really need Now I have gone into detail for both the time series and the fraud detection in separate videos That I encourage you to really check out if you want to know more about these queries I've explained it in detail Also, I'm planning to see if I could make a course that's more Application-oriented where we'll even do a deep dive in certain questions and try to tackle sequel from a very very Practical standpoint if you're interested in that do let me know in the comments down below and that's all I have for now So, please do join. Please do first like this video. Please subscribe if you like content like this We also have a discord server going on like I mentioned before so I would love for you to join them Let's have some happy discussions over there and that's all I have for you now So thank you all so much, and I will see you all in the next one Bye


And then the other side is on the profit and loss. Let's go into there and then run this one. So let's change the dates up top to make it to 23 and 12, 31, 2, 3 and run it. And then sales got reversed by the full 1000 here. So if I span that across the two timeframes from 22 to 23, we reversed it.


Hi everyone. My name is Mikhail. I'm a data scientist at Scraping Hub. We develop smart web crawlers and I want to talk about how to inspect your machine learning modules. So, how many of you used scikit-learn? Could you please raise your hand? Yeah, nice. I will be asking you questions to make sure you can understand my heavy Russian accent, okay? So, a typical machine learning project. You collect some training data, then you extract features from it, then you put it all to your favorite classifier and then you put it to production, right? What's missing? There is a lot of things missing, but what's missing? Please shout. Sorry? What else is missing? Yeah, evaluation, right? Cross-validation. So, yeah, there is a lot of things missing, but these are important. So, let's say we added evaluation. We computed accuracy score. We performed cross-validation. We got 90% accuracy of one score, but what does it mean? Is it good quality or is it bad quality? Maybe the good quality for this model would be 99%. And we got 90% only because we have bugs in our software. Or maybe that's actually a great result and we can be happy and deploy it to production and go party. But so, there is a lot of questions, like real-world questions, which can be solved just by looking at evaluation scores. We may want to know if our model is reliable or not. Like maybe it fails spectacularly on some examples, but we can see it in our evaluation. So, how to solve these problems? Does anyone know how to do this? I also don't know how to do this. I don't have an answer, so there is no silly about it, but if you inspect your models, you can get an additional tool, which can help you that. So, I came to the conference a bit earlier, so I had time to walk by the show. There are a lot of restaurants. And each restaurant, there is a pizza in each restaurant. So, who tried pizza in Rimini? Please raise your hands. Yeah, and who haven't tried it yet? Yeah, so there are people. Next slide is for you and for you. I visited these restaurants and I created a formula, how to compute pizza price. It's a linear regression formula, so you can see that pizza price depends on pizza weight, on distance to see, on ingredients, and if there is an old man playing the guitar, then it's very important for pizza. So, the question is, what can you see from the coefficient of this linear regression? So, please. Well, you can see, for example, that distance affects price negatively, so the farther you are from the sea, the less pizza costs, right? Or you can see that maybe old man playing the guitar is more important than mushrooms, right? And, well, yeah, but what are pitfalls? Yeah, this formula is awful. I don't say it's perfect formula. Sorry about that. Please don't use it if you want some pizza. Yeah, this formula is awful, but the greatest price gets negative. So, yeah, one more problem is that you can't compare coefficients directly, because, like, distance from sea doesn't have the same scale as, like, old man playing the guitar. So, if you just check the coefficients, you may think that weight is important, but mushrooms are not important at all, but this is not the case, because scales are different. So, as we can see, checking model parameters is helpful. We can get some understanding of what's going on, but we must know what we are looking at. And there are other pitfalls. For example, let's say we live in an alternative universe where each pizza always get meat and mushrooms together. So, they always come together, or almost always come together. In this case, weights of meat and mushrooms, we can choose them, we can choose any weights as soon as they sum to the same value. So, we may have a weight minus million for meat, and plus million and two for mushrooms, and the formula will give the same result, because meat and mushroom always come together. So, it depends on the training method of which coefficient we will get in the formula if they are correlated. And you need to be aware of this. In practice, it's not usually a problem, because you usually use regularization. But, for example, if you use L2 regularization, then coefficients will be about the same. And if you use L1 regularization, then one coefficient can get to zero, and another one will be twice as large. And so, if you assort coefficients by the absolute value, you need to be aware of methods used for training of this linear regression. Okay, so, some people here used scikit-learn. And in scikit-learn, like in many other libraries, there is a way to look at coefficients of the model, right? So, this is how to do it in scikit-learn. But this code isn't correct. Who knows what's not correct here? Well, it's not entirely incorrect, but it won't give you the whole formula, because there is one extra coefficient, it's named intercept, which we don't see here. So, we created a library called LI5, it means explained like M5. It started from a snippet similar to the snippet on previous slide. It knows where to get these coefficients from various machine learning modules. It supports five popular machine learning packages, more than 70 objects, and it has some features which allow to explain models and their predictions of arbitrary black box classifiers. But in the simplest case, it started from this stuff. By the way, this table doesn't make any sense, because scales of features in Boston data set is not the same. So, the library is open source, you can use it, you can join us, contribute to it, and raise a lot of issues if something doesn't work. So, it supports scikit-learn, XGBoost, LightGBM, there are a lot of less popular packages, and it has an implementation of line algorithm which allows to explain black box modules. So, let's go, so now I will give you like a more real-world example of how can we use it. How many of you follow it's text processing tutorial in scikit-learn docs? Are there people who have done this? Yeah, there are people, yeah, nice. So, scikit-learn has great docs, it has great tutorials, I learned a lot from them, and there is a tutorial on text classification. There are messages from forums and based on text features, the task is to classify them. Data sets named 20 news groups, but here we are using four categories. So, the final model in this tutorial is TFIDF features and SVM classifier. So, who knows what is SVM classifier? Yeah, there are people. So, it's also the same with a linear kernel, it's also a linear model, it's similar to our pizza formula, but then to check if class is positive or negative, you just compare score to zero. If it's greater than zero, then answer is yes, it's less than zero, answer is no. And who knows what's TFIDF? Yeah, there is a lot of people. There is a book by Christopher Manning, informational trivial book, it has a chapter about TFIDF, and there is a page which shows 60 different ways to compute it, like 60 different formulas. And scikit-learn is not one of them. So, every machine learning library uses its own formula for TFIDF, no idea why. So, this approach to classifier text is back-of-words approach. So, you have a weight for each word, and then you check if the word is in document and multiply this weight by TFIDF score of the word. If we inspect our model trained, can at least someone see what's on the slide? No, so I can read. So, these are top features learned by the model for different classes. For example, for computer graphics, we have words like graphics, image, software, images, 3D file. So, if some of one of these words appears in the document, it's more likely to be a document about computer graphics. It makes total sense. But if you look at, for example, at atheism class, you can see that while there are words which are related to atheism, like atheism, Islamic Islam, atheists, morality, there are some words which don't make any sense. Two of the three top words are Kate and Matthew. So, do we have any Matthew in this room? No? So, if some document mentions a guy named Matthew, then document is for some reason about atheism. This is what the model learned. And there are similar issues with documents about medicine. So, some guy named Pete, if he's mentioned in a document, document is about medicine. And the most negative word for Christianity is NNTP. So, something is going on here, right? It doesn't feel right. So, we can check our documents and find some of them which have what met you. And we can see that documents are messages. And we are using them as is with from header, with all the email addresses, etc. And model found an easier way to classify messages. So, instead of remembering, instead of figuring out how to classify them by content, it just remembered some authors and their names, parts of their emails. And so, what it thinks is like, oh, I see, this is my old friend Matthew, he only writes about atheism. So, this is document about atheism. It doesn't matter what he say, it's document about atheism. So, while... So, it depends on the task. Maybe this is what we wanted from the model. But maybe we wanted to classify messages using their content, like what's the message about. Of course, the model learns something about message. But top features, at least here, they are highlighted using LFA library. So, top features are these emails, etc. So, what does it mean? It means that if we don't look at the predictions and the coefficients, we may start to try different classifiers to hyperparameters. And in the end, our model best model will be just the model who can use this information better. But if we look at it, we may realize that maybe this is... there is a problem in our data. Cycadome provides a way to remove headers, footers, and quotes from these messages for this particular data set. So, we can do this, retrain the model, and we can see that the accuracy drops a lot. So, in the previous model, accuracy was more than 90 percent, and now it's 0.798. Oh, it's 6. So, why does it happen? Who can answer? Yeah, so previously, the model was overfitting, and for example, there can be Matthew, both in training and test parts of data. And so, evaluation doesn't show us that... this is not... doesn't show us that. And by... but also, we have to remove some useful information. So, like... okay, so let's try to improve quality of this model. We can see that there are words which are not related to a particular topic. Like, do you have, you are, don't, for this, what... these are like background words, and they are not... well, yeah, they are not related to a particular topic. It's... it is not seen, I think, on the screen, but what are is negative, and what you is positive. And it doesn't make any sense. Why... why... why? So, likely, these are just some background words, and they appear in documents about all topics. And the model just decided to use these words as... to learn preferences for some classes. And... well, sometimes, because of our dataset does not latch, these words don't appear equally in documents. So, some may get positive weights, some may get negative weights. The common way to fight this is to use... is to remove these words, to make task file classifier easier. They are called stop words, and second one provides a way to remove it. You can see we can pass stop words equals English to TFIDF vectorizer. And, indeed, so now these words are not highlighted, and quality improves. So, by looking at explanations, we can try to figure out how to process our data better. We can try to find new features. But if you look at it carefully, you can see that word don't is not removed. Why? Why is don't still highlighted? We are passing stop words English. Yeah, exactly. So, actually, this is a bug in scikit-learn that the default tokenizer splits on contractions, but stop word list doesn't account for that. So, by looking at explanations, we may find bugs. Okay, so we can add extra stop words, which were not in stop word list, and we have quality improved. So, the lesson is that pipeline worked even with bugs. And it's the challenge in machine learning because modules can adjust for your bugs, but if you fix them, quality may improve. There are other ways to process text. Like, instead of words, we may use character n-grams. It means we are using sliding windows of size in here, three and four and five, and just use these three, four and five letters as features. So, if five knows how to visualize them, you can see that quality is worse, and you can see that words are not highlighted in full, but some parts of words are now more important, some are less important. Quality is not good again. We can try the same approach, right? We can try to remove stop words. What would be the quality? Will it improve or decrease or stay the same? Who knows? What do you think? Who thinks the quality will improve? Please raise your hand. And who thinks quality will decrease? And who doesn't know what will happen? Yeah, and who thinks that quality will stay the same? Yeah, so someone thinks quality will stay the same, and it will indeed stay the same. Why? This happens because it's documented in scikit-learn that stop words have no effect if analyzer is not word. This is very easy to miss. I made this mistake myself. So, by looking at explanations, you can find bugs in your own code. So, by inspecting the module, by checking what's going on, we were able to find issue with data, issue with tax specification, we found a bug in scikit-learn, we prevented a bug on our own source code, and we got a better understanding of the processing pipeline, and we made our relations much worse. Well, I'm not saying this to show that scikit-learn tutorial is bad. It's very good, but I've seen these problems in every single machine learning project I've worked on. So, any additional tool which can help you to check data to debug what's going on, I think it's helpful. So, there are two ways, two ways, as you can see, two main ways to look inside models. You can see the model as a whole, you can inspect its ways, or you can explain concrete predictions of the model. So, like here, we are checking concrete prediction, and here we are looking at model as a whole. Both ways are useful, and so, it seems I spent too much time talking about pizza, so I won't have time to discuss all my slides, but, well, so far I was talking only about linear models, but of course there are methods to inspect other models as well. Like for decision trees and tree ensembles, you can use fishing potences, and there is a way to inspect predictions, to explain predictions of these models. So, here is the link. LFI's library has an implementation for XGBoost, for YGBm, for many of scikit-learn, gradient boosting methods, red and forest. Yeah, there are also ways to inspect more black box models. They're probably the simplest method. It's called mid-decrease inaccuracy. I first read it in a paper by Leon Briman. It was paper where Random Forest was introduced, so it's like an old method, and there is no reference to this method. If someone knows it, please give me a link. So, the idea is simple. You train a model, then you get predictions on the test part of data, and then you want to know how features affect the result. So, you can remove some features and train the model again, but training is slow. So, there is a walk around. You remove these features only in test data set, but you can't just remove features from test data set because the model uses these features. So, you may replace it with random values, but you can't just take random values because they may have different scale than in the data set, different distribution. And so, the walk around is that you just shuffle values for this feature and get random value from some other example. And then you run your model without training on this data set with features shuffled, and check how much these effects score. And you can do this for pairs of features, and so check well, which features are important for any model. We don't have implementation this in LFI library, but we should and we will add soon. There is also a way to debug predictions of black box models. So, the main idea is that we have a black box model. We don't know how to look inside it.  So, we don't train it to get correct predictions. We train it to get the same predictions as our black box model. And then instead of inspecting black box, which we can do, we inspect this white box model. Do you think this works? Well, better than nothing, but I think this doesn't work because, well, if you have inspectable model which can approximate black box model, then why you don't use this inspectable model? So, there is algorithm called LIME, and the main idea is to do the same, but not globally, not on the whole data set. You approximate your predictions only small neighborhood around a single example. And this method works. It got, I think, pretty popular recently. There are some issues with this method, like what's neighborhood? Neighborhood means that we need examples similar to a given example. We can get it from our data set, but like there won't be enough examples. So, we want to generate these examples. And also, what does it mean similar? So, we need a distance function between examples. And we also need to define neighborhood. So, if we, it should be some size of the neighborhood, and we must choose the size properly. So, to generate fake examples, we can, for text data, we can remove some parts of text. For image data, we can paint some parts of image with gray or with image min. And for arbitrary data, we can estimate its distribution and then sample from this distribution. So, here we have a trade-off. We don't longer care about the black box model, but we care about data. So, instead of writing code for each black box model, we write code for each data type, or maybe even for each data set. And there are challenges, like white box model, should be powerful enough to explain, at least in small neighborhood. We must choose neighborhood size properly. And if generated examples are not diverse enough, then a line may lie to us. It may give us incorrect explanation or incomplete explanation. And this error is very hard to detect. Because if you have chosen incorrect neighborhood size, you may check how well your white box model approximates black box model. And if score is low, then you see that something is going on. But if your examples are not good enough, then your simple model may approximate your black box model very good on these examples. But it doesn't mean that the explanation is correct. So, I'll skip this. There is a popular and high-quality implementation from LIME authors. It's a separate package. It has support for images, for aggression task. And we also have an implementation for LIME because some details are different and because it fits our library very well. We have a lot of inspectable models and we can use any of them with LIME without re-implementing anything. And so, we have export to JSON. We can show it in a Python notebook. We can export them to data frames. And we don't have to rewrite all this code for each model. And so, LIME can use everything. So, it's just a better code reuse. And we have unified API because of this. We have some cool features which you can read in the documentation. What I was explaining today is like a very ground to earth. I mean, down to earth.  There is research going on about models which incorporate explanations in themselves. For example, in deep learning, you often may use attention mechanism. And there are new ways to visualize models, especially for images. And there is a DARPA program called Expandable Artificial Intelligence going on. So, expect a lot of new research coming for this topic. So, the conclusion is that probably you should inspect your models if you can, but you should know what you are looking at because explanation may lie to you. And LA5 library may help, but it may not help, but you can help LA5. So, please join us. Thank you. Questions?


today to tell you about my favorite. Before I get started, I wanted to find a couple terms. I'm going to be using the word character a lot. There's a very technical definition of a character in a Unicode technical report that I've linked, but for now, I just want to keep things simple. When I say character, I'll be referring to a sequence of bytes which represents part of a letter or symbol. And I'll be using the terms character and code point more or less interchangeably. And when I say glyph, I'll be referring to the visual representation of the character. The character isn't actually visible on this slide, but its effects are. The zero with Drainer Unicode 200D is an invisible combining character used to combine two or more Unicode characters to one glyph that is displayed in its place. So first, I'm going to tell you really quickly what Unicode is and why we have zero with Drainer's at all. I think that this history is interesting and I think that we can learn a lot from the way things are today by thinking about the past. We didn't always have zero with Drainer's because we didn't always have computers. With typewriters, if you want to combine two characters, you type a character, you hit Backspace, and then you type the second character on top of the first one. If you want to write in two different scripts, you have to switch out the ball inside the typewriter that has the characters engraved on it, and I have a bunch of those in this picture from the New York Historical Society. So ASCII was the first standard for storing character information on a computer. It was an early standard. It didn't have a zero with Drainer either, though. It had English characters. So people would work around this in other languages by writing letters, not putting accents on them, kind of hoping for the best. If there's one or two of these, then it's pretty understandable. And this example is in Italian, and so you can kind of muddle your way through it. It's frustrating. So then there were a couple more code pages, which is sets of characters, and you could, IBM mainframes used EBSDIC, which is a code page other than ASCII, and had different numbers for different characters. And you could write in maybe Italian with the accents, and when you printed it out and showed it to someone else, it would have accents. But I say printed because you couldn't really, like, put a file on your, from one computer to another and have it still necessarily work. You had to convert between the computers because there wasn't one universal standard. So this program that converts between different code pages is still on some computers today. In Linux, if you type icon dash dash list, it will give you a list of everything it can convert between. You'll see how unfortunate you might have been if you had received a file and wanted to read it. So that's not true anymore. What changed? In 1991, the Unicode Consortium was created, and this quote pretty much says it all. With Unicode, the information technologies industry has replaced proliferating character sets with data stability, global interoperability and data intro change, simplified software, and reduced development costs. So the Unicode standard describes different code planes which can be used at the same time. So when you had your one character ball and you needed to switch it out, that was like changing between ASCII and EBSDIC, but now you just have a really big one. You can type whatever you want, and you can use it at the same time. A little more technically, if you have a sequence of bytes representing characters with Unicode, you can start where you are and keep parsing from there, and eventually you will reach a character that is comprehensible, hopefully. And you don't need to go backwards to find out where the characters start, where you want to have a complete character necessarily, but if you go in some systems, such as Shift.js, you have to see where the shift starts to see what a character, how long a character is. With Unicode, it's chunks that are regular. And how do we handle overlapping characters, which was really easy on a typewriter and pretty much impossible for a little while after that. We use the zero with joiner. So it's my understanding that Arabic letters are typically joined when they form words. I first learned about this from Ramsey Nasser, who I am honored to share a stage with. And so I believe he runs a Tumblr blog devoted to pointing out places which don't link letters that should be linked. A zero with joiner joins letters, so you can use it to make sure Arabic letters are written in a joined form. They should have properties specified that join them together when they're in words anyway, but you can force them into a joined form, even if they're not joined next to one of those joining letters, such as in this slide. You can see, sorry for those of you who can't see it, you have zero with joiners and a character that's in one form, and then at the other side of the equal sign, it's in another form. And Arabic is typically read right to left, but this slide, you can read it left to right, because I speak English. So to better understand the zero with joiner, I like to think of how it's different than other similar characters. The most similar character that comes to mind is the zero with non-joiner. So a zero with joiner joins two separate characters into one glyph. A zero with non-joiner, you have two separate characters and they are non-joined. It's like it's evil twin. So this example is in Canada. You can see the characters on the top. You put them right next to each other without doing anything else and they combine into one glyph. On the bottom, they are still next to each other, but not combined. You may have noticed that my first slide had three examples and I've only gone through two so far. I don't speak Arabic and I don't speak Canada, but I do speak another language, emoji. So the Unicode consortium names this glyph as couple with heart, woman, woman. The emoji typically are couple and family emoji using zero with joiner. One of the reasons I like the zero with joiner is that it destroys any assumption you might have about how long a string is. This is really apparent in Python. This is Python 2. Someone in the audience, how long do you think this string is? I heard eight. I see someone holding up three fingers. That's three. Someone guessed 40. Those are all pretty good answers. I saw someone doing five. Okay. I'm going to tell you 20. Okay. So this is because the default implementation in Python 2 for strings is a byte string. There are 20 hex escapes which comprise this string under the hood. So Python also has the concept of a Unicode string, which is actually the default type of string in Python 3 onwards. So let's try this again. Someone, how long is this string? Okay. A lot of people seem to think it's three, which is a good... Okay. I hear five, two. Those are good. That's pretty close. The correct answer is six. There are six Unicode characters in this string. There's woman 1, zero with joiner 1, heart, a variant selector for the heart which says that we want the emoji display of it. There's zero with joiner 2 and finally woman 2. So I have one more puzzle though. Perhaps like me, you program in languages which run on the Java virtual machine. The Java virtual machine, you might or might not be blissfully unaware, but it stores strings under the hood as UTF 16, which is a standard that Unicode puts together. So knowing what we know now, how long, what's this going to return? This is the length of a string in Java. 12 is not what I was expecting someone to say, but it's a guess. The correct answer is eight. So this is a trick question. The woman emoji, which is one of the six characters in this, doesn't fit in a UTF 16 character. So it uses two, a high surrogate and a low surrogate, which combine to form a surrogate pair to represent it. Since there are two women in this, each of them takes up two and it returns eight. There's a way to check this. You want string.codepointcount instead of string.length if you want the number of Unicode characters. So I want to go on a quick digression. The zero with joiner isn't the only way an emoji can change that breaks your assumptions. You may have seen the skin tone emoji. You can now have an emoji and specify the color of the skin of the person in it. That does not use the zero with joiner, but that doesn't mean that the length of the code points in it is what you would expect it to be either. So you have one character and the next would a second character, which is the skin tone, I think it's called the Fitzpatrick modifier, which is for light skin to dark skin. And this is much like the heart in my example, how to variant selector. This will tell you what variant you want. And this uses variants because it's a separate, it's part of the same thing. It's not a new concept that needs a new emoji, but that's pretty much a judgment called by the Unicode consortium. So there's one more wrinkle I have for you before you can display emoji. The Unicode standard was introduced last year, Unicode version 8 was introduced last year, and it uses zero with joiners for couple and family emoji. Not all devices support Unicode 8. This is my smartwatch. I'm wearing it right now. So I sent the text to myself with the emoji sequence for the couple with heart woman woman, and this is what it showed me. As you can see, there is a box which represents a woman, and then a heart, and then a box which represents another woman. So this, the reason it doesn't display like complete garbage is because Unicode defined a sequential fallback for when you see the couple with heart sequence, and it is person, art, person, all next to each other, not combined. And this is to prevent breaking backwards compatibility. And my question is, do all of you, when you're writing apps, do you need to be that careful to make sure things have a fallback and that they still work? So in conclusion, when you're writing code for something that takes user input, think carefully about how much space you're allotting for its display. And also in conclusion, we live in an amazing diverse world, and I hope that we can build apps for everyone in this world, not just people that speak English. Thank you.


So it's my pleasure to introduce our next sponsor welcome message. We have Austin McLean from ProQuest who they have been friends with us for so long just wonderful supporters. Austin I'm going to give you the video. Great thanks so much John and everybody it's I'm Austin McLean from ProQuest as John said and we're very pleased to be able to sponsor the USETDA conference this year since being involved with USETDA since its founding. We really want to thank the conference committee for holding the conference this year you know in these unprecedented times being able to connect virtually with the ETD community really helps and it helps many of us continue our work from home or wherever we happen to be at this time. So a big thank you to John and the conference committee team. We want to make sure that at ProQuest as many of you contribute your ETDs to ProQuest to make sure that the community knows that we're here to help you with any ETD submission challenges that you have that maybe have cropped up due to COVID. We have a team of customer service and technical support experts who would be happy to assist you with any ETD submission needs so you can feel free to reach out to any of us at ProQuest that you've communicated before or send us an email and we'll be glad to assist you with your ETD submissions. But we want to thank all of our library friends today for those of you that have the PQDT global database which contains those five million dissertations and 2.6 million in full text. We want to commend your efforts because we've seen great usage of the PQDT database during this COVID time so you've done really good work we can see by connecting the database to the users on your campus and remote users that are connecting from home so clearly the librarians in the audience have done a great job in connecting the researchers with important databases like the PQDT global database. So we want to thank you very much for your efforts and I'll put in a quick plug for those of you that are using the ETD administrator system this afternoon at 5.35 we'll be hosting a user group so we look forward to seeing many of you there. So thank you all very much and we're happy to sponsor and the conference is already fascinating and we look forward to learning more new things around ETDs today. So thanks very much. Great thank you so much Austin.