Get this and get it straight crime is a suckers road and those who travel it wind up in the gut of the prison of the grave There's no other end, but they never learned this time It was going to be a vacation in the wide open spaces But a black stallion a tiny emerald and a battered horseshoe met a 24-hour delay It could have been worse because to the dude from Manhattan. They meant death From the pen of Raymond Chandler outstanding author of mystery comes his most famous character and crimes most deadly enemy as we present The adventures of Philip Marlowe With Gerald Moore starred as Philip Marlowe.  She just bought me a rattlesnake mountain. I snapped at the chance Inside an hour. I was rolling down the highway towards San Bernardino and 120 miles later at five o'clock I turned in under a big arch of gnarled cedar that spelled out rainbow ranch But the layout beyond was about as primitive as a dry martini a Ranch house the size of Union station was backed up by blue tile swimming pool paved tennis court and a semi-circle of bungalows with all the rustic charm of a Hollywood motel I drove on in slowly as a broad brimmed hat red gabardine shirt Hickok belt and hand-tooled boots Bounced out the door and ran toward me. It was my host the ex-hotelman Harold our loss How are you boy? I am sure glad you can make it find out now show you around. Hey, what is all this Harold? From your phone call. I expected a shack with oil lamps or wood stove and at least a few head of cattle Well, you mean I didn't tell you why this is a guest ranch Phil guest ranch the best in the way Oh, brother. Oh, and incidentally don't call me Harold. No bad atmosphere for the dudes. The name is Buck now Buck Lawson Buck. Oh, no Spread here Phil real spread 14 big cabins string of 30 horses stables down there Oh Howdy folks sure is Mr. Mrs. Doverman. He's a big fan of storage man in LA. No as it was saying Who's coming red rider? Not funny. They're not funny. Look. It's thunder that black devil. He's loose again That horse will kick the fence down of those fools don't hold him. Hey, hey, that's some animal. He's a beauty Yes, and a renegade a skittish temperamental Bronco with anybody but Virgil Sawyer. Yeah, oh, they got a rope on him now That'll hold him. I'm not for long Sawyer's the only hand I've got him get close to that stallion. He's leaving tomorrow blast it How come well frankly, that's what I wanted to talk to you about. Yeah, I wait a minute I came up here for a rest not a job. I know I know you'll get it Phil you'll get it But since you're here, I figure you could sort of keep your eyes open for me. Lawson. It's a dirty trick No, no, please. I'm expecting trouble and bluntly. I can't afford it Every cent I've got is tied up in this ranch a serious scandal could ruin me and you're just the one who can keep that So okay, okay, so it's the old hotel business on horseback. How does this Sawyer mean trouble? Well There's a couple here from the east the Mortons.  How did you know that? Yeah? Well, it's standard like a bee picture plot Well, anyway, they came to blows this morning Maybe Virgil's innocent maybe not but I can't take a chance. So I fired him Ordered him to pack and get off the place by tomorrow. Well, that's that. What are you worried about? Plenty Sawyer's a proud man Well, oh he he was furious. He threatened to get even I'm not sure he means it but if he does Well, that's what we have to look out for the we Now look buck you built me into coming up here and I got a good notion to turn around What's the matter see that couple going into cabin number eight? Yeah, that's the couple I'm talking about the Mortons Paul and Judy cabinet. They don't tell me just let me guess. Yeah. Yeah You're right Phil. You've got number seven Okay. Yeah, sure number seven it is. I'll be seeing you buck The number seven and waited for the boy to show up with my bag and I started on pack but stop when I heard a riot next door At that point sprawling rainbow ranch was just a horizontal tenement nothing more The voices rattled on for a few minutes then dwindled off into a long and golden silence That said maybe a peace treaty had been signed But then a door slammed the number eight so I peeked out. It was Morton and from a look on his face I knew the peace treaty was nothing but an armed truce. I Followed him to the big lodge and into the bar and when he sat down I took the stool next to him Well, what'll it be gentlemen scotch and water. No ice. Yeah, the same with ice. Well, mr. Morton I guess that brands us as dudes, huh? Bourbons the only drink out west. I wouldn't know I'm sure. Oh, it's a fact Hey, that's a handsome ring you got there and the initials are the same as mine Those stones are emeralds, aren't they? That's right. Yeah It's supposed to be four of them one's missing. I see is that an emerald too? It was Happens to be my birth now. Here you are gentlemen. Oh fine. Allow me mr. Morton. There you are. Thank you, sir How'd you lose it stone? I mean I don't know it happened several months ago and in any case It's no concern of yours now if you don't mind. I just assumed he left alone Oh, well, that's too bad here. I was hoping I'd find out all about the silk business a silk What do you mean by that?  Name's Marlowe and Just why you're prying into my personal affairs mr. Marlowe because I got a little free advice for you Cool off before you start the kind of fire. You can't put out, huh? So that's it at Cowboy Sawyer Marlowe now you're getting too personal. I suggest that you mind your own business. I'm sorry Didn't mean to lose my temper that way Good night Yeah, it's bound to be Well, Mrs. Morton, where'd you come from? I was standing over there watching My husband has all the social grace of a tarantula. Well, maybe you should have looked closer before you made the leap Oh, that's the wonderful thing about him. Yeah, you're not up to like Paul much when you first meet him But once you get to know it you hate him. Yeah, I'm not sure that's funny. It's not supposed to be I've been living with him for six months now So jealous, it's unbelievable He wouldn't leave me in New York. Oh, no insisted on dragging me out to this this dust bowl with running water Why a rancher I'll never understand. He doesn't know one end of a horse from another well with his aptitude he'll learn You know, it might be figured you two might get back together if you had a chance to relax in the open Mrs. Morton So he said However, we weren't here ten minutes before he accused me of getting romantic with that leather-faced cowboy That makes sense. I don't know both gentlemen are justified. You're lovely to look at somebody ought to remind my husband His idea of welding a marriage is to spend all his time playing gin with that Doverman Oh Doverman the van and storage character from Los Angeles.  Is that it? Excuse me folks care to order another drink before yes, I would and I'd like it over there alone Megan Manhattan bartender strictly Manhattan and make it double Mr. Marlowe good night No, I'm not so sure It was almost dark when I left the bar and headed down to the bunkhouse where the working personnel of Rainbow Ranch called home The casual clutter of rumpled cot-scattered pulp fiction and dusty boots gave it the only sign of authenticity I'd seen in the entire place But aside from that it was empty and a noise from outside brought me around the building to the back where I ran up against six and a half lean feet of solitary cowboy with his hat shoved back on his head pitching horseshoes He was out of uniform for a flashy dude wrangler, which left him in a faded blue shirt and Levi's that Fitted his lanky legs like a pair of bent stovepipes He spotted me and stood there swinging a battered horseshoe in each hand while I walked up to him Hello, hi a Sawyer little doc for horseshoes, isn't it? Hey, hey, you're good Good at horses, too. I understand.  Yeah, what's the secret? No secret just have to treat him right What's on your mind mister fact that you're leaving tomorrow, I reckon you better keep out of my business now Look Sawyer it takes at least two to make a fight and fights are poisoned to Buck Lawson. So I don't like to see my friends poisoned Now why don't you take it easy, huh?  I don't know who you are mister But I'll tell you this anyway seen as you're so interested I'm leaving here tomorrow. All right, and I'm gonna square up with a couple of folks first before I go I got a raw deal here, and I'm just not the kind to take it laying down What do you mean raw deal?  You ought to know better than to get yourself all involved I'm not much for conversation fella, but I'm going to say something real plain so you'll be sure to sell it By the time I got myself untangled and back on my feet The strong silent fugitive from the old chism trail was gone However, my original theory that it takes two to make a fight was still valid I decided to find Paul Morton and spend the rest of the evening close to him His cabin was dark, but I remember the running gin game. He had with a big van and storage man So I went down the line to the Doverman cabin and knocked It was Carrie the perennial dude who galloped up to open the door Howdy stranger come on in and set a spell our latch is always stringing out. Well, I sure do. Thank you, ma'am My name's Marlowe or bill is as mr. Marlowe Howdy hope you'll excuse the looks of the place our box of extra clothes just arrived from town Carrie's been on Packing it sit down there mr. Marlowe. They're mostly old things just roam on the floor Oh, thanks, but really I can't stay I'm looking for Paul Morton. I thought I might find him here Morton say there's a nice chap met him day before yesterday for the first time and Won $90 off him in gene already haven't seen him tonight. Oh or bill was out looking for him himself Just a few minutes ago weren't you dear why yes as a matter of fact it was it didn't locate him Huh? No, I didn't you know he seemed to be all upset this afternoon couldn't keep his mind on the game I thought I'd have a little chat with him to calm him down Over the whizz at that mr. Marlowe. Oh, it's not me Carrie is this country I don't see how a man can keep trouble in his mind on a place like this ranch Marlowe Yeah, it can happen. Believe me poppycock. Why son there's something about this open land around here that cleans Out a man's head and his heart to you sound like a travel art. I mean it a few more days of this and mortal forget There ever was such a thing as a cash register. Yes, sir Give this untamed countryside a chance and it'll cure anything. Yes, well What was that wasn't the call of the world mr. Doverman? Find out about it lossy one of the boys told me hurt thunder racing a terrible fuss come over to check But then it was all over. Give me the Latin Harold. Yeah, yeah, yeah, holy smoke For more than all right. He's been tripled a bit. Oh, it's a ghastly accident And it's all my fault. I knew thunder was dangerous and I didn't get rid of it. All right. Take it easy Take it easy. There's a lot of questions to be answered before anybody takes them Hey, wait a minute. Wait a minute. Look at this here by the game.  Yeah, not like this one. Look at it. It's all batted up. Well, all right What's that supposed to nothing yet It gives me an idea because the last time I saw one of these was being pitched at an iron stake behind the bunkhouse What are you getting at let chances are at least 50-50 that Paul Morton's death was no accident. It was murder In just a moment the second act of Philip Marlowe, but first Perfect musical settings for a Sunday before the fourth will be yours tomorrow afternoon The symphony at the half hour of fine orchestral music and the choral air is a half hour of brilliant vocal music Are regular sunday afternoon features on most of these same cbs network stations Now with our star gerald more we return to the second act of philip marlowe and tonight's story the dude from manhattan Labeled paul morton's death something worse than an accident lawson's mouth fell open and the muscles in his face Jerked as his eyes moved slowly from me Out to the now quiet black stallion in the chorale who somehow or other seemed to sense the death at our feet Then as the trembling man's lips silently formed the word murder he gestured for me to help him carry morton's body out of this table After that he looked at the dead man's broken face once more said he was going to call the sheriff's office and hurryed away A minute later judy morton stepped into the small circle of light that surrounded what only a short time ago had been her husband But except for a thin line of perspiration above her lips. She was no different than when i'd seen her last I just passed buck on my way down here Told me my husband was dead. You tell you anything else judy not how paul died. I mean no It was a stallion wasn't it an accident i doubt it Why marlowe one thing this horseshoe too close to the body But this is a stable and this a horseshoe that's been used exclusively for pitching at a stake in the ground Here look at it And remember cowboy virgil's favorite sport is horseshoes Besides what reason would your husband have for coming down here at this hour in the first place It wasn't too crazy about horses, you know, no what he was about me Let's move a little away from here marlowe cigarette No, thanks I'm not coming apart at the seams because it isn't in me I hated paul hated him with all my heart marlowe I'm down here only because he pleaded with me begged me to talk to him once more to listen to reason about why About the decision I came to less than an hour ago, which was divorce unconditionally I thought you said you came out here to try to patch things up. I did But I also said that we weren't doing a very good job of it Then tonight a little after we left you with the bar marlowe I got my hands on the lever. I needed to prime myself loose from that jealous maniac It was a knowledge marlowe that my late husband was crooked silk business. Yes While he was drinking his dinner. I went to one of his suitcases for an aspirin Found what instead at least three dozen samples of the best silks made without any importers or manufacturers name and underneath that $1,000 in cash I know enough about the silk business to fill in the blanks marlowe All of which comes under the heading black market. Yes I added what I had found to the fact that this dude ranch he had insisted on was close to los angeles Close enough for him to run off and conduct his purchasing while I thought he was communing with nature of playing jinn with that mr. Doverman Then I had him. Mm-hmm. You also had a divorce. No strings attached right exactly Blackmail to get rid of your own husband Pretty, isn't it? Yeah Well, at least with this This accident or whatever it is It's no longer necessary. No How Judy only two things are necessary one the location of virgil soya and the other you and your own cabin Where I can ask you some questions later. Why do you want to ask me questions? Well, I might be making a big mistake, baby But it might be that virgil and you are out for the 200,000 bucks You know, honey that man in the saddle might like money too I'll see you When I started back for the bunkhouse the only place I knew of that might give me a lead on the strong silent horseshoe pitcher I realized the tagging paul morton's death a murder was one thing proving it was going to be quite another And when I was there and the place was empty without even signs of a hasty departure I was sure of it But not by intuition as was the gentleman standing the open doorway watching my every move Orville Doverman champion of the wide open spaces didn't believe that a clean cut cowboy could be guilty of anything more unrefined than Spitting on a putt-bellied stole. Oh, I think you're crazy buck told me about you're finding that horseshoe next to morton's body On the conclusion you jump to from there. You're being very hasty boy. And that's dangerous. And that's the reason I'm here I don't believe in necktie parties necktie boys. Hey, hey hold it Nobody said anything about lynching your hero I want to find soya so that if I'm right we can save the state the time in trouble of a man hunt But since you brought it up vigilante don't scramble for conclusions too quickly yourself I happen to have a little more to go on than the relative position of a horseshoe Not that idle gossip that's going around the same but the moment it figures two ways Virgil's unhappy enough for the status quo to liquidate the city slicker or Virgil and the squall light out after a clean start the hard way choose one Not a small in either case and especially the stupid suggestion that the girl and Virgil soya are in cahoots that I can't believe Well Sentimental reasons I can't either Besides judy morton found out enough about her husband within the last hour to make murder for freedom's sake very unnecessary She learned he was a crook mr. D if you can stand the disillusionment Jazz and shady dealings and silk Judy didn't go into details about it But I gather she found out enough to make him sit up and take notice And that brings us right back to Virgil boots saddle and all And we'll argue the fine points later But right now mr.  Goodbye I Spent the next 20 minutes talking to cowhands guess miscellaneous hired man any and everybody who might have been able to say he went that Away of Virgil soya With no success And to make matters worse when I'd given that up and was on my way back to the lodge to help loss and wait for the sheriff I found myself being paged western style, of course by no one else But mrs. Gary Doverman the capital d in dude ranch. Howdy. Howdy, ma'am Oh, mr. Marlowe. Yeah, mr. Marlowe. Look at this. Look at what I've found. I've struck it rich You might say much like the old rustlers the old rustlers mrs. Doverman stole cattle. Oh, yes Oh, yes, so they did. I guess I meant those panhandlemen, you know gold is where you find it Anyhow, look, it's a precious stone small, but nevertheless precious mind while digging for worms. No doubt Mr. Marlowe, you're teasing me. You know very well that this is a polished stone Funny thing though is where I found it. Shall I tell you? Oh, please please do mrs. Doverman Well, I was just unpacking those clothes that orville has sent up from los angeles Some slacks and things like that and well when I started to hang a pair up this fell out of one of the cuffs and then Now I wonder how a little old emerald like this ever got there. Well, it was probably mice mrs. Do Emerald Let me see that quick. Well, yes, of course, but believe me, mr. Marlowe. It can't be very valuable I'm not what are you talking about murder or a reasonable facsimile there of and a girl named judy morton If I don't hurry goodbye and bless you mrs. Doverman.  But now was the right time As I ran for judy's cabin I didn't know any more about the wise and wherefores of paul morton's death and I had before I made small talk with mrs. Doverman But I did know that unless lady named luck and I were on the same team The rainbow ranch was you for a second corpse When I was close enough to the rough oak door number eight and orville doverman who might ask to stand god was nowhere in sight The full impact of that responsibility sank into where the wingtips on the butterflies in my stomachs was scratching at my hip pocket Until I moved in still closer and There in the light of a single lamp that was halo enough for me. I saw the girl from Manhattan Nervously lighting one cigarette from the end of another but more important very much alive I didn't bother knocking What am I doing here honey?  Oh, that's the way it happened. Yeah, that's the way it Now look judy, baby. You can't know what I mean yet It's doverman honey the gin player with all the moving vans He's the one your husband was buying that black market silk from I didn't know that until a few minutes ago Which was after I told him where you could be found and that you knew an awful lot Oh, it's mr. Milo. He thanks you and warns you're not to move. Yeah, oh see what I mean phil. Yeah, oh sure I see You know, it's funny doverman when I was outside and didn't see you around That did see the judy here was still in good health I figured that either you had decided to sit tight until you knew exactly how much he did know that you Already started to run Yeah, this I didn't count on and this malo should point out what I said earlier about your jumping to conclusions It's dangerous handling hot silk is charles playing that has been for me for 20 years malo for your husband, mrs. Morton it was much more. That's why I had to come to you like this That's why I had to know if his stupidity went so far that even you knew of me You shouldn't have bothered mr. Doverman. I didn't know but you see malo did that leaves me even a correction doverman Paul mortons dead. You're out in front. I didn't kill morton malo and neither did virgil soya I saw it all my friend so I can tell you that the man who killed paul morton was paul morton himself Suicide are you out of your mind? No, not suicide mrs. Morton merely a plan for murder that backfired The intended victim was you his wife. Oh, no keep talking doverman Why malo I'd rather keep you guessing I wouldn't duck baby My shoulder now the man said keep talking. I can't I'm hit you'll be again if you don't so you know Stay out of this malo. Come on doverman. I'm not gonna ask you again.  You're not finished yet. I I figured that he was going to knock his wife out leave soya's horseshoe where will be found then half make it look Like an accident that would fool nobody, huh? What went wrong doverman? Why didn't it work? Oh he He approached thunder From the right side Instead of the left a horse got excited kicked out and caught him that dude now Let go soya sure doverman with play It was a slow but steady two hours of first aid and questions and answers mixed with a san bernadino deputy sheriff who Couldn't quite get over it before orville doverman was on his way to a hospital that featured barred windows Mrs. Doverman the complete innocent was on her way back to los angeles And buck loson julie and i were in the bunkhouse watching virgil soya watch a pot full of water boiled for coffee ranch style Well You know You can't ever tell mono this whole thing might have just the right effect Put the ranch on the map i mean After all it was a genuine 100 cowboy who saved the day for us Oh, no, that's not a ride buck. It was marlowe here. I only followed him Coffee's ready for oh, that's for me. Let's go. What did make you go up there mr. Marlowe Oh a little precious stone virgil an emerald that once fell out of paul morton's initial ring But marlowe that happened a long time ago three four months We're just after paul had returned to new york from los angeles. Yeah and negotiations with doverman You see honey, it was mrs. Doverman really who found the missing emerald tonight And the pair of slacks that orville had sent up here Then that was proof that paul must have been with doverman in los angeles before Yet they claimed to have met for the first time here at the ranch Yeah, that's what they claimed that plus what you told me judy made the man with the moving vans it And uh yo, hey virgil at coffee's hot. Yeah, but it's good Well anyway, since i told doverman where you were and that you knew your husband had been dealing in black market silks He took his cue accordingly. Yes And fortunately you yours Well that makes it two people who tried to kill me tonight my husband and his partner Oh fine Good night gentlemen virgil soya made good coffee and lots of it So another hour went by before we finally broke up and I was outside smoking a cigarette and Strolling toward my cabin in the start of a vacation that already had been postponed too long But halfway there I stopped the sound of raised voices ahead of me A man and a woman were arguing violently And a little away from them on the porch of my cabin watching the battle of the sexes with consternation while he waited for me Was buck loson my host I turned quickly and hurried back to the bunkhouse where I knew virgil soya would put me up for the night Where I knew that early the next morning I could sneak off find a quiet cool stream And fish The coyote high in the hill someplace Said I had the right idea Adventures of philip marlowe bringing you raymond chandler's most famous character and crime's most deadly enemy star jerald moor And are produced and directed by norman mcdonnell Script is by melden ellie robert mitchell and gene levitt Featured in the cast were charlotte lorence bill johnstone bill lally her butterfield dj thompson lou krugman and jack kerrington The special music is written by richard orant. Be sure and be with us again next week when philip marlowe says I went from a mansion in belair to a cheap flat in southgate Looking for a girl with a secret Who a man in a pork pie had a wise cracking secretary and a fat corpse didn't want me to find But who I found anyway because of the quiet number Three highly individual highly entertaining mystery adventure shows stand high among the top shows on cbs every sunday The green llama call the police Sam spade Go adventuring with them every sunday when they come to you over most of the same cbs stations This is roi roan speaking Now stay tuned for gangbusters, which follows immediately over most of the same cbs network stations This is cbs the columbia broadcasting system


All right everyone ready how many of you came from Stevens firehose. I mean talk All right, those of us that has seen Steven present for years know how overwhelming it could be at times So my name is Julia Cartwright. I work at National Instruments. I'm also one of the stable real-time Maintainers as Steven had mentioned the last presentation I've been involved in Linux for about ten years involved in the RT project for About six years I think at this point I Maintained one of the stable release actually two of the stable releases as as Steven mentioned But I've also followed the Linux RT users list. So there's a mailing list feature kernel at org This is a user list where a lot of people talk about the issues that they've run into lots of bug reports happen there And from this information gathered over years It's pretty clear to see that there are different classes of bugs That our users the preempt RT users are running into and reporting That are issues in actually issues in device drivers So the purpose of this presentation is to present these Issues as they exist and I'll talk about some of the features Steven mentioned like local locks and so on as we go along So how many of you are actually running preempt RT in production right now? I would have expected more honestly. So how many of you are like evaluating it as a potential future RTOS platform About the same. Okay. That's interesting. All right. Well, maybe I hope you will find this interesting How many of you work on upstream device drivers within the community? Great. You are kind of my target audience. So if anything I say is unclear you have any questions Please ask So I'm gonna talk about a class of systems that we all run into if you if you're divining Designing an embedded system that has Yeah, oh No, they don't see my slides. I don't know how to Have we not switched? Aha. I'm that one Nothing over here though Okay All right, great. So now we have them. Yeah. Thank you So I'm gonna describe a class of what I'm calling mixed criticality systems So most complex embedded systems might have might run two different types of tasks You're gonna run tasks that have some real-time requirements and latency bounded latency requirements Then you also might run some non-time critical stuff You know doing basic monitoring or disk or maybe not even any IO or you have some throughput tasks, whatever it is Those two need to run together. Maybe need to communicate with one another. It's mixed criticality in the sense that there's two different Degrees of time-sensitiveness So one way you might actually design this is actually split it up into two different hardware devices You might have Like an arm series our core on a board and you deploy all your super time critical stuff there Or maybe you're using like an FPGA or CPLD You're pushing all your super time-sensitive stuff onto a separate hardware device and then on unlike an arm a series Chip, maybe you're you're running Linux a full Linux stack So this is this is one of the models. However, it has some downsides It's expensive per unit cost now you produce a board. You have to have You know two different sets of hardware accomplishing two different things If there is interaction between the mixed criticalities like crossing the the domains Now you have to define some communication channel between the two hardware Pieces that's up to you to define But you do get really good isolation. So the stuff that's happening on the non real-time Linux side isn't influencing your time critical stuff So some might choose this different set of trade-offs if you look at something like Xenomai I don't know if you are all in Jan's talk today. He was talking about Xenomai and it's it's it follows this Hypervisor, I think they call it the co-kernel approach It's effectively like you can you run a separate operating system within the realm of a I'm going to call it a hypervisor here. It doesn't necessarily have to be a hypervisor There's amp solutions where you can partition like okay these these cores are going to run free our toss and these are going to run Linux Again, there's there's trade-offs to be made about usability And communication between these mixed criticality systems Pre-empt RT sort of has chosen a different path here Systems that run preempt RT are sharing the kernel. They're sharing a scheduler. They're sharing a device Stack all the subsystems are being shared So you you don't get as much isolation But you get a lot of usability benefits Applications can just use standard Linux IPC mechanisms to communicate It's effectively like thinking of that The traditional Linux process isolation model is good enough to isolate real-time tasks from not in real-time tasks and But like I said the driver stacks are shared between them and therefore the drivers can misbehave and so I'll talk about ways that they can misbehave But first I want to talk about latency. This is what I'm going to call the delta. This is what we all care about If you care about real-time guarantees you have A time at which an external event occurs. Maybe it's an interrupt firing a timer or some IO device saying hey I need to be serviced the time it takes from that event to occur Until a the relevant real-time task actually executes This is my bound of defining Naming Delta Most of you should already know this if you care about real-time systems Real-time systems will try to characterize and bound this in some meaningful way So this is how this is one of the example techniques techniques that the RT community especially the preempt RT community uses to characterize this delta So this is a super simplified version of a test. We call cyclic test So sick to cyclic test is pretty simple All it does is take a time stamp and it sleeps for a fixed duration say like 10 milliseconds And then takes a time stamp when that thread is woken up And if you take the difference between the those two time stamps I've collected That's the amount of time that the thread actually slept And then I'm expecting to sleep for the duration so I can subtract that out and that difference is that delta And so if you plot that in a histogram it can look something like this So in the purple is actually just a I ran this on my laptop. This is like the regular config config preempt Setting in a mainline kernel. This is what you would see the delete and see samples Look like you can kind of see the distribution has a really long tail I think I only ran this for like a minute and I saw that 342 microsecond spike. It's not a real-time operating system. So, you know, that's to be expected And the green is on the same hardware system This is the preempt RT patch you can see that it has and that same duration I don't remember how long I ran it You can see that it has a max bound of 16 microseconds much more well behaved for real-time use cases So if you think about this delta and what is Contributing to this delta I break it down into two different phases. So there's a IRQ dispatch latency This is the amount of time it takes between the hardware event actually firing to the interrupt for the relevant thread to actually be woken up to for the the interrupt dispatch to have occurred and the threat the scheduler the thread scheduler being told This thread needs to run that latency. I'm calling this lowercase delta IRQ dispatch And then the second one is scheduling latency. This is the amount of time It takes now that the scheduler has been made aware that this high-priority task needs to be run the time It takes from that moment to the CPU being given That task to actually execute it being scheduled in on that CPU So these are the two phases that I'm defining for the purpose of this Conversation and I'm going to dive into each one of them individually. So I'm going to start looking at IRQ dispatch latency specifically and sort of talking about all the causes of this Latency and issues you might might see So this is mainline model. So what one of the main contributors to this is actually long-running interrupts and mainline so and mainline because Interrupt handlers are executed in hard IRQ context They are implicitly executed with interrupts disabled And so here's this diagram the scheduling diagram like you might have seen before. This is all mainline again X-axis of time we have some thread executing in user mode at some point There's some interrupt that fires say for example, it's like network and you don't even care about network in your real-time application But it happened interrupts happen And then at some later time well at the time that that event occurs The CPU is going to vector off into hard interrupt context and start executing the handler associated with that network device or whatever device it is And say during the duration of that interrupt handler a High-priority event that I care about I'm here and calling this external event that I care about more say that it fires Well, it's not actually able to be scheduled on the CPU until that low-priority interrupt Is done executing so if you if you in any this this delta here between this external event firing and that red task Executing that's a direct contributor to The that IRQ dispatch latency as I mentioned And it should be obvious without the real-time patch if you want to try to define bounds on this Well, it would be a mess Because the bounds would be the bound of the longest running Interrupt handler in the system the longest running interrupt handler is going to be informing what your worst case IRQ Dispatch latency will be hopefully that's made clear And then at the point your task or you the interrupt handler execute and wake up your real-time task and everything's good This is mainline. So on On RT when you when you apply the RT patch and actually enable the RT patch And and Steven mentioned this as well. What we do is actually force IRQ threads So what this means is that there's actually very little code that we actually execute in hard interrupt context very little They're just little shims that execute because they have to and all those little shims do are wake up these threads that are that are Going to be executing your handler If that makes sense. So you might have some low priority task running You have some not unimportant IRQ that fires But only a very small portion of time has actually spent just waking up the associated handler thread And the duration of that shim can be well understood the bounds of it can be well understood for real-time purposes And it provides other guarantees Such that because it's now running in executing in a thread those threads can now be preempted So if I then have my higher priority RT critical interrupt firing That IRQ thread can be scheduled immediately So that actually reduces that IRQ dispatch latency. Does that make sense or provides a more Easily characterized bound So that's how that works now for you as a device driver developer I have some good news. This is just gonna happen Like we are going to be When you when you enable RT kernels gonna forced IRQ threads So Steven alluded to this thread IRQs actually exists in mainline right now You can boot a kernel and pass the thread IRQs parameter and it will thread all your interrupts So this a significant portion of this infrastructure is in place the things that aren't in place for RT is a forced enablement So that's good. It's good except when it's not So for and there are cases where it's not okay It's not okay when Your drivers are invoked as part of interrupt dispatch So if you actually are invoked in the process of delivering an interrupt, then you can't be threaded So the things that fall into this category are IRQ chip implementations Those are important to not be threaded So if you're a device driver author working on our Q chips, I keep that in mind And so yeah, that's all I really had there It's also not great in the case where you Can be invoked by the scheduler Directly, so there are a class of device drivers in the kernel tree right now that can be invoked by the scheduler itself Things like CPU freak These things cannot be threaded. So if these drivers are requiring interrupts Then you need to be registering these interrupt handlers with this extra flag So if you use request IRQ request threaded IRQ, there's a flags parameter as IRQ F No thread is what you can pass to actually cause the interrupt core to not actually thread your interrupts So keep that in mind. The fortunate thing is the vast majority of you I would hope are not working on CPU freak drivers not working on IRQ chip implementations if you are then I'm willing to help you out a few problems. So please send me an email or send an email to the Linux RT users list So that is talking about Implicit interrupts being disabled during interrupt Handler execution There is a different another class of interrupts being disabled that is explicitly Disabling interrupts using local IRQ disable or local IRQ save Device drivers should not be using these things But they are in fact, I think I did a grep. There's about 80 users of these APIs and the drivers portion of the kernel tree Which is I think manageable. It's something we can actually go through and audit each of these and figure out what they're doing But to give an example of why this is problematic I would hope you know at this point, but it's problematic in the case of the external event Can actually once it's fired it can actually execute the interrupt handler because interrupts are disabled And so it doesn't actually execute until interrupts have been re-enabled. Yes question. Yeah, so that's That's a question. So it's a great question. The question is what is my definition of external event? I should have been more specific. What I mean by this is a hardware event Whatever that means it could be a timer IRQ that fires. It could be an IO device that says hey, please give me a sample Depends on what you mean by priority. I'm not sure I understand the question So when I when you call local IRQ disabled the all the interrupts are disabled in the local CPU, correct? I'm sorry The same level. Okay, so you're referring to a hardware prioritization of interrupt features, right? So I don't actually have a complete answer to that question I'd be interested to hear about how you're using that feature because Talk to me afterwards. I'm interested in that question. But yeah, I'm not assuming a hardware prioritization of interrupts This is all purely a software dispatch IRQ dispatch But great question So there is a lot of questions I have like I told you there's like 80 Instances of this being used and the driver's portion of the kernel tree as a 414 I said don't use it Of course, I can't make that like really strong statement never use it There are situations where it's actually required again if you are Absolutely going to be executed in hard interrupt context or you're involved in interrupt dispatch in some way. That's GPIO IRQ chip implementations or IRQ chips Those sort of things it makes sense to use okla IRQ disable maybe But It could also be that it's just buggy or for example the drive the device driver is Wasn't written with SMP in mind. And so there's assumptions being made That the interrupts always going to be delivered to the local CPU and so it might just be buggy I'm hoping that there's very few of these. I haven't actually done a complete audit, but it needs to be done at some point And probably the most common usage here and Steven alluded this as well is that it's kind of a heavy-handed way to prevent the tasks being executed from being ripped away from the CPU so if that task is is Needs to access some per CPU data This might be one way that the driver uses that and there's a solution for those people that's using local locks So there's a question about local locks in the last section This is what local looks local locks look like currently in the RT patch. Now. I should be I should make this clear This is a feature in the RT patch currently that in the next few months will be proposed for mainline inclusion But here's what the API looks like you use this define local IRQ lock declarator macro And then within your function you can replace local IRQ disable And local IRQ enable with these local lock IRQ and local unlock IRQ And if you don't build on RT, these will end up redirecting to local IRQ Disable and when you build with RT It provides it the following guarantees. So writing your device driver. These are the semantics you should be Expecting from these API's Semantics are as follows basically What it prevents is? two different tasks running on the same CPU From running in the critical sections at the same time This is a we looked at the the semantics that that drivers are typically Requiring for these this sort of synchronization model And that appeared to be what is really required for a lot of people per CPU accesses and so on and local IRQ disable and so If these are the semantics that are good enough for your driver and what guarantees you need Then local IRQ lock is a good fit. I'm not going to go in each of these in details but under contention There's a mutex in place. There's proper priority inheritance and so on Again, if you have questions about this, you can send an email to me or Stephen or the Linux RT users list Because sometimes the semantics can be a little complicated But it might be a good fit So some of you might be wondering okay Well, I don't see a lot of local IRQ disable local IRQ enable calls and the drivers tree But what I do see is a lot of these functions. I see spin lock IRQ. I see spin lock IRQ save What's going on with them? Well, it turns out when you enable RT if you do spin lock IRQ Interrupts are not disabled.  It's mutex And we just ignore the hint that says also please disable interrupts and we can do that because interrupts are threaded And I'll talk about spin locks a little bit more when I talk about scheduling latency So now I enter into a different class of problems that still caused this IRQ dispatch latency and so this is a unique situation that we've we continue to run into run into our products and So I'm making device driver authors aware So to give you a little bit of background I was running a cyclic test test on my on my desk And I had an ethernet cable running to this box and it was one of those ethernet cables where the RJ 45 connector like the little plastic pin was broken off, you know You've always dealt all that with this I'm sure and the test was running and I bumped the cable out I got up to go get a cup of cup of coffee or something I bumped the cable out and I plugged it back in and there was just this massive latency spike And I was like what the hell happened like I just wanted coffee and So we did a lot of debugging and it was very confusing But we eventually figured out what the root cause was And it's something that you might not actually think about and I'm asking you to think about it so Here is the basic the basics of what was going on So the device when it was being initialized when I was unplugging the cable and replugging in the cable as soon as I plugged in the cable Did the network interface went up and down as soon as it went up or went down and up when the interface went up There was just a bunch of register rights to like some configuration block and this particular hardware device And then there was a read and the read was like, okay Well, I wrote all this configuration data now. Tell me when you're done doing your configuration, whatever Now On this particular device there was a pretty It was a PCI device and PCI Rights are posted so they're effectively buffered in the interconnect to be delivered to the to the device And that read that comes in is gonna push all of those rights out of those those store buffers in the interconnect until they get to the device and then It's gonna return the the read response back to the CPU But the thing we've encountered is that The CPU actually cannot dispatch interrupts until it gets that read response back So then you see something like this where you see okay I've seen this thread it did a bunch of right right right right right right and then did a read and it just like Was waiting for those buffers to propagate down to the device and for the device to do its thing and then to come back And in that interim time The timer had fired for the cyclic test thread and it can't be serviced because architecturally it's required that Interrupts are delivered in that particular instruction Order so there's some order architectural constraints in the hardware that requires this ordering and so therefore interrupts won't be let won't be delivered until the read completes and So and not when once that read completes then the interrupt can vector off But you can measure that latency and so I did So this is one this is the test prior to me bumping this cable out And so you see after this test run you have a max worst-case latency 17 microseconds not that bad And then I wrote a test case that all it did was just do right right right right like it picks a PCI device A memory bar that I knew was gonna be safe and just like pile down a bunch of rights and follow it by a read And then and then measure the latencies And this is what I get So this spiked all the way up to a hundred. That's like a tin tin X spike here to 174 microseconds That's pretty atrocious So the lesson is don't but my ethernet cable I guess But the question is what do we do about this And what am I asking you as device driver developers to consider when you're authoring your drivers? What I would ask of you is to consider this case where there's buffering in the interconnect What we actually did to solve this is actually follow each Right by a read back and that actually prevents the the rights from being stacked up on one another in the interconnect It's kind of a crappy solution. I wish there was a better one But it's what we have available If you have any other ideas, that'd be great there I've heard mumblings that in some of these CPUs you can have some Buffering at least in the CPU complex itself Construct timeout control timeouts and the store buffers and so on But that just seems like a lot of architectural black magic. So question. Yeah, I agree This is where the throughput and latency considerations just I Mean they they come at one another and I don't know what the right solution is Maybe the right solution is to provide a higher level Accessor that's like a ride L with read back and then like if you're building an RT then do the read back Otherwise, don't do it if you're not building for RT to bound that latency. I mean, I don't know we have options I don't know where they are. Yeah, the enemy is hardware. Yeah So the the solution to this class of bugs is to take your take the drugs away from the hardware people Yeah Yeah, speaking of drugs and hardware people Some additional MMI woes as I called him We've seen this Not not just buffering in the interconnect. We have one device That we've seen that actually will enter into some lower power state and then when when an MMI O transaction happens It will like exit from that low power state and that exit latency can actually be measured on the CPU side So again take the drugs away or at least ask the hardware people to provide some knobs that you can take and say Oh, don't go into this low power state, please Secondly Another thing we run out and run into was like this. I think it was like some spy EEPROM. It was just memory mapped It's a terrible idea, but it was done and don't do that, please So that's all I had on on hierarchy dispatch latency any questions there otherwise I'm going to talk about the other phase Okay, so at this point You have a thread that has been told this you've told the scheduler Please execute this real-time thread and From now on I'm accounting that the time spent from that point to the youth to that RT thread actually being scheduled in as the scheduling latency I'm going to talk about causes of of potential causes of the scheduling latency So to show you one of one such example This is very similar to the local IRQ disable local IRQ enable Case, but this is instead the set of accessor is the disabled preemption on the local CPU So just to give you what that looks like in the diagram So you have some thread executing it does a preempt disable And an external event happens and the interrupt interrupts aren't disabled So it just it executes immediately the CPU vectors off into this interrupt handler The interrupt handler is going to wake up this high-priority higher priority thread But it can actually be scheduled in on return from interrupt because preemption is disabled and So you have to wait until that thread has done a preempt enable and at that point you can schedule in the higher priority thread So it should be clear that usage of preempt disable preempt enable are going to Impact that that delta any questions about that The only reason that I can come up with after after thinking about this and maybe you can think of some other other ones, but is that Why a device driver should be using preempt disable and that is if you need to synchronize with the act of scheduling itself If you don't fall into this category then Probably should not be using preempt disable and you can use local locks as an alternate solution here And I did a grep. It wasn't that bad. I think it was only like 60 users in the driver's tree. So Not that bad. It's not like we're talking about analyzing, you know, 100,000 spin lock critical sections or something So it's manageable auditable Speaking of spin locks, however on mainline when you do a spin lock spin unlock on mainline Preemption is actually disabled So if you look at a mainline execution trace and you want to look at these these latencies You see the same sort of behavior the way that we saw them the RT case is make spin lock critical sections preemptable And we do that by turning turning them into PI aware mutexes And disabling migration is a little semantic quibble But in that case when spin lock when a spin lock is being held by a thread it can be preempted by a higher priority thread and so therefore you can Bring in that that outer bound Any questions about that that's a that You have to think about this for a while think about the spin locks sleeping for a long time before it's like How is that even okay? And if you combine sleeping spin locks and forced hierarchy threading you're gonna have to have both of those Pieces together and think about them together to understand how they actually Work and are successful for primed RT. So This is unclear to you. I'm happy to talk about it afterwards Because it's definitely Confusing and again the story is pretty clear. So if you are riding a device driver And you're using spin locks, then you're not really going to impact a RT thread priority Or RT priority thread So you're just lucky now again it works until it doesn't work And so if you're in the situation where you do need to synchronize with Some code running an interrupt dispatch Or things that can be invoked in the scheduler. So example for example like the spin the the scheduling core All of its spin locks internally are raw spin locks And raw spin locks are traditional spinning locks under contention and If you use the the raw spin lock API which looks pretty much exactly like the regular spin lock API by design Before you consider doing so Think about the cases where you might need to I need to do this because I am invoked from hard interrupt context I'm involved in interrupt dispatch. I'm involved in scheduling. That's a good good enough justification But think about it when you were doing it Because you need to think about the critical sections themselves and you want to ensure that they are doing the absolute minimal amount of work possible and Especially that it's bounded because any critical section is going to impact that worst case Delta, so please keep that in mind if you're going to do this in the future and Again, we're always happy to review things or at least I am So in conclusion these are the four things I hope you walk away with So the first one is if your drivers aren't involved in interrupt dispatch And you probably shouldn't be using local IRQ to say a bully probably shouldn't be using local IRQ save Look at using local locks Secondly just consider MMI O axis is I don't have a good story here just to think about it and see if you might get away with doing readbacks That might be suitable look at your own applications to see if you can Yeah, I don't know. There's I don't have a really good story here, but just think about it If your drivers aren't involved in scheduling then you probably shouldn't be using preamp disabled and Instead use it use a per CPU accessor. So the RT patch also introduces a Different set of per CPU accessors called get local CP get local Get local CPU and get local or something and put local local var Yeah, yeah get see yeah local var and put local var and those are access primitives that allow you to do per CPU access that only Disable pre only disable migration and don't disable preemption So that might be an option for you If your drivers are involved in interrupt dispatch and scheduling then you need to use raw spin locks You can't nest a regular spin lock within a raw spin lock That's bad In fact, we've discussed it for a long time about making that trigger locked up splats right now in mainline So we can catch these issues. We haven't done that yet But it's possibly coming in the future or certainly coming in the future. I just don't know when So I ended a little early and I have plenty of time So if you have questions, please ask me if you have questions afterwards or want Clarification you can reach me at julietini.com or juliet kernel at our work or my Twitter handler or Linux R2 users list. That's actually a great resource also great resources that pan linux or TNO if you see so We have a All the cool kids hang out. I'm told I don't know. Do I fall in that category? I guess Question up here. Yeah, so that definitely falls into the category of sure. Yeah, so Yeah, yeah, so this this gentleman mentioned that he had also run into what appeared to be some MMI Oh, what or in my woe as I like to call them And where he was saying you trigger an array cycle for a flash and then when you read the status register It blocks in the pipeline until it's delivered. Yeah, that's certainly definitely falls into that same category thing And I don't know what the right answer is But yeah, that would and he said power PC also suffers from from this pipeline stall So, yeah, I don't know what the right answer. Yeah, I'd be interesting to talk if you had solutions for that I don't know what you did or not you did. Okay. I like to talk about that Maybe afterwards. There's a question in the back here if you can get the slides Yes, they will be up on the on the scheduler page after this talk Also on my Twitter feed if you need them before that question so So the question is and and the question is how does IRQ threading work? So let me describe a little bit more about how it works and hopefully that answer your question And then if it doesn't please ask again In order to support IRQ threading it requires the use of the IRQ chip primitives is implemented So what actually happens is when the interrupt fires we jump into that little shim as I showed The shim actually will mask the interrupt at the interrupt controller Right, so the device is still asserting its interrupt because it hasn't been handled yet, but it's been masked, right? And then we wake up the the associated IRQ thread And if that IRQ thread happens to be the next highest priority thread to be executed Then the scheduler is going to execute it and on the tail end of that handler after the associated handler is executed Right before the thread goes back to sleep. It actually will do the unmask Okay, so the question is what happens if you have to real time like real-time events happening or interrupts happening at the same time Well it if that occurs then it's up to the user or system integrator to properly prioritize those interrupts Because you're now in the realm of scheduling you have all the tools available to you in that realm like adjusting priorities Or adjusting scheduling class even You can adjust these to be like round robin or FIFO or deadline even I'm sure I don't know Oh, yeah, so there's there is a Function called request threaded IRQ. So there's two main registration functions. There's request IRQ and request threaded IRQ So on mainline how this works is is the traditional ones request IRQ the handlers that invoked and hard interrupt context If on mainline you use request threaded IRQ you actually specify two handlers one of those handlers is actually what's executed in hard interrupt context and Then that handler can return IRQ wake thread and IRQ wake thread will actually wake the associated IRQ thread that will actually run in thread context So that's how it works in mainline today. So RT the way this works is that actually will force thread the hard IRQ handler and the IRQ thread in mainline Maybe I didn't get your point look at That line Yeah, so threaded IRQ is is also kind of a Generic work deferral for interrupt handling. So it falls into that category. So there are mainline uses where it makes a lot of sense As well, so thank you Steven. There's a question in the back corner. Okay, great questions Yeah, yeah, actually, yeah Yeah, yeah, I don't know if that's true. I think we'd have to look I believe this was changed In the past several releases because there were device drivers that we're doing a trying to acquire a spin lock in their top handler I don't know that we did we can look at I will absolutely look afterwards if you want to but Or Steven's gonna look right now, but I believe that even the top handler is threaded In RT as it exists now. Yeah Yeah, there is yeah, so tasklet is is one IRQ work. So the question is about is there anything special about tasklets Not really only that if you raise the task at soft IRQ, it's associated with the thread in which you've raised it and So when that thread is done executing the handler, then it will start executing the software cues that were raised in that thread So therefore in that thread context, right? You have your IRQ thread. It's executing It will execute the handler that has been force-threaded that handler is gonna raise the the task that software cue and at the Termination of the handler. It's still in that thread. It's gonna run the task that right then at the very end In the context of that IRQ thread. Yes And that's true of all so not just task list, but it's true of all software software cues that are raised in thread context Yeah, yeah, so the question is doesn't this break Any device drivers that might be involved in the real-time portion of this application of an application the answer is It depends You might see higher latencies because now there are more context switches But you should get better bounded behavior For those and you have to also do all the configuration necessary to actually get these things prioritized with respect to VRT application But maybe I would hope not. Yeah Yeah, that's a pretty tight bound depending on your on your system four microseconds is pretty Pretty tight. I don't know if you'll hit that Yeah, so even on the highest power like Xeon system that I've tested on I'm still seeing like 10 microseconds worst-case execution now in the arm systems we have we see maybe 25 30 microseconds, so Yeah, I mean and and like I said like for those really tight bounds and Yon mentioned this before in his presentation is if you need those tight bounds You might choose a different strategy. You might go to using Xenomai You might spin up and push all this stuff into another hardware unit and you can offload those might be solutions Or maybe it's not possible on this particular button. I don't know it really depends Any other questions Olaf was this was this just a general statement because I agree Okay, so for debugging this. Yeah, so first of all I'll repeat the statement that Olaf made he says that debugging priority inheritance problems that exist between a real-time application and a interrupt thread Can be painful to debug what techniques can be used to debug them? I typically just use the standard sketch timer trace points There's sketch a timer and IRQs or the typical ones I enable but it's really just slogging through some Traces and figuring out what's going on. That's not an ideal situation There was a conversation yesterday. I had it was a little bit interesting Which was that it might be possible to leverage something like? Well, let me let me back up a little bit. There are Unzini my apparently I don't I haven't used in a my much but for a an application You can have it such that if that application if that real-time application actually invokes a system call or something that that exhibits known unbounded Unbounded execution behavior it is sent a signal and then that task can either figure out what it wants to do a dump stage or Core whatever has an application debugging tool It should be conceivable to implement something like that using sec comp in Linux So I'd like to pursue that as like a usability to bug ability tool Especially for people that are taking applications from non Linux platforms to Linux But yeah, I would agree with you in general. There's a it's there's a debugability problem. Yeah Yeah, yeah Yeah, yeah So the comment is even if we chose some sec comp a solution It's going to be difficult to maintain because there's going to necessarily be a blacklist or whitelist that we use to determine whether or not a System call is allowed or not allowed, but exhibits be certain behaviors or doesn't and that's going to be a difficult thing to maintain Maybe it's going to be good enough for just application to bug ability might not be useful to actually enable in deployment So I think there's there's some trade-offs we can consider there any other questions Otherwise again, I'm happy to Answer questions via email how to review patches. Let me know Great. Thanks


Good afternoon everyone. Morning. Yeah, morning or evening or whatever where you are located, wherever we are. Is that is the areas framework. Setting up an agent initiative and credentials. Is that done by Indico or is that someone else. The thing that's happening like next week. You mean the workshop. Yeah, yeah. It's done by Ariel Karim and beer and so. Two from us at animal and then Ariel. Also does it. Yeah, I certainly won't be able to attend because I'm on vacation next week. But I think. Yeah, with spirit Karim and Ariel the workshop should be in good hands. I think we can get started. Okay, thank you for making recording. It is. All right, so. Welcome everyone to the Arizona jazz group called July. Six. Yeah, I need to remind you to abide by the high-pitched code of conduct and entity trust policy. You like the edge yourself to the tennis list for free to do so. I posted the link in the chat. Anyone new here that would like to introduce themselves. I think. Yeah. Okay. So for today, I don't have a lot on the agenda. So we can see if there's issues or PRs or things people would like to discuss and otherwise we can keep it short today. And I think it's also more vacation period and yeah, it's been less going on last week, but I know I saw some work on the did conflict to stuff and Ariel had left a lot of comments on issues and stuff. So maybe we can cover some of those. Are there any status updates was the Bible call this week. Yes. We didn't have a lot of new things, but a couple of things that I will mention is we're making some changes to the scanning flow. And there's some conversations around go codes. And I don't know if there's a potential opportunity for some of those go codes logic move to FJ. We're introducing the idea or the cons new concept of femoral connections or connection gets deleted when a transaction or operation is completed. The main driving force for that is the cure code density. Make that is smaller. When you were using connectionless for request. And I also thought that the areas community called there are some code about some talk about the pier did upgrade to feed did come. We did the two of three. I've added an issue to airs framework JavaScript. I feel like the, the one of the frameworks to, to implement those upgrades and not about a kind of a month framework timeframe, but they were asking what is the timeframe possible. And we say one month a month. We were not sure, but, but yeah, yeah, I think it's if I'm not wrong, it's, it should not be so complicated to add it this to a FJ right. What do you, what do you think. I, yeah, I just created that issue there so I don't forget but yeah there's not a lot of details so someone who understands more than me please update. Yeah, is this anyone willing to pick this up in the coming months. Yeah, maybe, maybe I can, I can do it but I will have to study a little bit how it works. Yeah, yeah. I can help you with the implementation or like guide you if you have questions. Cool. Yeah, and for the, a formal connection like make your code smaller. Have you considered using shortened URLs. Yes, we do not want to use certain URL service for. Possibility of some privacy concerns around it, whatever service is using now have the ability of track. Again, I piece and know what is happening. So the shortening URL was we decided not to pursue that side.  Yeah, okay. Yeah, no I think. Because when you like you connect to the HP endpoint of the issuer or like the inviter anyway so fetching a document from their servers I would say doesn't make tracking more possible because you already are connecting to their servers and domain for creating the connection. And we have in cases just integrated directly into the agent so the agent receives the connection request it also handles the. It handles the like the URL shortener I think, in a lot of cases you probably want to separate it but I think for easy deployment. Yeah, I think that has worked for us also quite well. Sorry, I don't understand the suggestion and so you're saying agent is in charge of the real shortening. The sure verifier. Well noted the creator of the invitation. This is a mobile wallet so they don't have a HP server. So we need to stand up like a mediator service something on the side. Okay, yeah. So if you have a mediator. There is a, an RSRC I wrote specifically for this and this is how I did it when I wanted to create invitations on device that were shortened is Is it in here. I can't find it. If you, if you are talking about your shortener protocol, it did come. Yeah, okay. Short. Here, yeah. So, this is a RSRC I wrote where you can use another agent to shorten it for you. And in our case, we used it with a mediator so the mediator, you need to contact mediator anyway to register the key, but you also can request the mediator to shorten the URL for you and then you get back like a shortened URL which the mediator then holds for you and there's like a requested validity seconds. This does add some like tracking possibilities, but you are already sending the key to to the mediator and when the other device wants to send a message to your device, it has to connect with their device to the mediator anyway so Yeah, still, all the connections between the servers are already made because the other device is already connecting to your devices mediator so having that having it resolve a another object from it I think doesn't introduce too much new things, but yet maybe something to consider. I'll look into that one. Thank you. And agenda. Does anyone have anything specific for the gender they would like to discuss. I think I think it would be nice to review some issues and PR just to because there are some PRs that are open for a long time, like for instance the one for revocation so maybe we can try to move forward or discard some of them. I don't know. Yeah. I think we can start with some ones that are really open really long. This PR, yeah, February 9 already to add concurrency conflict for process messages concurrently. I think the changes were quite simple. Instead of like waiting for the previous message to be processed we we allow them to be processed concurrently if you add this config parameter. It seems, I think. Yeah. Do we think they should be merged or I'm a bit out of like what it was specifically about. Yeah. Maybe we will, we will need to review it or just to see what what's missing there. Yeah. Okay. Anyone want to take them down. I will, I will, I will do a review. Yeah, I can also do it otherwise you don't have to pick pick them up. All of them. Okay, I'll take it. You can take the next one. Okay. Out of band proof proposal. Good addition. I think. Yeah, I think we also need to look into the transparency and see exactly what the concerns were with this PR. Yeah okay I think the problem is that we look without connection ID which in some cases we shouldn't do which I think can lead to issues if I send you a message with for specific threat ID I can like hijack a exchange I think. I wanted to keep this on hold until we have like properly integrated connectionless with out of band which we've never done. So maybe we can merge this with with some small tweaks. I think. All right. I can also pick this one of them. I think I left left comments on both of these. Okay. Yeah, this is I think on hold because the presentation exchange work is on hold. We have been integrating with it. But there were issues in the packs library that really prevented it from being integrated. I think. Once those are dressed I think they're currently being worked on in the in the presentation exchange library. We can integrate it and then we can also integrate this. Yeah, so I think we can keep it open for a while now so we don't lose the. The thing. Yeah, we don't lose the, I think it would be a good test to add. Okay. This one. And yeah, a lot of good changes in here. I think we need to update it to integrate with some of the latest runner updates. And yeah, so yeah, I'm not sure if anchors still going to pick this up so maybe. Yeah, we need to take it over and get it merged because I think there are a lot of good things in here like things that are going to break soon if we don't merge this and it updates it to all the latest versions. If we if we can add commits there I maybe I can, I can pick them up pick them up and. And just do it on the branch and. Okay, yeah, that I think can we. It's from checked you can push to organization things so you would need to fork their PR. Yeah, okay. Okay, yeah, it will be just a matter of asking anchor to accept the PR and let's see. Yeah. Okay. Okay. This one is open I just never finished it I don't. Yeah, I. I wanted to add a role to the credential improve records because it's sometimes very hard to determine which role belongs to a record, especially if they're in state dawn or like whatever state. But I just didn't finish it so something I'll have to pick up at some point. This one. Yeah, so using a multi use invitation multiple times. So I already connect received a multi use invitation and I want to use it again. The change being made here is to remove the check. I have already connected to this ID, and that allows it to just like create another out of vent record for ID. My concern with this approach was that as I understand it that these need to be like message IDs need to be unique within the context of an agent. And this breaks that model, because we basically have multiple out of vent records with the same ID then, which can get tricky. So I think if we want to reuse the same invitation multiple times we need to think about it like okay what's the impact of that or the implications. Yeah, I know Ariel you had done some research into this the last time right or. Yes, I think the only the only issue would be on the mediation recipient or module, because if you have this an invite or something like that. It will, it will try to to find a single connection from that invitation so it would break as you, as you say. But if we change the logic there. It should not have any, any particular issue I mean, according to what I understood, nothing from this from the spec are is preventing to have to two connections made or start bootstrap it from the same invitation. So basically, it's not incorrect to, to, to remove this check. And also, sometimes was something that that happens to me sometimes in our app is that you scan a QR code with an invitation. And if you are not connected to internet or you have a problem. You will have an error and, and, and if you if you are going to scan the same QR again you you have to make sure that delete the autofund record that has been created for that. Because otherwise the framework will reject. It will not allow you to to to to receive again this invitation I mean to with the same QR code so you will need to ask the other party to create a new QR code. And sometimes it's something that it's not so I know it. If you don't manually delete the autofund record, you will have several records that they are not used but I don't know. I think it should. But to have this, this, this change in the framework. But as I said, we will need to also modify the initialization of the mediation module because it will fail. If we, if we have more than one invitation, please show once wants to say something. I have a question over something that you say that when you delete a connection that have been records to remain so they're separate. I think so yes. Yeah. Okay, so we just probably to make some changes in by phone that so deleting a contact. Is it recommended to delete both the connection and out of band record should be fully removed. It depends on as a holder, probably yes. Because you as a holder probably yes because there's always a single connection type to an out of band record. But as like or like if you are the creator of a multi-use invitation, for example, you can have one out of band record with thousands of connections. And it could be that you remove one connection, but you don't want to remove all the other connections or the out of band invitation like new people can still connect to it. So in that case, no, but in in like the case of like I created the single use invitation, or I received a single use invitation or like received an invitation then it makes sense to also remove the out of band record. Okay. Is there anything on the flag or when deleting a connection or those are two separate steps separate steps because you can also have I can receive out of band invitations and reuse connections so that could actually be like a lot of out of band records also connected to a connection itself. So we could add some logic to remove associated connection records but it gets a bit complicated, I think. I just think we probably need to clean that up again from a mobile holder point of view, the user experiences they're think they're deleting their contact. So therefore everything should be gone. And it sounds like this out of band records is staying around left over somehow. Okay. Yeah, I mean, there's something to say to just remove the out of band record directly after you've created connection because you don't need it anymore. And so maybe that's also something we should advertise more. Alright, so upon completing the connection the out of band record can be purged as well. There's no need to keep them around. Unless, does it, if there is a, if there is any, if there is a multi use connection or not the out of band that call for a reuse connection, do you need that original out of band invitation. No, only, only if you create a multi use invitation. The multi use invitation will become invalid if you delete the out of band record because basically the out of band record is the multi use invitation. Yeah, if you're the one creating the invitation as a get it but if you're the one consuming the invitation then it's relevant. Yeah, then it can be removed. Okay, thank you. Which brings your question. Do you know if there's any plan for FJ to maybe create a API docs is in addition to the guides that already exists which are pretty good. Yeah, yeah, it's on the plan but like not something we have had spent time on or like I'm not sure if we have time to spend on it in the near future I think. Okay, thank you. I think we all hijacked as a conversation a little bit. No problem. So yeah, I think we can, we can. It would be good to remove this check but also about we will need to make sure that that the mediation module is probably updated. It is. What you call the trick Patrick can pick it up. Can you address this area or. Yeah, no problem. It should be a quick action. Sounds good. This one. Yeah, I think. Big PR. And I address most or all of the, the feedback you, you gave in May, I think. And there are still some questions, I think. But it would be nice if you can, if you can look at them. And I think you, you started doing a review when, when we were at the dice in three, three, but you didn't finish it. I think. Yeah, so I think I finished it. So I think. Okay, so if you, if you consider it finished maybe I can, I can do those small modifications and. Yeah, we can probably continue like that. I mean, it, we will be using only this manual way of our external way of handling the indexes and that stuff. And maybe in a future we can incorporate some automatic stuff in the in the framework if needed. And does it introduce any breaking changes besides to the interface of the. No, no, the only issue is that if you have. You don't have a registry and I don't get it in implementation you will have to update it because now you are required to, to also support these methods or include the method methods for revocation registry definition. Yeah. Right there before. I think we made that experimental right so that it can have breaking changes. You see, I made a PR for that right. Or it's probably this one. And so it's the Anacrylate for us module. Oscar. EBS checks to do to see credentials service. So I didn't add one to implementing your own registry, I think, or maybe on the guide. Okay, so I added it here. So I'm not sure. Oh, and it's also added. Okay, okay. Okay, so that's fine. Yeah. Cool. Okay. This one is a book I fixed to make the document loader work in note 18. But the CI now fills. So something I have to look into. Okay. This one is accepted and ready to merge. So, I think it's just a simple fix to take the role property into accounting in the request. We actually got something merge today. This PR experience still here. Yeah, it is. Oh, he, he left as soon as we started with the PR review. So, so maybe he's waiting to that. Yeah. All right, so we have actually incorporated these changes from this PR in a in the wallet we built ourselves but there is like it's dependent on a. PR that is in here. That we're also using and I think, yeah, so we need to wait for this to be really work so I can look into this again and try to get this merged. Then this one. Artem is on the call today. Let me see. So this splits up the demo into two cases. Yeah, cool. Maine and did comfy too. I think, yeah, we took a really long time to review this PR. I left a comment. Yeah, so I left a comment that I think it would be easier maybe if we can keep the same demo and then have Yeah, just adds like different features to it so did comfy one did comfy to so that you don't have to run completely different demos for different cases. Yeah, but what do people think of it and now for our to be of any comments on this PR or Hello everyone as I replied the intention was speaking the demo into parts it's like there are some models which are optional. For example, in the wallet and a scar wallet and you cannot just integrate both in the single demo. And in order to demonstrate this capability we need to create separate demos and similar for ledgers. And this is why I decided to split the common one and come with to also inter independent demos so in the future we can create new demos easily. Yeah. I think I'm fine in general with splitting it up I think it's just like if we can prevent by maybe adding like just if you add a feature that's only available and in some demo we say like this is not supported with in the SK or or something and it's always hard to decide I think where to split demos, because if you're, are you going to do it on this conversions or are you going to do it then on credential formats or exchange layers that converse open ID and, and then you get a lot of different demos that. Yeah, maybe hard to do but yeah, I think I'm fine with it. I don't know if anyone else has comments on this. I think I think it's the approach exposed by Artemis is okay. My only concern but which is not related to items work is actually because I don't know if you have seen that I have reviewed the problems on the D on the demo. In general, and it seems the demo as it is right now, it's not working fine. I think it's related to the problem that we have on the performance for ref nappy. Because you will need to patch the ref nappy in order to make it work properly. So, because I tried to run the demo, following the steps we, we say in the read me. And if we, I mean, if we do the repo clone and the young install and that stuff. We will not patch the ref nappy, because this is something something that is working only on CI automatically. So, it will work very slowly and, well, you will have those, those problems of timeouts and what has been reported sometime ago with this guys. Your suggestion I think was to move the demo to another repo. So, yeah, maybe, maybe if we are having more and more subsets of testing of demos, maybe we can do something specific or specific repo for demos and to show different flows and lectures and maybe we can do something like that in the future. Yeah, I think having a separate demo repo for the demo or maybe some example flows and that kind of stuff I think that. I think that's good, because then we can really simplify the whole setup you also have to do for the demo I don't have to need to have all the complex dependencies. And the issues we have here, and then we can just make the demo repository note 18 only. And I think yeah that would probably fix a lot of these issues. And also maybe I imagine that some people will will start from the demos to set up their projects. And if we are we are already using this ref nappy patch and that stuff. And we will not have these issues because you know, I know we we mentioned this in the on the documentation and the guides, but usually. Well, you know, we are usually people try to test the code and we will only read the documentation, they have a problem. So if we can if we can prevent them to have the problem, it will be better. Yeah, that makes sense. Okay, anyone objects to extracting the demo from this repo. Don't necessarily object. I just want to make a comment that you might also add complexity to maintain the demo. Because as the code API or a new version evolves. Now, there's an extra effort to keep that demo updated and it might be a red the case that it's already out of date for that reason but again just want to keep the risk that the demo becomes kind of afterthought, and it will just quickly become abandoned where which I've seen with other projects kind of what happens. So it depends how much you want to keep it in line with the code and, and if someone is trying maybe the main branch for AFJ, maybe it has latest features and they want to demo that one. So how they have to go back to another repository how do we keep those demo in sync with the features that are available in the main or in the features that were available in the, in a previous release, that's going to be a little bit of a challenge. Yeah, fair, fair comments. Yeah, it's kind of already like that. I mean, I don't think the demo is working for the past whole months. It probably should be hooked into the CICD or kept up with the date because, because certainly when I came to the project the first thing I did was look at the demo to try to you know get an idea of how to do stuff and there I just got a different mistake every time so it was fixed. So it was a little frustrating. Putting in another repo doesn't, you know, makes it a little harder to do. Yeah, maybe what's missing is that demo as you mentioned that that demo has to be CICD and tested as part of the build flow that's another test that it needs to be passed. But, but, but do you think demo that we can run the ref nappy patch if we put it into the resolutions of this package that Jason or I mean in the in the We could maybe buy like the problem is that in. I think there's a few problems. We could add it as a dependency and not with like the resolution. The problem then is that the package is published to only work with note 18 so if you try to then run yarn install in the whole repo in in in note 16 it will give I think an error or like your note engine is not compatible with this package and then it can install it. But if you were to bypass that issue. We could I think do it manually in the demo with a resolution field or something or we could just like check that if you're not in note 18, we throw an error either way, like, and we create an alias for the ref nappy package because we always use like the your patch version. I think that that should be possible but we, it would require some work and changes to the packages and how we setting something. Yeah, I do think there are some things to consider here, maybe some potential benefits with extracting it is that we currently now have to keep it up to date with every change in the repository, which is good, but if we don't do it, it's it's there's a lot more potential to break it. Well, if you extract it. And it can never be broken basically, because you always use a version of a fj. And the only thing it can is that it becomes outdated. But I'm not sure what's worse and outdated or broken demo broken broken is worse. It might have been outdated is like, oh, we don't do that anymore. We asked a question but broken is like what did I do wrong, as especially as a beginner coming in. It's hard to tell whether the error is like mine when I'm just a demo or if it's the demo itself that's airing out. Yeah, it's like, it's like the dogs right, right. It's like, if we wanted to, to have to keep the dogs in the same repo because we want to make sure that it is always up to date but it's, it is not necessarily true, because we are not running the demo on the CI CD step. So it happens what is happening right now that is, even if it's there it's it's broken. So, yeah. But yeah, I totally understand what what what what Glacier says it's true that we will need to, to keep an eye on other repo as well so yeah, it will be a little bit more of work but I think it's a good trade off. I mean, I know when when we make like a new big release also need to update the dogs at migration guides to the documentation website so I think updating the demo. It's probably not that much work. And then we would have at least a stable demo that doesn't break with every commit because I think running it in CI is quite complex because it's a CLI tool. So then we would have to write test for a CLI and interactive CLI kind of interface, which I'm not sure is worth the complexity. Makes sense. Okay, so. Yeah, I would then propose still to, yeah, extract it. And that we update it for like because it shouldn't even break between like zero or four one zero or four two so it should only break whenever. A, a breaking change release is made, which is like not often currently is this this can now be just a comment. Okay. Then cool anyone that would like to pick up this issue. Okay, and run out at this one here. And then someone can pick it up. We have the time. Let me see any PRs we haven't discussed yet. We discussed this one. Oh, wait, we didn't finish this one. But I think. Okay. Then I think we can get this merged. Can you resolve the conflicts, Artem? Oh, thank you. No, not this one, this one. Any comments on this one do we need to discuss anything about this PR. Yeah, well, I, I, I did a branch to address this. This what we discussed last week. And maybe you can open open there this link. You will see the changes. So, what I don't like about this approach is that we are. I mean, the code is quite straightforward to read and to, I mean, it's just a few lines of code. But what I don't like is that we are, we are, we are having the non credit API. And it's config as a dependency of the services. It's not something that I mean, it's not something natural, I will say, right. And also, the, I, I think that it's, it would be more straightforward to, if, if we can have this, this auto create link secret configuration as a configuration of the of our non credits a race or in the SDK directly. So in that, in that way, we will not be dependent on the, on the non credits module, but only on our own configuration, because we are using a config parameter from our non credits from another module. So, if, if our non credits module config is say something and we are, it's up to us to, to implement it or not. No, no one will force us to, to do what the config. The non credits module config says, I mean, we, if we, if we create another implementation of a non credits. I don't know, I don't get anything C, C++, I don't know. And we create another module. And we are not following this. I don't think it's out in secret parameter. Nobody will know about that. So, I don't know. Yeah, I think it form a user perspective I would say it makes more sense to have it in the honor credits module. Do you also think that's or I, for me, it will be the same because we will need to. To instance, both an uncrest module and an uncrest errors module. So, using the configuration in one or another, it will something that will be always. It will be on this on the same constructor in the end. Yeah. Yeah. I think the default value is true. So probably most people won't have to deal with this config option ever. Anyone has an opinion on this I see we're over time so anyone has like strong objection or like agreement for either of the two. Yeah, but again, we identified this is really, I don't think we're to opinion where the need to just you go it's just a matter of should be work out of the box. But if you have any orientation guidance yeah please provide as a comment and I can talk to Jason about it. Okay. I think, yeah. Both are fine for me so I think your approach is good to me as well. Okay. Let's. Well, yeah, we're out of time. I think we almost got to all of the PRs. Except for the. The memory creation. It's a very simple one. It's okay. It's just to to fix the issue that Charles found some time ago, that we cannot use the in memory wallet for Oscar. Okay, I'll take a look at it and review it. Okay. Perfect. All right, thanks everyone for joining today, and I'll see you or I'm at a vacation next week. So, yeah, I'll post in this court maybe somebody else will take over but I think we'll just cancel for next week because there's like less going on these these weeks, either way so that we can just skip the meeting for a week. All right. Thanks everyone. See you later. Bye bye.


don't mind being boring. I don't try to use visual things to attract you to make you watch my videos because, and this is counter to what most marketers teach, most marketers teach that you should use visual interruptions and wonderful looking things to attract people to watch and keep watching your videos, right? You know, really savvy editing to get to get and keep their attention and I don't do that because I would rather have people be trained to engage more thoughtfully with my content. Let's think about this here. If you use attention getting tactics, right? Videos with very good editing to like so that the most person who gets bored the quickest will not get bored at all. If you do things like that, one it takes a ton of time and effort, okay? But secondly, you are training your audience to be more and more shallow in how they relate to you to engage with you. You're getting more and more shallow engagement with you. And by doing that, let me ask you this. When people finally use your products or services, don't they need to do something to transform themselves? Don't they need more thoughtful engagement? So in other words, your product and services require more thoughtful engagement than even your free content. That's true with my stuff. When you buy my courses, you got to get to work, right? When you join my programs, you got to be ready to transform yourself with my guidance and help and all that. But it's thoughtful engagement. And that's the kind of relationship I want with my audience.  And what happens is I'm training them to engage in that kind of shallow way. So for example, if you look on my Facebook business page, right, I don't use any images. I use writings, and I use videos. And that's it. Now, the only time I use images, if I'm changing my cover image, because I'm selling a new course, or I post a Facebook event for my new course, there's an image. But those are the only times that's like once a month. The rest of the month, it's just words, and it's videos, both formats reading and watching videos requires thoughtful engagement. It's not a drive by like, right without. So if if you and then this applies to Instagram to notice on my Instagram, Instagram forces us to, to do either videos like this or to post images, right, we can't just do writings. But if you notice on my Instagram images, they are not just pretty pictures, right, to try to get your attention and then make you read. Instagram, my Instagram images are some message. And the message usually is cryptic enough where you have to actually read the caption to know what's going on, right? So it's like a cryptic message and you have to read the caption to like thoughtfully engage with it. So, so you might say, well, George, what's wrong with posting pretty images or attractive images to get people's attention so that they might engage? Here's what's wrong with it. Here's what's, here's the problem with it. There's technical marketing problem with it, which is the more pretty or attractive images you put up, the larger of an audience you're going to have that are drive by likers, okay, drive by likers. And so in a larger audience like that, therefore your ideal audience of people who actually want to do your work and transform with you will be a smaller and smaller percentage. Even though the audience might be growing, it's a smaller and smaller percentage of the total audience, which are a lot of shallow likers. I'm not saying those are shallow people. I'm saying that you've trained them to engage with you in a shallow way. It's, it's, it's your doing. It's not that they're shallow people. It just you're training them to be more shallow in how they relate to you, right? So then therefore your ideal audience, it gets smaller and smaller percentage because of the total audience keeps growing.  You're going to, you're going to have to, again, make something really, really exciting for them to buy anything. You see, which then creates the temptation to, to lie in our marketing, which is why if you notice when I sell things, right? When I sell things, it's very mellow. It doesn't, I don't use any hype. Oh, you're going to get these results. You're going to, you're going to make six figures. You're going to be able to buy an island or whatever. No, I just say, Hey, if you want to learn XYZ, that's what this course has been designed to teach XYZ. That's I'm not promising results, nothing. And I get plenty of sales because I've now created and these are, these might be the same people. When I say shallow engagement and thoughtful engagement, they might be the same people like you might be engaging me with me thoughtfully, but you might be engaging shallowly with other people who just post pretty images. Oh, that's easy to like easy. So I hope this is helps. I want you to think about this. And it will, if you have been posting pretty images, right, to, you know, to attract people and hopefully they'll be, it's going to take some time to transition them to more thoughtful way of engaging with you. So I hope this is helpful and food for thought. And I hope it helps you to create more and and and truer transformation for your clients as you go forward. Alright, take care.