Hi, my name is Tsu Min Chen and I'm a Principal Software Engineer at Red Hat. For the past year or so, I've been working on the Elastic Secure Infrastructure Project, or ESI, along with other Red Hat engineers, as well as people from the Massachusetts Open Cloud or MOC. So what is ESI doing? Well, from our README, we want to create a set of services to permit multiple tenants to flexibly allocate bare metal machines from a pool of available hardware, create networks, attach bare metal nodes and networks, and to optionally provision an operating system on those systems. One key thing to note is that ESI does not compel users to a specific service architecture. You can put the pieces together yourself to support your own architectures and project goals. That's your goal, but what is ESI actually doing? To explain that, I like to think of ESI as more of an enablement initiative that aims to fill the gaps in OpenStack in order to fulfill its goals. So we'll work upstream with the established OpenStack projects to expand feature sets, we'll write custom code where needed, and we'll provide guidance and support to other groups working in this space. The best way to illustrate the sort of work that we've been doing is to talk about an ESI project that we're currently working on in the MOC. That project is an internal hardware leasing system. The goal is to replace the MOC's current internal hardware leasing system that uses complex custom scripts that are powerful but difficult to maintain or extend. The idea behind the ESI version of this hardware leasing system is that users contribute their own hardware to the bare mill pool. Those node owners have the exclusive use of their nodes. However, when the node owner is not using their node, they can put them up for lease, allowing the less see to temporarily claim the node and use it for themselves. At the high level, this is what the hardware leasing system service architecture looks like. In the middle of the DAW, it's a bare mill service where we keep our inventory of nodes. A leasing workflow allows owners to offer up their nodes while they lease nodes. Then, a provisioning workflow allows both owners and lessees to perform provisioning actions. They can provision their nodes or configure node networking. Let's dig a little deeper into the hardware leasing system workflows. We'll start with the creation of offers. Let's say that the bare mill inventory has five nodes, A through E, with nodes A, B, and C owned by owner 01 and nodes D and E owned by owner 02. Owner 01 is not using any of their nodes, so they offer them up in the leasing service by creating an offer for each of their nodes. Owner 02 isn't using node D, so they offer it up as well. However, 02 is using node E, so they keep it for themselves. Now the lessees come into play. Let's see L1 wants to use two nodes, so they look at the available offers and create a contract for nodes B and D. Lessee L2 only needs one node, so they create a contract for node C. No one creates a contract for node A, so another lessee can take it. And no one can create a contract for node E, since it was never offered up in the leasing service. Once the nodes are leased, the lessee can move into the provisioning workflow. Lessee L1 can use the two nodes they've leased, nodes B and D, in conjunction with the public network and the two private networks that they've created. They can perform a variety of actions, provision a node, attach networks to their nodes, and network actions will be reflected on the switch. Lessee L2 can do similar things, but they're limited to node C and the public network. So that's what we want to do. What steps have we taken to get there? To answer that, let's look at the partial requirements breakdown from the Halcyon days back in 2019. It's not a complete requirements list, but it illustrates a lot of the work that we've done. Let's start with the bare-middle service, or Ironic. We need to be able to add and remove nodes from inventory, something Ironic already does very well. We also need the ability to assign owners on lessees to a node, and for those owners on lessees to be able to perform limited API actions. Combined, this translates into nodes multi-tenancy, and it's something that was not supported by Ironic when we started working on the hardware leasing system, and it ended up being something we implemented ourselves. Next are the network requirements. We wanted users to be able to create their own networks and to potentially share those networks with another project, things that Neutron already supported very well. The MLC also needed ML2 Ansible networking to support the Cisco Nexus switch, and we discovered that was something we need to code ourselves. Finally, we needed the ability to create and manage trunk ports. After some investigation, we learned that Neutron did most of what we needed, but there were some pieces that were missing. The next requirement is a bit of a strange one. Simplified CLI commands. We'll talk about this more in a moment, but the gist of the issue is that the default OpenSack CLI is comprehensive but not always easy to use, so we decided we would need to give our users some simplified CLI commands. Finally, there's a leasing service, which we created from scratch. It needed to allow node owners to offer nodes and for lessees to contract a node for a given time period. Let's go into more detail about the work we did. We'll start with Ironic node multi-tenancy, which was the key feature we needed above all else. We started by talking with the upstream Ironic team about our requirements, and they were amazing at walking us through our ideas and implementation, helping us refine our spec and giving us feedback about our code. The implementation steps started with adding owner and lessee fields to nodes. Ironic actually already had an owner field for nodes, but it was purely informational. Next, we exposed those owner and lessee fields to policy. OpenSack policies determine which users have access to which API actions and can be specified in an easy-to-update configuration file. Before our work, all Ironic API actions were only accessible to Ironic administrators. We added two new roles, isNodeOwner and isNodeLessee, that determine whether the user making an API request is the owner of the node or the lessee of the node. With those in place, we can now update the other policy roles. For example, we can update baremail node update to allow access if the user is an admin or the owner of the node, and we can update baremail node setPowerState to allow access if the user is an admin or the owner of the node or the lessee of the node. Note that these policy roles are fully customizable by each individual OpenSack installation. There were a few additional minor tweaks necessary, but that was mostly it. We tested our code with single Ironic CLI commands, and then we tried something more complicated, using Mel Smith, which is a client-side Python library for provisioning Ironic nodes. What we discovered was that Mel Smith just worked for node owners and lessees with our code. Owners and lessees could provision nodes that they owned or leased, and cannot do so for nodes that they did not own or lease, which was exactly what we wanted. Onto networking. We started by adding support for the Cisco Nexus Switch into ML2 Ansible networking with guidance from the upstream networking Ansible team. They were great at helping us understand what we needed to do, and helping us merge our code, and that was that. Next, we tested trunk port support. We discovered that Neutron already supported the creation and deletion of trunk ports. However, we couldn't attach a trunk port to a specific NIC, so that's a feature we added, actually, in Ironic. Lastly, we found out that updating a trunk port didn't fully work. If a trunk port was attached to a NIC and then updated, the change would not be reflected on the switch. So we made plans to implement this feature, and we got around to working on it. We discovered that someone had been messing with the punch by a week or two, which was great. Simplified user commands. Why? Well, because if you use the default OpenSax CLI, figuring out how networks are attached to a single node requires two plus two times the number of NICs commands, so six commands if your node has two NICs. You also have to use the output from one command as input for the next, copying and pasting UUIDs. Then, you still have to collate the results of all those commands. But what we learned was that it's actually really easy to extend the OpenSax CLI. No one on our team had much experience with the CLI code, but within a few days, we had developed OpenSax ESI node network list, a single command that produces the output you see here. We've created additional CLI commands to simplify various operations, and you'll see a lot of them in the moment in our demo. Finally, the leasing service. We started by looking at OpenSax Blazor, which is OpenSax Reservation Service. However, at the time that we looked, Blazor did not have full support for provisioning on Ironic nodes. It was also tied closely to NOVA, something we were trying to avoid as Ironic can provision nodes on its own, and we wanted as simple of a service architecture as possible. Lastly, Blazor did not support policy-based node API access. You might be able to provision an Ironic node, but you couldn't allow a Leci to power-circle node. So last summer, with the help of a bunch of Red Hat interns, some of who are now full-time Red Hat engineers working on ESI, we developed ESI Leap, which is a pretty simple alternative. It allows resource owners to offer their resources and resource Leci to contract and offer resources for a specified time period. For Ironic nodes, contracting a node just means setting the nodes Leci attribute. ESI Leap supports a leasing of Ironic nodes but can easily be extended to work with other resources. Let's take one more look at the service architecture of our hardware leasing system, and hopefully you have a clearer picture of how all the pieces interact. One thing I really want to emphasize now is how disconnected our leasing and provisioning workflows are. Maybe you don't need a concept of Leci's or maybe you're fine with owners assigning Leci's themselves. You can do whatever you want there and still take advantage of all our work in the provisioning space with Ironic node multi-tenancy and simplified CLI commands. Okay, let's move on to our demos. First, we'll see a demo of our leasing service and Ironic's multi-tenant node capabilities. We're going to start with three users, each with a different view into Ironic's node inventory. The admin user sees every node in the inventory. The owner sees the three nodes that they own, and the Leci has not yet leased any nodes so they see no nodes at all. Now the owner has decided that they'll share Dell 2 so they create an offer for the node by specifying the Ironic node resource type and the Ironic node's UID. Note that the owner can also specify a start time and an end time for the offer, but we'll just specify an end time so the offer will be immediately available. A Leci can see the offer and when it's available. They can now create a contract for that node. After the contract is created, note that the availabilities for the offer are also updated. Other Leci's can still contract the node for those times. When we reach the contract's start date, the manager's service will fulfill the contract by setting the node's Leci field to the Leci's project. When the contract's expiration date is reached, the manager's service will also unset the node's Leci field and clean the node. Now that the contract has been fulfilled, the Leci has access to the node. They can also perform specific actions as dictated by Ironic policy. For example, our Ironic policy allows Leci's to power nodes on and off, but not to set or unset node attributes. Here are our Ironic policy changes. This is the generic node update rule which allows API access to only admins and node owners. And here's the set power state rule which allows API access to admins, node owners, and node Leci's. This Ironic policy is restrictive but still allows Leci's to do pretty complex things. For example, using mail Smith to provision a node. This operation takes a while, so we'll skip ahead in time. Provisioning is now complete and the Leci can now lock into their provision node. That concludes this demo. Next, we're going to look at our custom OpenStack CLI commands. The base OpenStack CLI is powerful and comprehensive but often requires a user to run multiple commands. When I run an ESI CLI command, I'll also tell you how many OpenStack commands will have been needed for the same result. We'll start with a node owner who can see the three nodes that they own. Let's look at how neutron networks are attached to these nodes by looking at the bare metal nodes and ports. Looking at this bare metal port, we can see that this corresponds to this node UUID which is still two, and there's no information in internal info so you can see that this bare metal port is not attached to any network. Actually, we can also use an ESI CLI command to get the same sort of result. In here with one command instead of three, we can instantly see that none of our nodes are attached to any networks yet. Let's verify that there are no networks attached to Delta 2 on the switch and we can see that there's nothing attached. The next thing we're going to do is create a trunk port. Let's see what networks are available. We're going to use these three test networks. We'll start by creating a trunk port with test one network as a native network and test two network as a tag network. This ESI command would take five separate base open stack CLI commands to run. We've created the truck port and we can see some summary information. This summary information would have taken four base open stack CLI commands. Now let's attach this trunk port to Delta 2. I'm going to specify the trunk port, the MAC address and the node. This command would take seven to 10 open stack CLI commands for the equivalent depending on what state the node is in. The network attachment is complete. Let's take a look at our summary node network list. Now we can see the attached network on Delta 2. We can verify this on the switch as well. VLAN 611 and 612 should be accessible and we see that that is indeed the case. Next, let's add another tag network to the trunk. This would take four base open stack CLI operations. Let's verify this update with our ESI CLI commands. We can see that test three network has been added as a tag network right here and our summary node network list also shows that same information. We can verify this on the switch as well making sure that VLAN 613 is accessible and that is indeed the case. Okay, now let's clean things up. We'll start by detaching the network from the node. This would take three base open stack CLI commands to execute. Let's verify that this update happened on the switch and we can see that the switch port is now shut down. Finally, let's delete this trunk port. This command would take five base open stack CLI commands for the equivalent. That concludes this demo. So what's next? Well, we're working towards a trial implementation of our hardware leasing system in the MOC with just one key feature missing which I'll talk about in a second. There are other scenarios where we might use ESI. A red hat colleague working on OpenShift thinks that ESI would be perfect for creating OpenShift bare metal clusters that can span and shrink on demand. There's also FlockX, a PhD project at BU. FlockX is a hardware marketplace where users can buy and sell use of bare metal nodes. We also have some feature development to do. We've already started to look at no attestation with KeyLime. The MOC also needs Cep IScuzzi support in OpenStack for its hardware leasing system and there's an upstream patch that we'll be testing out. Then there's improvements to the leasing service, reporting reservation limits so that one SCCAN leads all the nodes until the end of the time in the UI. Further information, the Red Hat Research Quarterly has an article about our multi-tenant Ironic implementation called isn't multi-tenancy ironic. So if you'd like more details about what we did, read all about it there. The ESI Get Repository is a good place to look at our code and documentation. And finally, you can reach us at any time through IRC on the free-node server in the MOC channel. Thank you for listening to this presentation. I hope you enjoyed it.


President Mahmoud Buhari has pledged his commitment to hand over a strong, repositioned, agricultural-led, diversified homegrown economy, also a stable democracy and revamped and security forces at the next administration. Buhari stated this at a dinner in honour of the 2022 Committee of Business, Political, Media and Civil Society Leaders at the State House at Buhari on Monday. The president said he was looking forward to completing his tenure in 2023, leaving a legacy for a united, peaceful and prosperous Nigeria with 24 years of uninterrupted democracy.


So, well it would be unfair if you have a conference on failure scale and we don't talk about microservices. You know, I've been sitting here wondering why is no one talking about it. Anyway, so we'll talk about it. So, this is about, I'm going to talk about microservice architectures, design patterns that you consider while building a system using microservices and how a framework that we call Gilmer is going to help you do all of that. First of all, who am I? I'm Piyush Verma. Sadly, I'm not Walter. The, my Twitter handles me's on 10. You can write, drop me an email and in case you do want to know more about my website, you can check that out as well. So, let's start with how many of you involved in microservices or intend to do microservices in your architectures? Okay, that's fairly a lot of people here. It would make a good talk then. So, what exactly are services? This would be a quick recap of the rest of you who didn't raise your hand. Actually, first of all, let me see how many people can raise their hand like in general? Oh, cool. So, everyone can raise their hand here. Now, I just wanted to make sure the maths do right. So, what are services? Services are logical entities that run separately from code units. So, think of them like processes in your operating system space. The idea is that they don't interfere with each other, each one of them being absolutely narrowed down to the lowest possible thing. So, I was having a word with Anand yesterday and he gave me a very good definition of this. He said, so, we as software engineers, you know, keep, we don't have an end to our software. We keep writing software and realize, okay, it doesn't end. But the whole microservice paradigm has actually changed that we are going the other way around. How much can we trim it down? Where it does nothing beyond a few bunch of lines and that's where we call it an end. So, that's what a service would be. So, as I said, they're independent of each other. They can be comprised of one or more services like they could be a service which does addition, multiplication, subtraction, or could be one which does all of them put together and calls the other services invariably. Services span across multiple servers for load and tolerance. And this is the part that we have been talking most about at this conference on how do you handle multiple servers and the services comprise of many servers put together. So, you could have like tens and hundreds of notification servers and one service which is responsible for all of that put together. Then services help you scale and are flexible. What do you mean by that? Primarily is I mean because it spans across service it becomes scalable but also the fact that you can do it in multiple languages. The same service could be written in Go. There could be another service which is being written in Java. There's another one which is written in Ruby and all of them comprise to make one system of yours. And that allows a service of rented architecture really easy to adapt to and really lucrative as well. This is something that I found on internet and I was stuck with it like this is like an amazing definition of why we need services. Everyone fails. I mean we've talked about it repeatedly in the past 36 hours and but as from the management perspective we need to know who do we blame it on. Might sound funny I mean the managers are not the bad people because on why it comes into the picture I'll explain. So let me actually do this the other way around. Let me tell you what the management problem is. So let's say you have a payment service now. Usually how we design our systems is we have a monolith which comprises of a bunch of interactions happening and we usually get by saying that we are a small team and we cannot really afford to have multiple distributed servers and we're going to make it in one single thing. So there are three people who are actually contributing to the same service. Let's call it a payment service. It's 1 a.m. in the night and something crashes. I bet you it will crash one day or the other. Who do you call which one of the three people is responsible for it. Now to avoid conflicts like this is where I say Microsoft is architecture has a lot of business value. This comes from Netflix School of Thought as well which is you few own one component of the system. You are entirely responsible for scaling it out for deploying it for maintaining uptime and for troubleshooting and fixing. If there's something that you cannot do you might as well offload it as another service to another someone who will be responsible. If your service is not spitting out errors when it should it's your problem. This accountability is something for which I personally prefer Microsoft is architecture above all the other things. I mean infrastructure problems can be taken care of. I mean of course they cannot be at the scale of large companies like Facebook Google etc. But to for the most of us we can do tackle it. I mean there are dozens of tools and frameworks out there which help us do it. But this is one single problem where we'll all fail every time and then the software problem is scalability. I'll cover this part later when through my talk on what exactly by what I mean when I say scalable architectures. Now just like any other computer science problem or any other thing we have design patterns to any Microsoft is architecture. So how do we identify them. Let's go through one by one. One of the foremost important design patterns is a request response architecture. This is what we traditionally have known as what we do with the city. A lot of people who use how many of you have used go kit or use g rpc in some form or the other in production. OK that's some some digital ocean guys OK a fair few other hands. So these are so request response is primarily where you actually I mean as the name suggests you request something. And you would get a response. Let's see what a sample would look like. I have a caller and I have a service. I send hello and I get aloha. There's a response back or I could get a wrong number. Whatever be the case I'm always assured that there will be a response. Be it a good data or be it a bad data. Now comes a tricky topic here. How do you differentiate a failure. When you get a response over here at first I got aloha then I got wrong number. How do I differentiate what is good and what is bad. So usually the essence is that every response comes along with the code. This is what is CTP status codes is also known for. So if you get anything which is 200 it's good. If anything that is greater than 400 or equal to 400 is another condition. Now this is important because we don't have to really parse the output every time and see if it's really adjacent really a hash of an array or a string or something. Now if you're using Gilmer now this is the time when I actually start using some code samples. Most of these are written in go but the framework is pretty much available across Ruby and Java as well. So I'll just I mean most of this is just pseudo code sort of a go make start easy. So every data transport that happens using Gilmer always has a sender ID. The importance of sender ID is I'll touch later but just to uniquely identify every single message that is being sent out and also it returns a code. The code if it's greater than 400 we know it's an error otherwise 200 is good. And then there's a sender the data is any interface that you intend to send back from the service. A typical request response structure done using Gilmer would look like this. I create a new message and I attach a new data to it. I create a request and then I send it when I do G dot request. So over the course of the stock every time I say small G that would mean a Gilmer instance and a capital G would mean the namespace or a class whichever you want to say it. So when I do a G dot request the first argument to that is echo echo is a topic. So all communication exchanges unlike STTP based transports will do our topic exchanges. So this is one thing which is inherently different from conventionally when we say macro service architecture. We always say that there is some URL or endpoint that we're going to hit. There'll be some IP address and then we'll path back into it with Gilmer. We all do it. We do it over purely non STTP transport and that is where readers comes into the picture. We use readers underneath. You can have your own choice of backend as well. That's really pluggable. So yeah, the choice becomes of a topic. So echo is a topic where you send your data and you get a response back. Now on the other hand on the server side or since there is no server on the service handler side, there's a new handler that you create and you say I'll reply to echo with this function is then the function basically just sends a data and says Pong. I'll do a demo of this later so you can just just quickly go through this code. The one of the most important patterns apart from this is asynchronous or signal slots. How many of you have used cute one of the core KD libraries that used to be around. I mean still around but was fairly popular. Okay. How many of you use JavaScript? Okay. Good. So every time you see the event pattern which is running in your browser is pretty much running on the same concept. So there are signals and there are slots a slot is a receiver for a message. So if someone who has to get something done emits a signal and says and has almost no control over whether a slot would be received or not. Sounds like there's a loss of there could be a potential loss of data here. But this is pretty much one of the most flexible and scalable architectures. I'll explain now. So let's say we design a sample shopping cart example. So and the Gilmer is an engineer. The web could be anything which is a web client or it could be another service calling another service for the sake of convenience. I'm assuming that this is this is somewhere hidden at your back end where after a purchase was made, you're supposed to send out notifications. So an item was purchased and now you're supposed to notify the user based on this and SMS notification to be sent out and there's an email notification to be sent out. And suddenly a few months weeks later, your management realizes that oh, we're going to make this app mobile only. And now you have to add a push message server to it as well. Why would you do that? I don't know. That's a business decision. Now each one of these service the services that you're bringing up now will listen on a topic item dot by an item dot by is now there's a star next to it. The star is pretty important because every time you're talking about UDP like systems, there is a notion of broadcast and that translates as wildcard topics in subscriber publisher patterns. The so each one of these services that you have actually is listening to item dot by star. So if you purchase an item, which is item dot by dot 500, all of these would get it because they are subscribing to a star pattern. Now let's say there is an online carnival going on and now you have to attach a new service. This service is responsible for payback points or a free delivery or a free good or anything something which is really unique to a particular set of items. So you attach a new service which is VIP item purchase and this one only works on item dot by dot 420. Now the next item that is purchased if it was again item dot by dot 500, since the topic does not match this service is not going to handle it. Whereas the rest of the three services that are currently running out there will continue to work. Now let's say you have too much of traffic going out on the push notification service. This push message server you have to scale up. So you keep adding more and more servers out there and since all of them continue to listen to item dot by dot star. Every time you buy an item, the message would be relayed across to that service. However, this will raise a challenge here. If you notice this would result in if I had three servers there, this would probably result in three messages being sent out. So I still have to trap that. So how do I do it? We include we introduce this concept called exclusion groups. How many of you use Kafka here? Or have you so you must be aware of what groups are. So the promise here is that in this group, if I tag it by saying a group push message, only one of them is going to receive this message. So even if I scale it to say 100 nodes, only one of them is going to process that message. This is an example of how you can achieve function point scaling through asynchronous patterns. So despite me and now if I was to scale a SMS notifier, I would do that as well by just adding another server there. This is a typical example of an asynchronous pattern that you will implement using Kilmer. The signal part is fairly easy. If you look at it, it says G dot signal, you specify the topic that you're sending it to. You construct a new message, a set data, which is a string on the receiver side. You have a G dot slot which says I'm a slot for this log example dot log and there's a handler next to it. Now if you notice carefully, there is no response here because since there could be a broadcast happening as well, there could be so many of the servers who are processing the same message. A response doesn't make sense. So if you have to think signal slots, think UDP on how you would design a system like that. The question now arises is what do you, which pattern do you use in your application more? Would you prefer using a signal slot or would you prefer using a request response pattern? The answer lies in, I mean, as a usual answer is entirely depends on your application use case. But let's start with the request response. Where would you use it? You would use it where you want a confirmed delivery where you're sending a message and now you want a response for it. The response could either be an error or the data, which is a valid return, but there has to be something. A request response pattern is something which is purely STTP like. So traditionally, whatever you've been thinking of a server architecture where you call a URL and you get a data is what you would use. And also the delegation and the responsibility of the error is what defines it the best here. If your caller is responsible for carrying out an action based on an error condition, which could be you try to make a payment and the payment failed. Now what happens next? Is this something which is which the caller is supposed to retry or call another alternate? Now the responsibility is for the caller, you would use a request response. Where would you use signal slots? Well, it's absolutely UDP like if you think about it, it supports broadcasting. So if you have multiple receivers for the same message and you want to send it out a fan out operation is something that you would use signal slots. Wildcard topics when you want want utmost scalability where you really think that there's a there's a caveat to it that because there is no response being sent from signal slots, you can never really guarantee whether the message was lost or transit or the server just died or something happened. So if you believe that at an infrastructure level, you can really do a good job of maintaining servers which are constantly alive. You would use a signal slot. Unreliable delivery when you're okay with this fact that there could be someone who is not receiving this data, you would use it. And the receiver is responsible for error. Let's say I'm relaying the error, but the service who's sending it does not really care about what like let's take an example of log forwarding. So from the application servers, you're actually emitting out logs. What you're using there is mostly UDP. So you use how many of you use ours is log D or services like paper trail. Okay, that's a you guys rest of you should use something logs are important. But if you're using your your UDP, your application doesn't care whether your log reads to the eventual destination or not. All it does is it just keeps spitting out log messages. There is a receiver at the other end, which decides now what to do in case that the log was not sent. Do I buffer it? Do I retry the file? All that is a responsibility there and hence you would use a signal slot. One of the bigger another challenge that these days is pretty hot topic. I would say with all these any of the hashikov guys out here. There was one yesterday. So there's a tool by them called console and then they have zookeeper. How many of you use zookeeper somewhere? All right, root 53 for load balancing and discovery. So service discovery and load balancing is a great topic to talk about. So let's see how service discovery actually works. So let's say you have a notification server and now you have a load balancer on top of it. The first step is we that you have to tell the service discovery that hey, I'm around. Now this could happen using anything.  A decade old DNS tool. So it's pretty much your in-house root 53 replacement. So you would either register your server and say that hey, I'm around and I'm supposed to do this. This is my responsibility. This is my path. Or well, you could use something like console which effectively does the same thing internally. Now, beyond this point, there'll be a constant exchange between the load balancer and the notification server. It will keep saying, are you healthy? Are you alive? Now the replicated load balancer itself is something that you have to scale up and you have to maintain the uptime of it. So this is any of the ops guys out here. You would know that I mean, they're all of their tools to actually handle this, but this is separate infrastructure that you have to maintain for yourself. Now, if I had another notification server to this, I would run through the pretty much the same process which would always be listened to. And in case notifications over to diet and have to bring notifications over three, I'll run through the same process again. And now I have a call the service which can extend data with the load balancer. But oh wait, how do we get the IP address? I still have to pass that on or hard coded in the application. And that address could change in case it's replicated load balancer. Now, how would you use this using Gilmer? This we realize that is actually one of the biggest pain points for not writing that big an architecture. Like if you're not running at a scale which really requires that many discovery problems, why do it? So in Gilmer, since everything is a topic exchange, let's redo the same problem. There's a notification server which comes through a topic which is a manager dot notification. You can call this anything that you want to. Now I can have more of such manager dot notifications. And if I have a now caller service which wants to send data to a manager dot notification could be any of the patterns could be a request response could be a signal slot either doesn't matter here. It still has to be routed to all of the servers. So this message would be sent to all of them. Now this is important part because when I introduce groups back then I'll explain how this actually works. Now the message would be sent to each of the servers out there. And once it is sent the next step that they do is they try to acquire a lock. It's a distributed lock here and each of the service now would want that lock. The lock identifier is the group that we define. So the previous one it was push message lock service. That's the lock identification. But the lock shall only be provided to one of the services here. So your server notification server one acquires that lock and processes that message for you. This guarantees that now you're not going to process this message over and over again for three times over where it should only be done once. So what happened here if you realize there was no service discovery because they were topics. So when you are sending the data you actually didn't care who you are sending it to. There is just a function call or a path that you care about you don't care about which server is going to process it because you don't need to know that. There is no querying of the discovery process. Effectively what you're saying is that I'm going to call a service and I'm not going to call an individual server because so this is like you know if you have a wallet if I ask you do you have money in it. If you have 500 bucks the answer would be yes or no. You don't particularly care about whether I have that particular note in there or not of a number. It's only a denomination that you care about. So servers are volatile. I mean they're here today and not there tomorrow. They're going to crash. They're going to be replaced. And the service is something that you utmost care about. You don't care about whether your payment server is service is able to run 100 servers. You don't care whether your payment service can handle payments or not. No load balancing either as long as there is capacity you will serve it. If a message has been delivered to one and all the fittest node whosoever acquires the lock first well is Darwin here. You don't really have to fight it whosoever takes a lock process that you don't care. Now another thing important about most of our distributed systems is errors. Errors happen all the time. You must have an ability to detect when an error has happened and what to do with it. You can't really just ignore them. So Gilmer provides you on a framework level multiple ways of handling these errors. The most important error that you care about is actually a timeout, a server side timeout or a client side timeout. A server side timeout is whenever you register a slot or a request handler, you say that I'm only going to serve this request for say 10 minutes. Imagine if it's a video encoding going on. You want to do it for a certain amount of time, but if it fails, you don't want to waste any further resources. A client side timeout is imagine you're making a payment and you expect a response within minute or so. Whatever the timeout is, which is comfortable for you. And if either of the timeouts are not met, then the request fails. The one other error detection that is really critical to your system is confirm subscriber. If I'm sending a request right now, since because there is no server notion here, so you don't know whether the server is actually going to process it or not, is a confirm subscriber. So while sending out a request, if you mentioned that it's a switch where you say, I want confirm subscribers to be true. So the library itself would actually raise an error of four or four in your application code in case none of the services out there were willing to listen to this. Now just going through a sample of how this error would work. Imagine the signal slot example where the signal was sent and the slots expected an error. They would just forward it to an error reporter. The error reporter is a small component which is built into the part of Gilmer itself. I'll explain what that exactly does. And now think about the service as well. Request response pattern. In case it's a wrong number, it forwards it to an error reporter. Now, every error should carry as much information as possible for you. And once you use Gilmer, irrespective of what language you use it in, the error structure would look like this. It carries a topic on which the request was sent, a request data which is what the request had, user data, in case you want anything extra to be sent out. Sender is that particular ID which is unique to every message. A timestamp back trace is an optional thing which might be available in case if there's a trace back available. Although if you use Gilmer, we always give you that. And then there's a code. Now, once this is sent to an error reporter, what can you do with your errors? Now, in a system, these are primarily the patterns that we have seen that you either ignore your errors. How many of you actually believe in that? Okay, no one. So, or you publish your errors. The publishing of errors or a queuing of errors. This is unique to a system. You could do it either way. So, once this is done, so each of these message, once you start a Gilmer server in your application, you can set the error policy on which error policy is it going to use. If you use publish, we use the same mechanism which is every error message is sent out to a signal to Gilmer.error. Now, this is where the interesting part comes in. Now, in your application architecture itself, if you're using this, you would have an error receiver which listens to which is basically a slot on Gilmer.error and every error that is actually sent out is received by Gilmer.error. And just to show you a demo of how we do this in production, this is, can anyone identify what this is a screenshot of? Do you guys use pager duty? Anyone? Okay, yeah. So, this is just a screenshot of pager duty. So, we have a component which is listening on Gilmer.error all the time and any error that comes in is instantly paged out to whosoever is responsible for taking that action on that particular service. Now, monitoring. Monitoring is another really, I would say, an interesting topic in a distributed system is where traditionally we talk about server monitoring, where we have log forwarding and aggregation. Every log that is forwarded, we'd like to mine it later and draw patterns out of it and see if there's an anomaly there. If you use paper trail or log stash or any such thing, loggly as well, these services, you can draw grip, regex-based patterns and then raise tickets out of it. Error reporting, all errors that you have in your server, I mean, these are basic de-factors of any server-based system that you have. All servers should religiously have this. So, I'm not going to touch points on those because pretty much everyone talks about them. From a service standpoint, what do you care about? As an architect, I care about only are my services running right now? Are my services able of handling capacity? Do my services respond well in time? This is where Gilmer gives you a component called health bulletin. This bulletin is written in Ruby. So, it's a separate process that you run in your system. What it does is, every server of yours which is actually registering slots or request via Gilmer generates an ident for it. It's a UUID and that message is sent out to the health bulletin. Now, the health bulletin sends a message back every whatever minutes we are now configured to and it sees whether it's alive or not. Now, this is a part that is to replace whether or not, I mean, this is a part to actually replace the server part monitoring whether this particular piece of server is alive or not. This is important from a system administrator point of view. Now, if we go one step higher, what do an architect cares? Is my service listening to all the topics? Is my service able to serve well in time right now? And this is what also health bulletin does. So, you define, these are the crucial services in my system. So, request item.buy.420 could be one example. Then you say item.buy.500, item.purchase.anything. These are crucial services that should never go down. Health bulletin will actually look for any active subscribers for that particular topic and if it's not there, it will raise an error. Now, the error is also done through Gilmer.error.  We use the same pattern everywhere. Every log message that Gilmer processes is sent out on Gilmer.log. A Gilmer.log listener emits messages like this. This is a message from paper trail. And if you look at it, that's the topic that I was listening to. The first one is a host ident if we really care about what the host was. So, it's back in manager0.datascale.io. Then the topic is Gilmer.health and then every message that is coming out is carrying a unique message ID. Now, all messages that you emit in a particular scope of request carried a unique ID. This is very crucial because now in a distributed logging system you do not know what request came where and what belongs to what. The unique identification helps you club all those messages together. Now, beyond this, if you follow a UNIX principle and we believe mostly in following the UNIX philosophy is very obediently and one of the policies is that every time you define a service, think of service like a UNIX command line. All these command lines can be clubbed together beautifully because they extend data over plain text. Now you know why you guys hate system D because it doesn't do that. And using composition, a command should be able to send data to another command or call other commands in parallel or pipe it. Let's take a sample example of what Gilmer allows you to do here. So, you can have a service 1 which basically pipes the output to service 2 which basically pipes the output to another composition. A composition could be any of these things. I mean, people who are familiar with bash would actually do this in the sleep as well. This is basic tools for any bash based guy. You could have an and the or and the parallel being in a really an exception here. So how do we translate that to a software? Let's take a sample example here. This is what a composition would look like. So what I'm doing here is look at the middle line. It's a new pipe. The first pipe is basically it fetches example dot fetch is a topic. The output of that is sent to example dot words. I'll give you a working demo of this. Basically, when we're talking about distributed system, it would be unfair if you don't have a map reduce problem with does a word count. Now, the pipe output goes to example dot words where the pipe output goes to example dot count. The pipe is sent to popular. A popular is another composition up there, which is basically running two services in parallel, which is find the most popular three letter words and most popular four letter words. All of this working. What you do is you create a new data, a set data, which is a URL of the file and you just say message batch dot execute. So any of these compositions could be restructured like this. Think of it like Lego blocks here. Now, if I was to alter this and how scalability comes to the picture and flexibility comes to the picture again, if tomorrow I have to change this where I say I'm going to add stop filter because the words that are coming out are usually carry a and the those sort of things. Now I just add a new service here, which is composition example dot stop filter and in a parallel and add a new parallel service, which is popular five. So the same code. Now I can actually do much more by just probably three lines of code change here. This is another example, which I'll do a demo later. This basically what it does is it fetches the weather data for a particular city for the past one year. Then it groups it up by every month. The next up is it creates a lambda function which pulls out the weather data for the month of January and then in parallel it finds weather dot main weather dot max main and maximum temperature in that particular month for that particular city. It will give you the hour as well for that. Yeah, this is another composition and if you see is a new parallel, which is actually trying to do it for the month of January and February in parallel. Well, now let's just see a few working demos of this. It's okay if you cannot read this part because this one is not really that important. There's just a basic demo of a Echo server and a client on the left hand side. I'm starting one server, which basically is an Echo server. Then I start another one in another team accession on the right hand side. I have a client where I start sending and request. These are basically sending a request at every one millisecond and both the servers handle this in parallel because this is where the locking comes into the picture that both of them won't do it. I could have more servers on the left hand side, which basically could be 10 or hundreds of them and only one of them would do it. Now the sample text example that I showed earlier, what does the problem statement look like? So I'm going to give a S3 URL, which has a lot of random text, trust me a lot. And the goal is to eliminate stop words from that file and find the most popular three, four and five letter words. And the idea is to actually do this in parallel. And the output should be with the three letter words, the popular five letter words and the four letter words. So this is a working demo of it on the left hand side. I have a server again, which basically this. So this is all done and go, but you have Ruby examples as well. If you go to the GitHub repository, then I have another one. Now on the right hand side, bottom right hand side, I'm starting a log server, which is basically just a slot receiver for Gilmore.log. Now, once I'm sending in the first message, basically what it says is it fetches the file. After the file has been faced, it sends it to a service. One of the service has a word count in it, it groups it, then it eliminates the stop words and eventually the output, which is the most popular three letter words in this file were way and for the four letter words, text and copy and then blind and blind. Now let's take a bigger example of weather data that I talked about a little while back. So this is given that you have weather data for of this form. The form is basically a JSON hash of day, month, hour and degrees. Degrees are in Fahrenheit. Could have been Celsius as well, but I just opted to take it Fahrenheit because it shows a varied degree of temperature difference. Celsius doesn't do much justice there. Then the goal is for the months of Jan and Feb, I'm going to find maximum and minimum temperature recorded. And if possible, I'll do the tasks in parallel. The output should pretty much look like this, this temperature of the dates and here's a demo of it. I think this should be much more readable. Same thing, left hand side, two servers, right hand side is a log client, which is just receiving the log messages. Now it pretty much does the same job here. It basically sends it on two servers, computes the data and fixes the output, which is on the top right hand side. So if I do a quick recap now of what Gilmer does, it's basically a library. It's not an external process. So if you don't have a problem which requires the scale of Kafka, this is where it fits in. There is no load balance needed because you don't need one. There's no service discovery because we do topic exchanges. There's no SCTP, there's no server endpoint to be remembered. It supports broadcast and wildcard topics for the function points, scalability, exclusion groups to ensure that a message is only processed once within a group. Synchronous request reply pattern, which is the traditional SCTP model, error and health monitoring through an external plugin called health bulletin. This is what you can configure is available on the GitHub documentations there as well. Failure detection through timeouts and confirm subscribers, compositions. This is the real microservices as we say where everything should be treated like a function. Now the good part is that all these microservices, you don't know what language they are written in. You are oblivious to that. One of them could be written in Ruby. The other one could have been in Python like the log messages receiver or all in Ruby. This is a message persistence. It's just a transport that you care about. Credits, data scale.io, that's where I work and we use heavily in production. We are a Cassandra hosted and managed Kafka and Spark Shop. So all of this is done using Gilmer. I can assure you that all the error reportings, management, everything around it. The repository, you could go to github.com, check out Gilmer Libs. Yeah, that brings a conclusion to my talk here. If you have any questions, I'm open for them. Yes, please. So I had some questions around the composition bit. Yeah, please go ahead. I was wondering how so you can basically design workflows using composition. Correct. So I was I wanted to know how you would handle any errors that happens in the middle of something because I cannot do timeouts because in my use case, at least I can either run two calculation engines or a hundred thousand and that would take a few days. So I can't really do it and I don't do it using requests. I do it using signals. So I was wondering how the rollback strategy would be and the error handling strategy would be right. Good that you actually brought this up because I forgot to cover it in my talk. So the compositions are reusable. Now if a composition errors out at any point of time, whatever is left in the composition can be executed again. Now it's an interface. So a composition has executed to it. So whatever. So let's say if you had a composition, which is add, then subtract and then multiply. Now if it failed at add because you pass in a string or whatever reason, whatever is left of it, you can re execute that. And or maybe you can pass it to another composition as well. And the at what case the error happens is could be value in your case. It cannot be a timeout. So you don't let a timeout be there because in that case the job will run infinitely. If you can afford that much, you let it be that way. But in case there was a 500 internally in one of the services, you would be able to reuse that composition very easily. And what we are working on next is being able to send composition on the wire. So if you use spark, you have a spark and you have a fire and forget kind of thing. So you should be able to do that. So in case the caller, which is the context right now dies, the composition itself pretty much become meaningless. So how we are solving this is you should be able to send the composition itself on the wire. Of course, in that case the lambdas will not work because there's no way to serialize or pickle a function. And I mean, if even if there is, it's a it's a challenging problem and no worth going there. So it will raise an error that your composition has a lambda here. So this cannot be sent over the wire, but anything else can actually be Jason serialized and sent over another service to run. And in case it's a really long running job, which fails eventually, one of them will be able to pick it up and continue from there. Like all your services that are deployed. Does that answer your question? Yeah, yeah, pretty much. So just to confirm, you can send compositions as Jason over. So when you say composition dot execute, you would say remote equal to true and Gilmer internally will actually send it over the wire by serializing it over Jason. We have a notation for that we're working on it. Thank you. Hi. Yeah, I can see you now. I have a question regarding the operations of operation side of it. I was just going to thinking if you can you come a little closer. Yeah. Yeah. So the operation side of it, wouldn't your pub sub service be a single point of failure in this case? So yeah, I mean, it's a fairly interesting point that you've raised here. We use Redis right now as a back end. If you check out the source code, the back ends are swappable. You can pretty much use any back end. But read is why I think it's one of the most resilient and the best transport layers around present out there, which and using Sentinel, you can actually go cross geography as well. So the stats are out there on the readers pages on the level of messages that readers can handle. And it's really tremendous. I'm not saying that this was this is something that actually beats Kafka scale. And I specifically mentioned that if you have that sort of scale, probably you would not use this or you would use this on top of a Kafka back end itself because Kafka would not support wildcard topics, et cetera, stuff like that. But yes, fair point. I would say where your scale and if readers, I doubt though that readers in a Sentinel mode will not be able to handle the sort of load that we're talking about. What's the sort of load that we're talking about? I'm just thinking up front. Okay. All right. So we are pretty good for a fairly large wide. Have you also looked into RabbitMQ or ActiveMQ or ZeroMQ? Correct. So the stats say that readers actually beats RabbitMQ hands down. And interestingly, the project actually started with the RabbitMQ back end, but it wasn't as scalable enough and hence we moved over to readers. But we still have it somewhere in the branch.


He does start the new year off right. Okay, we'll deal with that. Good afternoon. We'd like to call the Durham City Council meeting toward at 7.01 p.m. on Tuesday, January 3rd. And certainly want to welcome all of you that are here with us this evening and certainly wish everyone a happy new year and hopefully a good year for 2017. We could just take a moment for solid meditation, please. Thank you. I'd ask Councilman Davis if he would lead us in the pledge. This is Swarov. Madam Clerk, would you lead us in the role, please? Mayor Bell. Mayor Bell. Present. Mayor Pro Tem Cole McFadden. Present. Councilmember Davis. Present. Councilmember Johnson. Present. Councilmember Moffitt. Present. Councilmember Reese. Present. Councilmember Shul. We have one proclamation that we'd like to present this evening. I would ask the early sounds, Bell and Sheryl Thomas and Sheryl and Reese and everyone else if you would join me, please. This is a resolution that speaks to the issue of human trafficking and declares human trafficking awareness month. And it speaks to the fact that whereas the community, we care about the dignity, security, physical and emotional well-being of all of our neighbors and we recognize the right of all individuals who live their best lives and safety and freedom. It speaks to the fact that whereas human trafficking is defined by the United Nations, as recruitment, transportation, transfer, harboring or receipt of persons by improper means such as force, abduction, fraud or curation for improper purpose including forced labor or sexual exploitation. Whereas anyone can be a victim of human trafficking, regardless of gender, age, race, socioeconomic status, nationality or immigration status.  Whereas we recognize that the form of human trafficking has consequences for victims, families and society and can lead to long-term health problems, psychological distress, instability and homelessness and increased risk for future violence. Whereas Durham Crisis Response Center, transforming hope, ministries, justice and values and Salvation Army are partnering with local, state and federal law enforcement and criminal justice agencies to identify and meet the needs of human trafficking survivors and end this modern slavery in our community. Whereas we must increase awareness and action to effect change and then trafficking begins by reducing vulnerabilities and increasing opportunities for those at risk, including jobs, education and housing, promoting healthy, respectful and nonviolent relationships and building a caring, inclusive and connected community, we are strongly committed to empowering victims through collaboration among organizations and systems that confront this crisis. Whereas local government, health professionals, law enforcement, faith communities, educators and civic organizations to work together to learn more to identify and serve victims of human trafficking and to speak out about this issue so that all may live in freedom and safety. And now therefore I will rebuild the Elmira City of Durham, North Carolina to help our reclaim January 2017 as Human Trafficking Awareness Month in Durham and hereby urge all citizens to observe this month by increasing awareness of human trafficking, taking action to change the culture and working together to end the violence. What's my Hand Corp. City of Durham? This is the 3rd of January, 1917. I'm going to present this to Aurelia and then I'm sure she will introduce the others that are with us. Good evening. Thank you, Mayor Bell. Good evening, Council. My name is Charlene Riess and I am working with the Durham Crisis Response Center to develop trafficking services for human trafficking victims in Durham and I am joined by Aurelia Sands-Bell, who is our Executive Director, Cheryl Thomas, who is our Outreach Coordinator and Abby Tonalia, who is the Founder and Executive Director of Transforming Hope Ministries, which is another organization in Durham working to fight human trafficking. And we just, on behalf of all of us, I want to say thank you for taking the opportunity tonight to declare January as Human Trafficking Awareness Month. It's been a long time coming for this important issue that is important in Durham, in North Carolina, and the U.S. Many organizations have been working a long time to get here where we can speak about it publicly and people are listening. We have a lot of partners in the community, in the city, in the county. We appreciate the work of the Youth Relationship Violence Task Force that has many multidisciplinary agencies in it, as well as the Child Sexual Exploitation Protocol Task Force and Social Services, the Police Department, and then our other partners. And we ask that you all continue to do the good work that you do in Durham to find ways to create affordable housing and reduce the vulnerabilities of our citizens and our youth so that they do not end up in trafficking. Thank you. Okay, I would ask other comments by members of the council. If there are any. Okay, doc. Recognize the count? Well, I'll be bold enough. I have no singing voice, but on a unique occasion like this, I will blaze the way, if you all would join me, the mayor. Happy birthday. Happy birthday to you. Stevie Wonder version. It was clear that the better singing was coming from that side. I know, I know, I know. I appreciate it. And I was telling Eddie that we're fortunate to be on this council and have birthdays and have Councilor Davis as a part of this because he always finds a way to make very special, very, very kind and grateful words. And for that I'm very appreciative. Having said that, let's move on to any other announcements. If not, recognize the city manager for his program. Thank you, Mr. Mayor. Members of the council, good evening and happy new year to everyone. No priority items. The city attorney. Thank you, Mr. Mayor. Members of council on agenda item number one, municipal election process. The date of the proposed public hearing is January 17, 2017, not January 16th, which is what was in the memo. And that change didn't get made from the work session, but that is the Tuesday. The motion of the city attorney's priority items. It's been proper to move the second. Madam clerk, we open. Close the vote. It passes seven to zero. Likewise, the city clerk. Any priority items? No items, Mr. Mayor. In that case, we'll proceed with agenda. Consent agenda items may be approved with a single motion. If a council member of public removes an item, we'll discuss that later in the agenda. And I'll read the heading of each one of the consent agenda items. Item one is the municipal election process. Item three is the contract amendment with SunTrust Mortgage Inc. Item four is the final approval of home, community, housing, development, organizational funds to the Durham Community Land Trustees Inc. For the construction of an affordable rental housing unit located at 1207 Kent Street. Item five is parking, pay-by-phone, mobile payment services contract. Item six is construction contract award to C.T. Wilson, construction company Inc. The downtown parking garages, elevated modernization project. Item seven is interlocal agreement between the cities of Durham and Raleigh, the towns of Cary and Wake Forest, Wake County and GoTriangle for administrative distribution of the Wake County, $7 vehicle registration tax. Item eight is Eno economic development zone of Orange County amendment number two to interlocal agreement between Orange County and the city of Durham. Item nine is a bid report for November 2016. Item 10 is proposed acquisition of approximately 50.173 acres for future park project located at 632 North Hoover Road and 621 Cheek Road. Item 11 is Duke University Health Systems Office of Community and Local Government Relations City of Durham FY 2016-2017 grant. Item 12 is the City of Durham training to work grant project ordinance. Item 19 is an item that can be found on the general business agenda. I entertain a motion for the approval of the consent agenda. So moved. Second. It's been a proper move. Second. Madam Clerk, will you open the vote? It passes seven and zero. Moving to the general business agenda. Item 19, proposed acquisition of approximately 2.19 acres for future fire station. 18 located 6911, 6919 Herndon Road. Mr. Mayor, members of council, at the work session you had asked me to provide you with some information regarding a potential involuntary taking of that property if need be based on some of the concerns that you've raised about the price. I did provide a memo to you all late this morning that I hope you all have seen. I'm happy to answer any questions about that if you have concerns about the purchase price and want to explore a potential alternative mechanism for acquiring the property. You've heard the city attorney's comments, recognized first councilman Moffitt, do you have a hand? Thank you. I just want to take a moment. I have learned that I was a little intemperate in my language that I used at the work session. I should have said I want to apologize to the sellers for my characterization of the deal. And I just simply want to say that after reviewing it carefully, I had come to the conclusion that I still hold that the contract price was higher than the actual value. And I think it merits our careful consideration. Thank you. Are there other comments, questions about the item? Recognize, is that councilman Moffitt again? Yes, that was me again. Since I put this item, I've raised the issue. I wanted to at least try to explain how I got to my conclusion. And I've got, I prepared a very simple little set of examples. I think there's enough there. I've learned that the seller has some prior experience with selling lots in the area that they sold seven lots for $782,250. And so the very first little section says if you took that, you come up with a value per lot. And if you look at this as five lots that we're buying, it's a value of $558,750. That to me is close enough to the appraised value of $600,000 that it's not, I would not argue with the appraised value. And in fact, I have not ever argued with the appraised value. What I have argued with are the adjustments to the appraised value. And so I'm starting by saying, I'll give you an example. So the value to the seller, what the seller said regarding a methodology is that they would need to receive the price for the property less any incidental developmental savings from the change in use. And then on top of that, they would need to receive anything that they had to spend because of the change in use. I'm paraphrasing. So the value to the market, these are examples because I don't know the actual numbers.  If those added up to $100,000, if the appraised value is $600,000, by the time that they had made the improvements and sold the lots, they would net $500,000. Which is in keeping with the methodology that the seller has put forth. So if you, that's the second section in what I handed out. The third section says if you start with that net value and you add back to it as compensation to the them, any improvements they've already done because we backed those out to get to the net value. So if they've already graded it, if they've already put in roads and curb and gutter, well that's a sunk cost to them and they would need to recover that. And then if they do have a cost for changing the use, for example if they had a contract that they had to break and it cost them $25,000, I'm just using it as an example, you'd add that back in too and then you come up with a fair purchase price. What we know is the appraised value is $600,000. We know that they do, there was a memo from our storm water department that was on our agenda and it includes that knowledge that they are going to need sand filters for storm water on the road that they have to build and dry ponds for the lots. So there is an expense there as well as the other things. What we don't know is any improvements they already have in place, what the value of those improvements are or any cost the seller might have to us. So the question is, the contract we have in front of us tonight is $700,000. And the question is, and maybe staff can provide us with some information on the improvements that have been made to date. Any penalties that the seller may incur by selling to us and so forth. Good evening, David Fleischer, General Services Department. The seller is being forced in this case to forego building houses, which is the business that they are in and profiting there from. I think in the best of all possible worlds the seller would move forward, would choose to move forward with the deal that they had that is to build houses on this property and make a profit were it not for the city's desire to acquire this property from them. So what we were told in the work session is that they had a deal in place to sell the property to a builder, which is different than what you just described. What you just described is that the seller is planning to build houses on it. So if the seller is planning to build houses then there's an opportunity cost that they have to forego. But if they plan to sell these lots because they did the other seven lots in the area, then their total value that they're going to receive from it is when they sell the lots. The builder will then have the opportunity cost, the opportunity, the value add from building a house on it. So I'm a little confused, maybe you can clear that up for me, whether this seller is building houses or selling lots. Thank you. This seller has conveyed a great deal of their existing parcel there to a builder to build houses. What they were retaining for themselves was five lots where we are building a fire station so that they could build houses themselves there. And they are unable to do so and part of their deal to convey most of their property to the home builder included the reservation for them to be able to do a property exchange, a property swap with that builder and take back some of the property they conveyed to them so they could sell that property to us and so that builder could comply with open space requirements and I appreciate this is confusing. So that the builder could comply with open space requirements by moving part of that open space onto some of the five lots that they were proposing to build houses on. I hope that answers the question. Not precisely. Did you tell us at the work session that they had a contract that we needed to close on this by the fourth that we would lose the opportunity? They do have a contract that they conveyed the lion's share of their property to a builder on November 2nd. That contract has a hundred and fifty day clause in it to do the property exchange that would enable the seller to convey property to us. Sixty days have passed of that and if we receive city council authority to purchase this property we will have 90 days per hour contract to close and thereby get in before the termination or expiration of a hundred and fifty day period. The 90 days to close is something that we put into the contract? It is. It's a fairly standard conveyance for the city of Durham to be able to close within 90 days subsequent city council approval. Could it be 80 days? Could it be 80 days? Right. It could. So the January 4th date that we were given is actually not fixed in stone. It is fixed in stone on the agreement that we made with the seller. That the seller's option to the city to purchase the property expires on January 4th. That was an extension from the prior date of December 20th. When you all had the property appraised, did the appraiser, all of this was made known to the appraiser? Is that correct? That is correct. The future use of the property, highest and best use? That is correct. The appraiser provided two reports, one that valued the raw land of 2.19 acres and another that valued the sale ability of five lots in that neighborhood. And that was the one that came to $600,000. Correct. Right. And then have you provided to council, we've been talking about this now for several weeks. So the news that the seller is building lots, not selling lot, building homes not selling lots has taken me a little bit by surprise. Have you provided council with an analysis of the lost profit that you're using, what I understand you to say now is that the appraised value of the lots is $600,000 and that general services has added another $100,000 to the selling cost, to the purchase price in order to compensate the seller for a lost profit, the opportunity cost? I don't know that it's tit for tat. I believe in our original memo it spoke to the potential profit of how much of a house being in the 18 to 20% range of the sale price of homes that would be built and that was figured into it. More so than that being the differential between the $600,000 appraised value and the $700,000 purchase price, more so than that was the seller's negotiated price based on us asking them to change their plans. They've been held up or they've been working with us in good faith for the past 13 months as we've worked toward this and they've changed their plans. They've gone through expenses with land planners, with survey and they will continue to incur those costs as they move toward resubdividing the property so that they can sell 2.19 acres to the city. Okay. The question I asked originally just so that I know the answer to that. What improvements have they put in place there? The sellers have graded the lot. They have removed the existing structures and before conveyance to the city, they will deliver sewer to the rear of that lot so that we will have something to connect to. Okay. So to date they've graded and in the process of grading they've removed the structures. That is correct. Okay. All right. Thank you. So I don't know exactly where to take it from here. The news that the seller intends to build on it, as I said, has taken me by surprise. I think I would love to hear from everybody else and see where we want to go with this. Recognize Councilman Shul? Mayor Pro Tem? Is there anyone else that got Councilman Shul? Mayor Pro Tem? Councilwoman Johnson? Thank you, Mr. Mayor. I appreciate Don raising this. It's been valuable and I've been really interesting to delve into this and try to figure this out. And I really think it's been, you know, what good city council work is. But I feel like now at this point, once I've heard David's explanation, I am very concerned about the administration's recommendation, because now my understanding is that there are five lots that the, these five lots were not just to be sold to a developer with no upside, but were actually to be developed by the current owner. And so the owner is there by a foregold. And so I think, once I've heard David's explanation, I am ready to go forward in the future. And so the owner is there by a foregoing some upside, which I think would then make this a very reasonable price, given the appraisal of the other five lots and the comparative sales price, which Don has listed on his sheet here. So I'm satisfied with the administration's recommendation. I appreciate Don's work on it and think that we've probably got the right price. Thank you. I was merely going to recommend that we go with the administration's recommendation, because I don't know of another viable alternative at this point, unless the administration can share a viable alternative other than imminent domain. And I don't think that's viable. Mayor Pratim, I think, you know, two issues. One, viable alternative to negotiate a different price on this piece of property. If I'm correct, David, that's probably not a viable option based on the seller's communication to you. I spoke with David Fleischer, again, General Services Department. I did speak with the seller today and attempted to renegotiate the deal or inquire if they were willing to have some latitude in their purchase price, and they were not. Thank you. And then the other viable option is to look for another piece of property. And I think as you've indicated previously, and you may want to repeat that again for the record, that's pretty difficult. And in fact, we may in fact be willing to pay a higher price if something was available for this piece of property, because it's a better suited location for the fire station to be located. That is correct, Mr. Manager. We searched the target area as identified by the fire department, and there were no other properties that were accessible for purchase. And furthermore, this property in particular is at the epicenter of their target range, target area. I can ask Councilman Johnson and then Councilman Moffitt. Councilman Johnson. Thank you, Mr. Mayor. I just had a couple of questions that from information that's in the first memo that we got from you all. That's item number six in our agenda documents. So the city is buying two and a half buildable lots, but are impacting the other two and a half lots because of open space requirements. Could you explain a little bit more about that? So like if we're only buying the two and a half lots, what happens to the other two and a half lots? Thank you. Two and a half lots is primarily what comprises the two and a half, the 2.19 acres that we're purchasing. However, there is some property behind that, those two, behind that 2.19 acres that was part of the open space as required to meet impervious surface for the development as a whole. As part of that property is conveyed to the city, that developer, that being the home builder that purchased the lion's share of this property, is required to move their open space elsewhere and they will be moving it to the other two and a half lots just north of our property on Herndon Road, and that's where the open space will be located for this development. I think I'm still a little confused, so we are paying for five lots, but we're only getting two and a half. We're getting two and a half lots plus the land behind it that had not originally been planted as being saleable lots. Rather, it was planted as open space for the development, and now that open space is being relocated to accommodate us getting a square lot. So the city will own that land? That is correct. We're going to own the whole 2.19 acres. Excuse me, Councilman Johnson, but I think maybe I misunderstood what I said. We will not own the two and a half lots north of our property, but rather we will own two and a half lots along Herndon Road and the property behind it that coincides with our property. I hope that clarifies it. So we're only going to own two and a half lots? These lots are a size that approximately, I'm hoping to make this easier, these two and a half lots come from what is fronting Herndon Road. We are buying two and a half lots worth of width along Herndon Road and the corresponding amount of property behind it. Total 2.19 acres. Okay. Okay, thank you. My other question was on the property behind it. I don't know if it's appropriate. Thank you. My other question was on page two. One of the questions from council was whether we had discussed with the family buying three acres for 500,000 and that their response was that the property had been placed in conservancy so that it could not be subdivided sold or developed. Is that a different property that the Herndon family has placed all of their property that's remaining in conservancy and is not open to sale or breaking of that trust? Great, thank you. So this property here is the only property that's available for us. That is correct. Thank you. Thank you. Let me recognize Councilor Moffitt who else wants to speak that I'll finish out the question. I just want to be clear just so we know when we vote like what we're doing. So the five lots to reach their appraised value of $600,000 need to be improved and buildable. Is that correct? That is correct. And part of that improvement and buildability includes grading, includes sewer. That is correct. So to get to the $600,000 that they have they also would have to do on top of that water, storm water, the road that they're showing on the plan that you provided, curb and gutter, sand filters and impact fees. Is that right? It's partially correct. I spoke with the seller about that today and they depicted the road going there, not with curb and gutter but essentially a glorified driveway going from the other part of the development where the home builders buying to these five lots. It would not be a city street, it would not have curb and gutter and be so improved. Furthermore the developer phrased it to me that they still are building sewer across the creek from the main part of the development to the area of these five lots. However it's being slightly shortened that they're not delivering exactly to five lots and the value of servicing each of those five lots would be nominal. He threw out a number of approximately $1,000 per lot that he's saving. Yes, yes but to get to the $600,000 they would have to install the sewer line. The in order, the appraiser said they should be able to sell the five lots for $600,000. In order to do that they would have to install I think it's a 585 foot long sewer line, is that correct? The same sewer line roughly. They are installing that same sewer line for us, albeit slightly shorter. Alright, so I just want to say that we start with $600,000. You deduct off the money that they would not spend if they were selling to us for water stormwater and impact fees. So from that value, whatever that dollar figure is to $700,000 is the premium that we're paying. So it's not some number that $600,000 plus a sewer line, it's $600,000 minus those other improvements. To get to $700,000 is the premium. So I just want to make sure everybody understands the premium, we're not paying for the sewer line but we're paying for things that they're not delivering. That's all I have to say. Recognize Councilman Schuhl. Thanks Mr. Mayor. Councilwoman Johnson's question reminded me of some comments that I wanted to make relative to the Herndon family's land which I just wanted to say that in case members of the public don't realize that the Herndon family has made a really strong commitment to keep their family's land as green agricultural space in the midst of all the development down in South Durham in the South Point area and this is really a gift to all of us that's made by the Herndon family which is foregoing the potentially enormous profits on the sale of their very prime land and I just wanted to appreciate that as part of this discussion because it is really a very very unusual thing that they're doing and a real gift to our whole community so thank you. I want to ask one question and I'm very familiar with the area I live less than a mile from the site that we're talking about and I should have asked this a long time ago. Is the fire chief here? He's not. Okay. Tell me why this is an epicenter. Both Ferguson deputy city manager for operations so in my discussions with the fire chief his staff has a fairly robust planning operation based on calls for service reviews with the land use plans for the area and all of those sort of go into a formula we have a software package actually that determines calls for service and radiuses around certain points this is considered the epicenter because that's what all of that analysis kicked out for us this is the optimal location in terms of shortening the calls for service to the intended service area for fire station 18. This is not very far from the parkway fire station. That's correct. The major concern about the parkwood location is the access roads into and out of that center of the parkwood neighborhood. This location provides access to much more high capacity roads which gives the fire apparatus and the responding crews the ability to get to more of the area in a timely or fashion getting into and out of the parkwood station is problematic in terms of the road network that serves the parkwood station. I wouldn't buy that I mean but it is what it is so how would the station work in conjunction with the parkwood fire station? As you may be aware and I know has been discussed somewhat publicly we have had discussions with the county fire department which now operates what used to be parkwood volunteer fire department in Bethesda we're in an analysis period looking at the possibility of some sort of cooperation up to the possibility of merging those. Our understanding would be that if that were to go forward the city and county would cooperate on the development of fire station 18 and the parkwood location would go out of service. The county agrees that the location for parkwood is not optimal for serving that area and given the growth patterns in south Durham. That makes more sense to me. If in fact you want to build this fire station where it's proposed and they have the parkwood fire station which to me is 5, 10 minutes away I know 5, 10 minutes to be a long distance when you're trying to fight a fire but it makes more sense to have this fire station serve the area than the parkwood it does make sense to me to have the parkwood fire station that they are. I should have asked this question. I should have led with that. In our discussions with the county we have all presumed based on comments that they've made that the parkwood station is not in their long term plans or ours and I think we are certainly hopeful for the benefit of city and county residents that we can cooperate with the county to have one station that serves both needs. Thank you. Is anyone else before I recognize the mayor? I'm sorry. I would like to move that we accept the item as outlined. It's been properly moved. A second. Further questions? Madam clerk, will you open the vote? Close the vote. It passes 6 to 1. Council member Moffitt voting no. Thank you. Are there other items that need to be moved? If not, we are adjourned at 7 41 p.m. Thank you. Thank you.


What we are working towards is a policy. And the nature of the policy is such, if you pass the policy, one thing is it involves very heavy financial outlets. Another thing is there are many complexities in execution. It's not a simplistic policy. It's not something we'll all agree and okay, tomorrow morning all solutions are on. It's not like that. There are soil challenges, there are topographical challenges, there are cultural challenges, there are various kinds of challenges. So it is not a simple, simply executable policy. There are many, many complexities. Any government will have to invest a lot of both money and brains to make this happen. Even if you do it, let's say very aggressively we do it, then it will take ten to fifteen years to implement the policy. After that it'll take another five to ten years actually to see rise in waters in the rivers. If we implement the policy in the next ten to fifteen years' time, for sure in the next twenty to twenty-five years' time, the water in the rivers of our country will rise by minimum fifteen to twenty percent. Before we die we want to see that. So this means four to five governments would have come and gone. Many of the existing political leaders may not be alive. Many of us may not be alive. So who is going to invest in a policy like this? They will only invest in the policy if the nation speaks in one voice and says, we want this to happen. This is why the missed call, missed call is your vote. We want a minimum of thirty crore people to give missed calls. I'm saying thirty crore because this is nearly forty percent of the electorate. If forty percent of the electorate says we are for the river policy, we are a mature citizens today. As a democracy we have matured, we are not looking for freebies. If you go for something which is long-term well-being of this nation, we are with you. In one voice we have to say this, otherwise no government is going to invest their time and money in doing something which will show results twenty-five years later. This is not an election winning policy for sure. So and for all of us there will be some pain. And we have to say we are willing to go through the pain for long-term well-being of this nation. We are not irresponsible citizens who don't care what happens to future generations. We are concerned about it and we will do the necessary thing. This is the commitment you're making when you make this simple call.