So there are three steps to distribute the key the aim The aim is that a and b know the session key and that happens after step three Step four and five are just some extra checks They're called the authentication steps to check really B is checking With a did you really send this message? So B sends back message for which is has the meaning did you really send me message three or maybe someone else replayed it from before and If a really did send message three it will respond with message five If a didn't send message three it is an attack by some malicious user who replayed it That is it was sent before by it by user a but some malicious user intercepted and And then at a later time replay that Then when a receives message for it detects something's gone wrong If I didn't send message three, but I received message four then something's gone wrong And they'll take other steps to make sure that B is informed or they will not continue so that they will not use this session key similar if B does not receive message five then it assumed something's gone wrong Also, what's inside message four and five? Message four contains a nonce value Generated by B and two in this case so another for example random number a different random number B chooses a large random number Encrypts it with KS the session key Only a should be able to decrypt a malicious user should not be able to decrypt and find the nonce value and They should not be able to guess the nonce value because if it's a large random number It would take too long to guess so you very there's no no practical chance to guess it so if a malicious user did send message three and Receive message four and they try to respond with message five then they cannot send message five Because they don't know the session key and they don't know the nonce value Message five it says here f of n2 a function of n2 It can be any function a simple function like increment by one So B chooses a large random number and to check That it really did come from a then aim sends back some Modification of that large random number for example that large random number plus one if B doesn't get That correct modified number in a response then it assumes something went wrong So these are just to check that an attacker doesn't replay a message Note that the KDC doesn't send the session key direct to be in this case It doesn't need to to cut down on messages There's one message to KDC saying I want a session key to be KDC responds. Here it is. Here's the session key and A forwards on the part of that message that contains the session key that B can decrypt So as a response as a result of message three being received B has KS and Previously from message to a has KS. They have the session key When a now sends data to be so after step five everything's done. They start sending data They encrypt with what what do they encrypt their data with which key? Which key now they're sending data. What key are they going to use? They use the session key So remember the session key Session means your communication session for example, you're downloading a file from a web server So the session is that duration of that TCP connection while you're transferring data or Maybe if this is for example for voice communications the session maybe for a voice call a Session can be defined different ways, but it's usually for some Some group of communications can be multiple messages Maybe we use it today this session key and tomorrow if I want to communicate with B again I may follow these steps again and get a new session key because it's it's an advantage to Keep changing your session keys.  There's that home lock But the idea is that if you change the key that there's less opportunity for the attacker to find that key So this uses the KDC when a sends data to be using the session key to encrypt Note that KDC can see the data It's possible for the KDC to encrypt as sorry to intercept any message sent from a to be If it's encrypted with a session key the KDC can decrypt that Because it was the KDC that generated the session key the KDC knows KS so a B and the KDC know KS and Therefore we must trust the KDC not to make use of that data So that's where we say it's a trusted third party That's the notation that was used So this approach involves we need this other server type entity in the network a KDC. So let's say All the computers in s it's land all the desktop PCs at least in the office Officers in the labs. We wanted to use encryption between all of them Then we'd have some special server that acts as the KDC and There will be a master key programmed into each computer shared with the KDC and Then there will be a protocol that when one computer wants to communicate with another the first one contacts the KDC the Katie across the land the KDC sends a session key and that first computer then forwards that session key onto the Destination computer and then the two computers send their data securely between each other using the session key to encrypt So we need some server or this this KDC in the network In large networks, you can have a hierarchy of KDCs because Let's say a large network SITs network. We have two campuses We have one kitty KDC in this campus a second one in the other campus to serve the users in that campus and we can have in fact the two KDCs can Obtain and distribute master keys via a top level KDC So we can have some hierarchy of KDCs that help in distributing the keys So one for each land or building and then a central one to exchange keys between hosts in different subnets and As you get larger you can have for performance reasons multiple KDCs and having multiple KDCs also Spreads the effort around It means that one KDC doesn't become a performance bottleneck and also means if one KDC is compromised then the others are not okay compromised if If someone can get access to this KDC a Malicious user then they can find all the master keys and all the session keys that the security of our system fails So we must keep the KDC secure usually physically secure if it's a server Because that's a key point of the network so having multiple KDCs means that if for example the KDC in Runksit is compromised someone gets access to it at least it means the Communications in this campus can still be secure so long as the KDC here is secure So there's advantages of having multiple KDCs. So a hierarchy of KDCs another concept How long should we use the session key for? Every time we want a new session key we must go through this step these five steps There's some communication overhead So every time my computer wants a session key with another computer we go through these steps We send some message to the KDC get a response If I want to change session keys, then we go through these steps again So there's a trade-off here. You want to change your session key as often as possible for security because the Fewer times you use the key it gives less opportunity for the attacker to discover the key So you want to change as often as possible But the more times you change the more communications overhead of implementing this protocol So there's some trade-off there Shorter lifetime of the session key the more secure but the more overhead of the communications The more impact on performance So when in practice do you change the session key? It depends on the application on the network some examples are You change you could have a session key for each TCP connection Every time you connect use establish a TCP connection from client to server For that connection get a new session key. That's one example Or you could do it on a time basis every day get a new session key Or every five minutes get a new session key the more often the more secure So depending upon your applications and what your security requirements are you can have a scheme to change session keys Using a KDC has a big disadvantage. You must trust the KDC So it's okay in an organization. That is all the users in SIT all the employees For communicating securely inside SIT We need to be able to trust the computer center who runs the KDC. That's okay Because we all work for the same organization. We must trust each other, but What if you are in a Circumstance where you do not want to trust some central third party the two entities want to communicate securely But they don't want someone else to know their key Because again the KDC knows the session key So you can vary Modify the scheme to be a decentralized approach There's no KDC involved so a protocol for exchanging Session keys The end result A and B have a session key KS The big difference here is that for this to work Each user must exchange a master key with each other user So in this example, there's a master key KM shared between A and B manually delivered in the past and If A wants to communicate with C Using the same approach there must be some master key exchanged between them and Then they use this protocol to exchange a session key So in this case, there are many more master keys That's the problem with a decentralized approach every user Exchanges a master key using manual delivery with every other user and we get we no longer have it our end by end Minus one divided by two master keys many master keys. Whereas in Our centralized approach we need one master key for every user and master keys Whereas with decentralized we need end by end minus one over two approximately in the order of n squared master keys, so That's the problem with a decentralized approach if you don't want to use the KDC you need many master keys You may ask well if we already have a master key then why use us why Perform this protocol if a has a master key with which stuff shared with B Why do we need to do this? What's the advantage? So this scheme assumes AMB all already know KM the master key So why perform these steps? So we can change the key that we use to encrypt our data frequently So this assumes a nodes KM and B nodes KM Let's write it down This decentralized scheme assumes that AMB have already shared a master key KM Then they follow these three steps and the result is that they both have a session key AMB share a session key KS Then when they want to send data they encrypt using the session key Why not not do these three steps and just encrypt our data with the master key Well, it's this concept of we want to be able to change our keys on a regular basis Let's say We've got a Million packets to send a million pieces of data Let's put some numbers to it a million packets to send between AMB if we don't use this scheme and AMB have some master key and we encrypt All of our data using that master key Then we can think of those one million packets. We've encrypted All of them using our master key. So if let's say we've used our master key one million times Now by using this scheme AMB share a master key But then they use that to generate a session key then what they do is that they encrypt their data with the session key Let's say we use a session key for just 1,000 packets and After encrypting 1,000 packets we change session keys and then encrypt the next 1,000 packets and then change session keys again then with session key one the first one We'd encrypt 1,000 packets and then we'd repeat this these three steps and get a new session key session key to and Encrypt a thousand packets and We do that a thousand times KS 1000 why a thousand because we have a million packets in total to encrypt So just put some numbers to it so we can compare How many messages? Using this approach how many messages were encrypted with the master key? So what we do is we have a master key Every time we want a new session key. We apply this protocol One of these messages message to we encrypt using the master key You see one and Three we do not So we use the master key just one time in this case We use it One time when we every time we change session keys. We use the master key once So the total number of times we send something across the network using the master key in this example is a thousand times One for each time we change the session key The result is By using this scheme we use the master key on total one thousand times We use session key one one thousand times session key two one thousand times and so on The less times we use a key the harder it is for the attacker to try and discover that key If we didn't use session keys, we use the master key one million times By using the session key we use the master key just one thousand times in this simplistic example That is we use it much less than in this case. So that's why we'd consider this more secure So yes using the master key allows us to change session keys But the problem with this approach We must Manually deliver master keys between each pair of users. So only very useful for small networks before we move on any questions about these schemes the KDC and the decentralized approach so those approaches were about getting a session key Between A and B and then they use symmetric encryption to encrypt their data And the way that we got that session key securely was we encrypted with a master key Also using symmetric key encryption that is The E in here is using symmetric key encryption AES triple desk, whatever Let's consider a different approach The goal now is to distribute a shared secret key same as before But to do that we will use asymmetric encryption. That is public key cryptography And we'll go through three different schemes for how to do that. We'll go through two ones that just a combination look Remember that with public key cryptography or asymmetric encryption We generally don't want to use it to encrypt all of our data So the idea is for the data we send between A and B use symmetric encryption You only use asymmetric encryption when you have a small amount of information to send because it's too slow otherwise So as you saw in your assignment one way to use asymmetric encryption is to exchange secrets exchange secret keys Let's go through some examples. One of them you've seen is Diffie helman, but let's go through some other ones Here's a very simple one. A Sends its public key to B and Its identity Saying I am user A. Here is my public key and B selects or generates a session key KS and Encrypts that session key with a public key of A Sends it back to A and now B has the session key B selected it and A can decrypt this message because A has its own private key No one else can decrypt this message because no one else has the private key A and Therefore the session key has been received by A and not no one else So this is a very simple way for A and B to get this shared session key KS It provides confidentiality for the session key It relies on public key cryptography this encrypt here this E. We're using something like RSA Using the public key of A to encrypt and you know if you use if B uses A's public key to encrypt Then only a can decrypt because only a has the corresponding private key So only a can decrypt and discover KS simple See if you can attack it There's a simple what's called man in the middle attack that can be performed in this case try and work it out You're the man in the middle between A and B. See how you can defeat this system Not a meat in the middle attack that we saw with double desk. This is a man in the middle attack Give you five minutes to see if you can break this system So what you can try and think about is okay if you are C in the middle And you could intercept these messages. What could you do to fool A and B? Into thinking they have a session key that's secret see if you can defeat this system So one way to do it is start drawing a diagram where you have a C and B and send some messages There'll be a bonus for the first person who can show me the correct answer before I draw on the screen On the board come on now use your brains and start performing an attack not correct yet So think you're the or there's an entity in the middle Entity C for example, what can they do sounds on the right track draw the picture Any attempts? Any solutions? People are on the right track on the right track Hint there should be there'll be four messages at the end First person to draw on the board will get the up bonus bonus Bonus for the next quiz the next in class quiz Next or the last one if you get it wrong you get a penalty No, no, no is he correct Is he correct? Doesn't work have a look see why or what happens when this man in the middle M Sends this second message Think about what what is this achieved? So there's a question should this be IDA or IDM It should be IDA because M is pretending to be a That's the idea masquerade So let's see if it works, and I think I don't even have to draw it because I think what you've got is correct so the idea is that A and B Need to get a session key ks and their their goal is that no one else knows the session key What the man in the middle attack does is that this man see intercepts the messages and tries to do something such that see also knows the session key and A and B don't know that anything's gone wrong. That's important that A and B think everything's gone correctly Because if they detect something's gone wrong, then they will not use that session key so a sends its as shown a sends its a normal original message the public key its own public key and Its identity so that's the original message and The man in the middle then Pretends to be a and the way that it pretends to be a is sends the identity of a but Sends a fake public key so We see here the identity of a and a fake public key that is the public key of this man in the middle so I'll copy your diagram and I've just called mine C so It's still the identity of a So B thinks it comes from a but the public key has changed. I call it public key of C B uses its own public key Sorry C uses its own public key the man in the middle uses their own public key So B is received a request really according to the protocol B receives a request. This is from a Here's the public key Generate a session key encrypt it and send it back Using the normal protocol when B receives such request it encrypts with the public key it received and Sends back the encrypted session key so B responds not knowing it's from C thinking it's from a and we encrypt using The public key of C in this case this public key. We received the chosen session key KS Leave your board leave it on the board Okay so B chooses KS the session key that it's going to use with a and Encrypts it using the public key that it received from a Well, at least the public key that it thinks it received from a but in fact it sees public key What does C do C decrypts because C knows the C knows the private key So decrypts and when it decrypts it gets KS So C knows KS and Now to make sure that a and B think nothing went wrong C sends a message to a encrypting using the public key of who of A Okay, the public key of a is known. So we encrypt using the public key of a The same KS Not a different one So when a receives this he thinks oh everything's okay. I will decrypt with my private key and I'll get KS and end result a Has KS and be have has KS. That's good from a and B's perspective and Nothing's gone wrong from a and B's perspective nothing. They can detect because a sent the expected message and Received the expected reply that is some key encrypted with a's public key Because it successfully decrypts with a's private key. They trust this receive message similar be receives a request Identity in public key encrypts with the public key the session key So a and B don't know anything's gone wrong The man in the middle now knows the session key So what happens next is a has some data to send to be a encrypts that data with KS So now a takes some data encrypts it with KS the session key sends the data encrypted with KS to be B receives it But at the same time C takes a copy and decrypts it Okay, because C has KS now anything sent between a and B encrypted with KS C also receives And a and B don't know Okay Okay, so what about be receiving this message the public key of C concatenated with ID of a You in your assignment you saw they were Diffie Hillman public keys We remember in the assignment one one step was to generate your public key and there were two values that in the public key Let's see if I've got one to show you an example Whose public key can I show you? Let me find an example public key and we'll see here. So Here's a public key So when B receives this public key as well as the identity of a This is a Diffie-Hellman public key. It can be any other type of public key How do you know whose user this is if you look at this? There's no information in there that says who's it is so When you receive this public key Okay, between the begin public key and the end public key when you receive that there's no Information in there identifying who it is It's the ID that identifies who it is So there's no way to just to know is this really a public key or is it? sees public key When B receives this message it doesn't it cannot tell it trust takes the identity. Okay, since the identity is a Then let's assume that the public key is that is that of a and that's the weakness in this scheme That B cannot confirm that this public key is in fact someone else's So B does not detect anything went wrong there So in this case this says the public key of Steve, but in fact It is not my key you if you look at them. They're identical in fact Just because the file name says Steve it's someone else's public key in this case Okay, so there's no way by just looking at the public key to know who's it actually is and That's the problem with distributing public keys We need some other way to confirm that this is in fact this user's public key. It's not something else someone else saying It's a user's public key, but it's in fact pretending to be someone else's public key. So This in this case was one user Saying here's Steve's public key, but in fact it wasn't my public key It was another user's public key pretending to be me. So this simple scheme for key distribution Does not work if we can have a man in the middle Who can intercept and and generate messages? It's only useful if there's no opportunity for an attacker to intercept and modify messages So if an attacker cannot insert messages, then this scheme works fine But if an attacker can intercept modify messages, then it's insecure When may they not be able to maybe the link if the link is secure It's an optical fiber link and there's no way for them to Create new messages on that link Then it's impossible for them to attack this scheme. So it's okay in some cases, but not all across the internet Not good because here's my computer. Here's the Facebook web server All someone has to do is get access to some ISP somewhere in the middle Somewhere in the internet intercept the message and modify the message and they can attack this scheme Modification or an alternative scheme and will we go through it? I think you can study this one in your own time It's just an alternative scheme also using public keys There are others as well. Okay, I want to get on to the next topic. This one's about done Diffie helman we saw was another way to use public keys to exchange secrets So we saw that in your assignment There's not much to say about this one Maybe the only thing to point out in this one. We provide some form of authentication We encrypt with a private key and a public key encrypting with a private key. Remember is signing something So in this scheme that uses both signing and also Encryption or confidentiality Again, this one assumes that the public keys are known and correct. So we see a encrypts with the public key of B Of course a must know the public key of B in this case So there's some assumption about what's known prior for this work Other ways for distributing symmetric keys is combined with a KDC so so you can use a Combination of the KDC for secret key distribution and also the public key distribution So we saw with the KDC. We assume the master keys were distributed manually on alternative and a foot an alternative to that is to use Public key encryption So to distribute the master keys use public key encryption using one of the schemes that we've seen And we'll see later some issues with Key distribution of public keys So the mainly that simple one and the man of the man in the middle attack is what I wanted to cover on on those alternatives Let's introduce the next topic and then show a few examples and then next week we'll finish on the next topic This is now different the previous approaches were about getting a shared secret to the other side Now we have this problem of how do we get public keys to the other person and we just saw a problem. We saw the problem that user B Received a message with the public key and they didn't know whose public key it was They just assume since it came with idea. It must be a's public key and that was the floor in this Key distribution scheme That's why it didn't work because be received a public key and there was no net way to be sure that it was in fact A's public key or M's public key So distributing public keys is not easy Because anyone can send out a public key and say this is Steve's public key So we need some extra mechanisms We'll go through and look at some general approaches and then some specific protocols for doing it So by design public keys are made public everyone can know them But we have this issue how do we ensure that a public key of a actually belongs to a and not someone Pretending to be a that's what we had the problem here different approaches and some of them are related public announcement post your public key on a website on a bulletin board or Print it out and put a stick it on your door of the office, whatever, but somehow announce your public key Or have some directory Some server that stores public keys and when you want someone else's public key You contact that directory that server and it gives you the public key of another user And then we'll see some some protocols related to those two public announcements so Okay, at the end of your email you include your public key the like the value that I showed for your Diffie helman public key You include This text at the end of your email or you include it on a web page or in some forum and then people know your public key It works except as we know How do we be sure that the public key being announced is actually for that user? It's not someone pretending to be that user Well Okay, you may say if you receive an email from Steve at SIT to you a CTH and it's got a public key Then you know it must be my public key, but I think we saw in one of the Earlier lectures that it's very easy to fake an email address so you can receive an email that looks like it's from Steve at SIT, but it's actually from some student and It's got a public key. You think it steves, but it's in fact someone else's so We need some way to To be sure that it's a user's public key Just making public announces announcements Not enough you could have some form of directory to automate this process You have a server and again each user Who has a public key publishes this their Key in some server and when you want another user's public key you contact that server and download their key Now this will work if If the users when they publish their key provide some form of identification Okay, let's say this is some special web server It's run by me and all the students need to provide their public key So what you do is you come to my office. You show me your student ID. I check your photo Okay, I'm I confirm this is actually you and this is your correct public public key So if there's some identification when the key is published in the directory Then when someone receives the key, they assure that it is the correct key So we need some way to ID the correct users Of course, we need to make sure that the directory cannot be compromised that someone could not put in a public key that is a fake or for some other user How can we automate this process? What protocol can we use to? Publish and obtain public keys of other users That's what we're going to cover in these two slides. We'll just introduce I think one of them So here's a way that we have users that want to communicate and To communicate they need to know each other's public key so B wants to know the public key of a and Similar a wants to know the public key of B in this case. We have this this directory. It's here. It's called a public key authority It's a case of publicly available directory and This protocol assumes that user a has published their key with the authority So the authority has a list of keys and then the protocol shows how do we obtain other users keys? Let's go through it So the assumption before any of these step steps happen is that a has gone to the authority and saying I am user a and here Is my public key? So let's take a record of who knows which keys at the start a knows its own public key and its own All right. We use lowercase a and B A knows its own public key and private key Every user knows their own key pair Sam is user B B knows PUB and PRB and Also, the authority has a key pair. So everyone has their own key pair in this protocol and Also, we should So we assume that each user has published their public key at the authority What that means is that a has gone to the authority and said here is my public key and We trust that so there's been some form of identification taking place there So the authority also knows PUA and PUB and Finally when you go to the authority and say here's my public key at the same time the authority Gives you their public key So user a knows the public key of the authority and so does user B This is the assumption at the start of this scheme The aim is for a and B to get each other's public key and to do it securely so they know for sure it is when B receives a public key that this is a is public key and Same for user a and it's quite simple to get started what happens is let's say a wants to communicate with B So a will need to know B's public key So a sends a request to the authority saying I want B's public key some form of request and Here we have some new notation t1 is a timestamp Similar to our nonce before it's used to identify this request a Different request at a different time would have a different value of the timestamp So a sends a request saying I need to know The public key of B so inside this request would be identifying a and B The authority looks for the public key of B and and Sends it back to a but importantly The public key of B, which comes back to a is signed by the authority When I say signed I mean encrypted with the private key of the authority Recall when we encrypt with a private key Anyone can decrypt, but it's a confirmation that it came from that user So a request B's public key in step one in step two The public key authority sends back B's public key because the authority has it it's stored in the directory and Importantly they sign it using their private key When a receives message to they verify the signature How do they verify it? How does user a verify the message to? using the Using the public key of the authority good Because it's encrypted with the private key of the authority that is signed by the authority when we receive it we decrypt So a decrypts using the public key of the authority which they already have and they trust And if it successfully decrypts that it means it must have been Encrypted with a private key of the authority which means it must have came from the authority It didn't come from someone pretending to be of the authority That's the purpose of this signature It's not from some malicious user saying here's the public key of B well We must trust that the message we receive in fact is the public key of B and we do that using the signature a Has now the public key of B and they know it's B's public key because they it was signed by the authority And they trust the authority The next step step three Because a wants to communicate with B. They send a message to be Confidential message encrypted with these public key Saying I am user a and I want to communicate with you and then B does similar to what a did requests a's public key and The authority sends B the public key of a So steps four and five Almost identical to steps one and two Except from B to the authority instead of a to the authority and after step five B knows the public key of a so after step after step two a Knows the public key of B and after step five B knows the public key of a They trust them because they were signed by the authority and we trust the authority in the system and Similar to what we saw in the KDC we have some final messages to prevent replay attacks some final authentication we see again sending nonce values and Responding with nonce values just to confirm this message actually came from you. It's not someone else replaying that message so this is a way for using an authority to Discover other users public key and be sure that That public key is of the correct user 15 minutes to go. We're okay Any questions on how we use of an authority? So think of this for example I'm the authority You come to me You show me your ID you show me your public key I store your public key and every student does that I store every student's public key So that happens at the start and at the same time when you come to my office. I also give you my Public key. I'm the authority So every student knows their own key pair and the authorities public key. That's the setup of the system Now when two students want to communicate They need each other's public key and Because you don't trust other students what you do is you contact me first you say I want student bees public key I Give you student bees public key and importantly. I signed that message. So you're sure it came from me You contact the other student Saying I want to communicate with you that triggers them to contact me the authority asking for user a's public key and I Give the student user a's public key signed by me and because you trust me Anything signed by me you trust the contents. What we'll do next week is go through an alternative provide similar features, but it performs better That is it's more practical practical to use in a large network and it's Certificates and this is what's used in the internet when you access secure web servers a digital certificate So we'll spend some time going through how we use a certificate. What is a certificate? what do they look like and What your web browser? How they store them and manage certificates? Let's do that next week We'll stop there and What you'll do What you'll do is So that you can be up to speed next week so we can go straight into certificates Make sure that you look at those these diagrams that we've gone through for KDC through to decentralized and finishing with this one so Given a quiz or given a exam question, which normally gives you such a diagram You're not expected to remember these diagrams, but given one such a diagram answer questions about what's happening What can go wrong so understand the notation and what type of attacks may or may not be possible? Let's continue that next week enough for today


Hello and welcome. My name is Shannon Kemp and I'm the executive editor of Data Diversity. We'd like to thank you for joining the current installment of the Monthly Data Diversity Webinar Series, Real World Data Governance with Bob Siner. Today, Bob will discuss using agile to justify data governance. Just a couple of points to get us started. Due to the large number of people that attend these sessions, he will be muted during the webinar. If you'd like to chat with us or with each other, we certainly encourage you to do so. Just click the chat icon in the upper right corner for that feature. For questions, we will be collecting them via the Q&A in the bottom right-hand corner of your screen. Or if you'd like to tweet, we encourage you to share our highlights or questions via Twitter using hashtag RWDG, Real World Data Governance. As always, we will send a follow-up within two business days containing links to the slides, the recording of the session, and additional information requested throughout the webinar. Now let me introduce to you our speaker for today, Bob Siner. Bob is the president and principal of KIK Consulting and Educational Services and the publisher of the data administration newsletter, TDAN.com. Bob has been a recipient of the Dana Professional Award for significant and demonstrable contributions to the data management industry. Bob specializes in non-invasive data governance, data stewardship, and metadata management solutions. And with that, I will give the floor to Bob to get today's webinar started. The last webinar of the year for Data Receive. I was just going to say that, right? It's an honor and a privilege to be the last webinar. There are so many webinars that Dataverse produces, and I'm glad to be a part of it. And also, in a minute, I'm going to share with you what the first couple of upcoming webinars of the Real World Data Governance series are in 2017. Yeah, it's our 89th webinar of the year. This is the 89th webinar. And so this webinar series has been going on a while, and I am really glad that we've got so much participation, and it's really great to see so many people that come time and time again. And I also welcome all of you new folks as well. So, again, as I always do, I want to thank you for your valuable time in attending the webinar, and hopefully it will be valuable to you. So as Shannon mentioned, the name of this webinar is Using Agile to Justify Data Governance. And over the years, we've done several webinars on Agile and Data Governance. So this was a good angle to take for this webinar, talking about how we can use Agile to justify data governance and how we can bring the two together and show the relationships and make sure that we're addressing the audiences that need to know that not only is Agile important, but data governance is important as well. So as I mentioned, I wanted to outline for you what the next three webinars are for January, February, and March. We'll be talking about three specific and unique approaches to data stewardship. In February, it'll be the opportunity to talk about my new non-invasive data governance framework, and then in March, we're going to talk about applying data governance to Agile efforts. So it's really kind of a part two to what we're talking about today. So there's some specific items that we're going to walk through today that are in the agenda. I'll go through that in a minute, but just to let you know that the Real World Data Governance series takes place on the third Thursday. The month at 2 p.m. eastern time, and that is where we all are today. So again, welcome today to the webinar. Before I get started, oftentimes I just kind of run through a couple things that I'm involved in. Several of them involved with Dataversity. First is, as Shannon had mentioned, I talk a lot about non-invasive data governance, and just to let you know that there is a book called Non-Invasive Data Governance that's available. Also, if you're looking for more information on non-invasive data governance, you can go to kikconsulting.com. I mentioned that I work with and partner with Dataversity quite a bit. So there's a couple different events that I wanted to bring to your attention if you're not familiar with them. But the next one that's coming up is in January, and it's the second Enterprise Data Governance Online event. And again, I have the honor and the privilege of kicking off that online conference. So please go to Dataversity and register, and I hope to have you there. I'll be talking about selecting the right approach to data governance in that conference. And then there's Enterprise Data World that you're probably all familiar with, but if you're not, please go out to the Dataversity site and learn all about Enterprise Data World. I'll be speaking at that conference, and I'll be talking about comparing approaches to data governance. So there's a bunch of different approaches. There's non-invasive. There's traditional. There's command and control. If you're interested in learning the differences between those, please stop by at either of those events to learn more. A couple other real quick things. One is the Data Administration newsletter, the online publication that is produced by Dataversity, and then I'm the publisher of. The content is now being released twice a month. And if you're not familiar with TDAN.com, please go out there. There's lots of great articles on data governance, data modeling, anything pertaining to data administration and data management. And last but not least, I wanted to introduce to you the new online training series that we have through Dataversity called Non-Invasive Data Governance. So if you're looking for online training, please go to the Dataversity Training Center and learn more about non-invasive data governance and all the other great courses that are focusing on data management and data practices. So let's start talking about what we're going to talk about today. So the abstract that we used to present what we were going to talk about today is the such. We all know that the agile development methodology is here to stay. Many organizations are embracing agile to different levels and to different degrees, but agile is here to stay. And we also know that data governance is not going to go away anytime soon. The need to manage data as an asset of the organization is going to be important at the same time that we're being expected to deliver major projects for our organization in an agile and iterative way and be very quick and successful in the implementation of our project. So we know that agile is here. We know that data governance is here. We've got to look for ways to bring agile and data governance together. We want to look at the two disciplines and they share some common ground. And I'm going to talk about that in the webinar today. Share with you a bunch of the major principles of agile development and delivery of systems, but also share with you the aspects of it where I believe that we as data governance practitioners can really have some impact on these agile initiatives. So the disciplines really need to find ways to work together. And so what we're going to talk about today is using agile to justify data governance within the projects. So if you've seen the abstract, what I did do is I took the liberty of just kind of rearranging a couple of the items, but I kept the items the same, but they're just in a little bit different order. I'm going to talk about the audiences for both agile and data governance. We'll talk about the governance aspects of agile and why data governance practitioners should embrace agile while at the same time looking at what can we provide to those agile practitioners to help them to understand why data governance is important and where it fits into their project schedules. So we'll talk about agile considerations for data governance and then we'll talk about how to use agile to justify data governance. So that's our agenda for today. For those of you that are new to this webinar series, I oftentimes start out, not all the time, but oftentimes start out with definitions that I use for key terms. And so I want to define data governance for you. I want to define data stewardship for you. There's a lot of different definitions of data governance out there. Depending on who you ask, they go anywhere from the orchestration of people in process and data, the harmonization of people in process and data. I like to have a definition of data governance that has more teeth to it. So I say that data governance is the execution and enforcement of authority over the management of data. We want to apply governance in a non-invasive way, and I'll talk a little bit more about that in a minute, but the goal at the end of the day is to be able to execute and enforce authority over data, whether it's the management of the data aspects of large projects, whether it's data protection, the improvement of data understanding or data quality. At the end of the day, the idea is that we want to be able to execute and enforce authority over the management of data. And my definition that I use for data stewardship is more in line with the whole idea of being non-invasive in the approach, which is that we've already got people in the organization that have a relationship to data, that relationship may be through defining data, producing data, using data. But one of the things that we need to do is we need to formalize the levels of accountability that are associated with those relationships to the data. So again, the definition that I use for data stewardship is that it's the formalization of accountability over the management of data and data-related resources rather than handing it to people as something that's brand new to them. Typically, users of data need to understand the rules associated with using the data. People who are defining data need to know the rules associated with defining the data. So let's try to stay as non-invasive as we can in our approach to implementing governance. So the last definition I wanted to share with you is that of non-invasive data governance, which basically plays to everything that I just talked about. The practice of applying that formal accountability and behavior through non-invasive roles and responsibilities, and we've done webinars on those roles and responsibilities, we're going to apply governance to processes rather than redefining or defining everything in our organization as a data governance process. And we're going to assure that the definition of production and usage of data does all the things that we want out of our data governance program, which would be to assure regulatory compliance, security, privacy, protection, and quality. So really, non-invasive describes the approach of how we're going to apply governance in the organization to be in a non-threatening way. And again, a lot of organizations look at the non-invasive approach and say, this is something that we need to embrace because there are already people in the organization that have responsibility for data. We want to be transparent, supportive, and collaborative wherever we can when we're implementing governance across the organization. A couple other quick definitions, and these aren't really definitions, but it's kind of descriptions of the agile manifesto. And as it's defined, it's a development methodology that is basically set up to lead to good data design. And there's five primary things that I found in the work that I've done with organizations that are taking the agile approach. Five primary items that they focus on are getting the right people involved in the interaction between the people, being speedy with the delivery and being effective at the same time, so being iterative in the approach, delivering working software products at every step of the way, making sure that we do requirements and that we do collaboration. And one of the things that you'll find when I speak about this a little bit later in the webinar is that we've got to talk about what level of requirements we as data practitioners, data governance practitioners require of the agile folks to make sure that the data that results from the implementation of the agile project is successful and that it covers all the needs of the business folks within our organization, the last one being responsive and getting the right people involved at the right time, again, evolving the solution to what it needs to be. And this graphic is one that I found repeated a bunch of different places. It talks about the agile approach where we develop functionality, we integrate and we test it, we develop some more functionality, we integrate and test it. We go through those iterations numerous times before we release a piece of software to people, get the feedback from them, and then continue to do these iterative development and testing and implementing steps. So the agile process basically goes iteration by iteration and it moves quickly in the development of the projects that we are focusing the agile approach on. The opposite, or should I say, the other side of the coin when we're looking at the agile development methodology is the waterfall approach. And many of us traditional data people over the years are very used to the waterfall approach, which is a very linear and sequential approach to developing software, but it's a very business requirement laden. We need to make certain that we're involving the right people, like agile wants to involve the right people, we want to make sure that we design the business requirements, that we define the process, we lay out exactly what we're doing and we have a very structured manner to how we're delivering projects. Oftentimes the folks that embrace the agile approach believe that the waterfall approach, this traditional approach is something that's of the past and that we're moving to a world where we need to be more agile, we need to be able to adopt and adapt to those things that are going on around us. So the waterfall approach is in some organizations following by the wayside, as organizations focus their agile efforts on their major initiatives, things like ERP transformations, redevelopment of data warehouses, implementations of large data integrated projects across the organization. Those are the efforts that organizations are attempting to use agile on in order to deliver them quickly and timely and get them to be the best that they can possibly be. So the first thing that I wanted to talk about is the audiences of both agile data governance and the fact is that our senior leadership in a lot of our organizations are telling us that we need to govern our data and they're also saying that we need to deliver projects quickly and we need to deliver them effectively in an equality manner. So what does senior leadership want from data governance? They want to make sure that the data is understood and that we have good governed data definition. They want to make sure that the data that results from these projects is good and that we have governed production of data across the organization. And they also want to make sure that the data is protected and that we are using the data the way the data is intended to be used but we also understand that we have people in our organization that understand the rules associated with handling data that has to be protected certain ways. So senior leadership typically wants these things. They want the data to be understood. They want it to be of high quality and they want it to be protected at the same time. That's typically what senior leadership is looking for from data governance initiatives. So what does senior leadership want from agile, from agile development methods as well? And if you'll notice the guy in the middle of the screen, if you're as old as I am and you remember Mr. Rogers, Mr. Rogers had a guy named Mr. McFeely who was the speedy delivery guy. He was the guy that was always delivering packages, delivering them very speedy. Just as a quick side note, a lot of people talk about growing up in Mr. Rogers' neighborhood. Well, I literally grew up in Mr. Rogers' neighborhood in Pittsburgh, Pennsylvania. In fact, I've delivered his newspaper at a time. I didn't run through his door and say speedy delivery. But the fact is senior leadership wants their big project to be speedy. They want to have incremental delivery. They want to have high quality delivery. So they're looking for speed and agility at the same time as the data practitioners are looking to have high quality, well understood and well protected data across the organization. So senior leadership is basically an audience for both data governance and for taking the agile approach to implementing projects. And we need to recognize that and we need to help to educate people at the senior leadership level as to how these two different disciplines that seemingly on the surface have some different goals instead of being fast to be accurate. Those types of things that we understand how we can get these two things to work together. So what does senior leadership think about data governance? Well, let's talk about a couple of the reasons why senior leadership and organization are taking the approach that they want to govern their data. So typically they're saying that we want to manage data as an asset of the organization, that we want to protect the data, that we want to use the data to drive analytics and decision making capabilities so that we make sure that we're delivering just in time data to the appropriate people at the appropriate time for the use that they need for the data. We want to make sure that the auditors are going to senior leadership and saying that we need to have auditable and demonstrable results of how we are improving the quality, improving the protected aspect of the data. And they see that other organizations are governing their data and they're saying if we're going to use data as an asset of our organization, we need to make sure that we have the formal execution and enforcement of authority over the management of data. If you remember that, that's the definition that I use for data governance. And since other organizations are governing their data and the organizations that are just now starting to govern their data are looking at their competition saying if they're going to get a competitive advantage out of their data, we need to do the same thing with our data. So therefore, we need to put our formal governance in place in our organization. So what does senior leadership think about data governance? Well, there's several reasons why senior leadership also are not finding it important to implement governance. First of all, they don't understand different approaches that are available to implementing governance. They may think that data governance has to be about command and control, that we need to assign people into new roles. And in one of the future webinars that I spoke about and the presentations at the events that I spoke about, one of the things that we want to do is make sure that we as data practitioners understand the different approaches that are available to governance and that we're able to communicate those to senior leadership. So they're not... Any fear or concern that they have of data governance being threatening to the organization, that they understand that there are alternative approaches that will be successful that don't have to be all about command and control. So they have the perception that data governance is very complex, and it's often based on the size of the organization, which is true. It can be complex, but again, depending on the approach that we take, we can limit that complexity and we can simplify the way we're putting governance programs in place. A lot of senior leadership thinks that governance moves slowly, that it's going to be expensive, it's going to require a lot of resources. We definitely understand that it's going to be a change in the culture of how we manage data as an asset of the organization. So we have a lot of education that we need to provide to our senior leadership to help them to understand why data governance is important and kind of steer them away from thinking... What they're present thinking is that data governance is going to be difficult. There are alternative approaches that organizations can use to implement governance in a very effective way without being all about command and control. So what does senior leadership think about agile? So what are some of the reasons why senior leadership want to be agile in their approaches to large initiatives where data integration across the organization is going to be critical? And so the reasons that they're implementing agile is that the large projects, as they know it, are taking too long and oftentimes they cost way too much. The large projects need to be managed and to deliver quickly and accurately. They're implementing the agile approach because the projects must be delivered early and continuously rather than late and over budget. The project communications must be face-to-face. We are looking to deliver high quality and timely data to the organization. And we want to do that in such a way that it's going to be quick and effective, but at the same time, we want to make certain that the data that comes from the implementation of these agile projects is high quality and it's going to bring the value that we expect from the investments that we have in those projects. So what's some of the reasons why senior leadership are not implementing agile? Well, the fact is they are. They are implementing agile. In fact, they're finding the only projects that the agile approach is being focused on is these major initiatives that are time-consuming and resource-intensive. So the fact is that they are attempting to deliver projects using agile. They're sold on the virtues of being agile and all the things that come with the quick and effective delivery of systems and the data that goes with it, but they're not necessarily as sold on the virtues of data governance. And so, really, agile is only being used for these major initiatives and those are the ones that are most critical to the organization that we need to be concerned about making certain that we deliver high quality data alongside those initiatives. So what can we do? What does senior leadership know about relating data governance and agile, the agile approach to application delivery? My experience is that senior leadership don't typically have that conversation with folks who say, how can we bring data governance and agile together? And we, as the data practitioners, need to make certain that we educate them on how we can be agile and also deliver high quality data as a result of the agile efforts. So how much effort has been made to relate the two and what effort has been made to get the programs and the different projects aligned and how well are data governance programs acting agile? How are they at being agile in their approach? Well, there's very little concentration that's being given to bringing those two together. And we need to look for those things that we can leverage that are in the goals and the opportunities presented by both data governance and the agile approach. So we know that agile is a development methodology and the idea is that we know who does what when and how when it comes to the development effort. It's very structured, it's very disciplined, and it's very timely when the fact is with data governance, and I'm trying to, there we go, governance, we know that governance is the execution and enforcement of authority and they want the same things. They want to detail who does what, when, and how when it comes to governing the definition, the production, and the usage of the data. And oftentimes the governance initiatives are very structured, very disciplined, very timely, and if I go back and forth between the previous slide and this one, you'll see that basically the bullet points are the same. They want to know who does what, when, and how when it comes to the things that we're trying to deliver through these disciplines, they're very structured, very disciplined, and very timely. And one of the things that we can do as data governance practitioners is understand what level of requirements we need in order to be successful with the data without interfering with the timelines that are associated with our agile efforts. You know, one of the things from my experience that I've seen is if you go to an agile effort and say, no, no, we've got to stop here, we've got to do some gut checks, we've got to see do we have all of our ducks in a row, do we have all of our guidelines. Well, the agile teams are working much quicker than that. So if we, again, as practitioners can limit the information that we need from them or we can find ways to be able to get that information from them more quickly and more effectively, that's one of the things that we can do as practitioners to bring data governance in our agile efforts together.  And it's not a bill of rights like a traditional bill of rights where it's, you know, we the people, you know, that we have certain rights. It's really, it's getting the right people involved at the right time in the right process, using the right data to do the right things, to get the right results at least most of the time. Well, I'll be darned if that's not exactly what the agile is trying to do, but they're trying to do it in a very quick and effective manner. So if we, again, can look at the approach that the agile development teams are taking and we can look at the approach that we're taking to implement governance and the goal of both being, is being to get the right people involved at the right time in the right process and so on and so forth, then we want to look at things associated with agile where we can use those things to justify the need for data governance in our organization. So if governance is what we say it is, it's the execution and enforcement of authority. Well, and we take a look at, we remember what that diagram was that I shared earlier about the agile iterative approach. You know, it talked about the different steps within an agile approach, being the development of the functionality, the integrate and the test, the demo release, the client feedback, all of those things. Well, basically, in order to do those things effectively, we want to make certain that we involve the right people at the right time in the right effort and all those things that I mentioned in the Bill of Rights. So basically, governance, theoretically, should be already built into a lot of the steps of the iterative agile effort. And if we can highlight what those things are and get people to understand that they're really not coming at things from the opposite ends of the spectrum, then we can use that knowledge to help to apply governance where it's needed within the agile efforts without interfering with the agile efforts. So again, maybe what we want to do is we want to take the Bill of Rights that I've associated with data governance and implement those in terms of development. So getting the right development people involved at the right development time and the right development process, all of those types of things, but focus on it from a developmental perspective and make sure that we at least are involving the appropriate people in the development of the requirements that we need in order to be successful. So the focus of agile, as I mentioned earlier, is people in interaction, speed and effectiveness, delivering working products and pieces of software, evolving the requirements over time and getting the right level of collaboration across the organization. And the idea, again, is to evolve the solution and be responsive to what our business community is asking for. Well, the fact is that data governance practitioners want the same thing. They want to involve the appropriate people at the appropriate time to get the appropriate level of requirements in our projects. A couple other things that we need. We know that the data governance practitioners want speed and effectiveness. Our idea is not to slow down the process, but to assure that the data associated with the process is the way that it needs to be so that when we deliver our agile projects, that there's not a lapse in the quality and the usefulness of the data that's associated with that project. The data governance practitioners want things to be working products. They want people to see how the products work for them. So data governance practitioners are not adverse to delivering things in incremental ways of working products and software. We know that data governance practitioners also want to make sure that we're gathering the appropriate level of requirements and that we're bringing the right people into the fold as we're developing those requirements. And we want to be responsive and we want to make certain that we are evolving the solution into what it needs to be. So we want to make sure that the data governance practitioners look at agile as an opportunity to highlight those areas where data governance is going to add value and focus on those rather than taking what a lot of people in the traditional sense think of as going to be a process that slows down everything. We want to be able to respond quickly to the initiatives that are focusing on the agile development method. And in the past, basically, I've talked about different core principles associated with governance which are managing data as an asset, formalizing accountability, following rules and being consistent in the way that we govern data. Well, we want to apply that to the agile efforts too and make certain that data isn't an afterthought and that we have the appropriate level of focus and that we're involving the appropriate people and that people have the formalized accountability for making sure that the definition production and usage of data is well-defined. Even if it's going to be an agile effort, we need to make sure that we're managing the data that's a part of that effort as an asset, formalizing the accountability, following the rules, just because we're developing and delivering something in an agile manner doesn't mean that we don't need to follow the rules associated with compliance and regulatory controls and protection and privacy and those things that are being truly governed from external sources telling us that we need to do these things. We want to make sure that the data associated with the agile efforts follow the rules and we want to be consistent in how we do that across the organization. So how do we begin to bring these approaches together that are different? Well, one of the ways that we're going to do that is to look at what does agile efforts and data governance efforts have in common and we want to outline those types of things. We want to strengthen that. We want to build on that. And as the shampoo commercials say, we're going to rinse and repeat. We're going to continuously look for things where there is some level of coordination and cooperation between what agile is looking to deliver and what data governance is looking to deliver. And we're going to highlight those things. And again, as I said, we're going to rinse and repeat using those things to further develop our agile efforts to take data governance into consideration. So one of the outline for you on this screen here is a bunch of the different principles that are associated with the agile development efforts and that's complete features before moving on, testing early and often, active user participation. All of those things that you see on the screen in front of you now are some of the primary principles that agile efforts have. They want to do all of those things, that they want to provide good design. They want to have simplicity in their approach to developing their systems. They want to maximize the amount of work, but they want to minimize the amount of effort that it takes to deliver successful projects. So if we look at these, how many are there? About 15 different primary principles associated with agile, what we want to do is we want to identify where in those principles data governance can have some justifiable level of impact on the agile efforts. So the ones that I've highlighted for you are the active user participation because we know we want to get the right people involved at the right time. The requirements at a high level, the management of technical debt, which I'll describe in a minute if you don't know what technical debt is, but we want to make sure that the requirements not only evolve over time, but that we understand the appropriate level of requirements before we can even get started with our agile efforts. So let's take a look at each of those things individually as we discuss how to use agile to justify data governance. So the agile principles that need to be targeted to justify governance are those for the active user participation, the requirements at a high level, the technical debt, and making sure that we're involved in the data requirements as they evolve to fit the timescale that we have associated with our agile efforts. So the first one was active user participation. So data governance typically speaks that same language. We want to get the right people involved at the right time. We want to make sure that we get the level of requirements that we need to be successful with our projects moving forward. So really, the essence of being non-invasive in our approach to data governance is that the users follow a governance process, that we formalize the user's accountability, and we get the appropriate people involved to define those requirements for us, and that we can escalate issues to formalize decision-making. So we want to activate the appropriate people at the appropriate times in our efforts in order to apply governance to an agile effort. These are things that both agile want and the data governance want in our projects. One of the areas where governance can potentially have its greatest potential for justifying governance for the agile efforts is the level of requirements that we are capturing associated with the data that's associated with our agile effort. So this is one of the areas where potentially we have the greatest ability to be able to justify data governance in these agile efforts. So data governance practitioners pride themselves on capturing the required metadata, and what I've heard from individuals is that they feel, from people that are focusing on agile, look at the data governance practitioners and say that they want, you know, 15 layers, 15 pages of documentation associated with each piece of data and saying that the data governance practitioners are asking for too much documentation. So one of the things that we can do is we can either streamline our approach to gathering the requirements or we can make sure that we are involved enough in the initiatives to make sure that the appropriate level of requirements are being documented. So one of the things, one of the words of wisdom is that we need, that I would suggest is that we need to bring our data practitioners and our agile practitioners together and get them to agree on the appropriate level of requirements that we are looking to gain for our data as part of their effort without interfering with their timelines and the things that they're delivering as part of their agile efforts. So as I mentioned, the requirements at a high level, this is, again, one of those areas that has the greatest potential to justify data governance for agile, but we as data governance practitioners are looking for the vocabulary metadata, the business metadata, the technical metadata, excuse me, the lineage metadata and the linkage metadata that's associated with the data, and that's a lot that we're asking for. And it's very difficult necessarily to document to the nth degree or to the degree that's necessary in your organization those things and help them to stay agile and quick in their delivery of their projects. So we want to, again, focus on the things that we can limit in our requirements to get the information that we need and make sure that we have a justifiable use for all of the information that we're collecting and that we're not just slowing down the agile efforts or interfering with their ability to move forward. Oftentimes, as data practitioners, we look at things like logical and physical data modeling, and in agile efforts, oftentimes the modeling of the data is not necessarily something that's at the forefront of their minds. So models require questions about things like cardinality, retention, growth and those sorts of things. Models promote a common understanding of data across the organization. Models require the participation of data analysts, data architects, those types of people that really have data as their most important focus for the initiatives. And so we want to make sure that we're collecting that information, but again, we need to do it in such a way that we're not interfering with the agile delivery efforts. So without detailed requirements, it's pretty much a given that projects can create their own understanding, which in turn can result in ambiguity and replication and other things that we don't want to see when we're governing our data. So we know that we need to document the requirements, but we need to come to some level of agreement with agile, the folks that are running the agile projects as to what level of requirements are appropriate and that we streamline the process for gaining those requirements and sharing those requirements and making sure that those requirements are addressed throughout the agile effort. So really the question that we need to ask, if most of the folks that are on this webinar right now are primarily data people as compared to agile people, and typically in an organization, you don't see the agile people sitting down to lunch. With the data people, they seem to be again at opposite ends of the spectrum, but questions that we can ask the data people are, you know, how much documentation is too much documentation? What's not enough documentation? What are the bare basic things that we need to collect as part of the project to be successful with the data aspects of these agile efforts? So do we use everything that we collect? And how do we use it? Can we justify it to the agile teams that the reason that we're asking these questions are that we need to have this information moving forward, or can we put a limit on what we are asking for and again streamline the manner in which we are collecting that type of information? Can we work out an agreement with the agile folks and isn't making really no decision on the requirements and basically continuing on without the level of detail data requirements, is that really a decision that we can make? And oftentimes we need to take that back up to the senior leadership that I spoke about earlier and get them to understand that yes, we can be fast, but at a bare minimum there are certain things that we as the data people associated with these projects need to collect. So another word or wisdom is consider the value of each of the different types of metadata and requirements and things that you collect through the process and let's minimize that and let's gather only those things that we know are really going to be necessary and that we can justify the need to collect these pieces of information as we move forward with our projects. Another one of those core things was requirement at a high level. We want to limit the data demands for the requirements. We want to achieve and agreed upon level of requirements and again that's bringing the appropriate people together from the agile teams and from the data teams and come to some level of agreement. I've seen organizations that have set up service level agreements between their agile teams and their data teams to make certain that at least these bare bone requirements are going to be in place so that we can have our ducks in a row when it comes to the data that's going to be used as part of these initiatives and how that data is going to evolve and basically be the way that it needs to be by the end of the agile delivery effort. We want to provide tools to the agile folks to effectively capture those essential requirements. Again, the limited requirements that we know that we need in order to be successful from a data perspective and we want to provide the resources. We as data people may need to provide resources to the agile efforts to make sure that we're focusing on again that relationship between data governance and the agile projects. So we want to align what data governance needs with what the agile efforts need. And again, another word of wisdom is to incrementally apply that agreed upon level of data requirements to our agile efforts. So let's introduce to them a bare minimum of what we need and if we find that there's additional things we need then let's incrementally apply that to those efforts so that we again as data practitioners make sure that we're getting all the rules associated with the definition of data, the production and the usage of data as a result of these agile efforts. There's something that I mentioned earlier called technical debt. And so technical debt is typically requirements that eventually need to be addressed but they're put into what I would call a parking lot list. These are things that we know that we need to address but we'll get to them at some point and we can't address them all right now because that's going to slow down our agile efforts. So one of the things that we as data practitioners can do is take ownership of that technical debt and help to drive to the resolution of those things that are associated with the technical debt. Oftentimes in organizations those things are referred to as design debt or code debt but it's also, if you think about it, the debt can be thought of as the work that needs to be done before a particular job can be considered to be complete or properly finished. So if the debt's not repaid as we go then it's going to continue to accumulate making it harder to implement these changes later on and that's information that I got from kind of a general definition of how agile efforts work in an organization. So if we as data practitioners take ownership of the technical debt, of the data debt, of the design debt and we make certain that those issues are not just being swept under the rug and that they're being addressed, that's one of the things that we as the data governance folks can do to help to help the agile efforts and make sure that nothing is being swept under the rug and all these technical debt issues are being addressed. So another set of words of wisdom is that this is one of the biggest reasons why data people are skeptical about agile is that so many of these things are being put on these lists, on this technical debt list and they're not being addressed until too late in the game. And if they're addressed and it changes something in the requirements then that means that the agile efforts need to go back and readdress these things. Well, what we want to do is stay aligned with the agile projects, make sure that technical debt doesn't build up over time and that we have data people that are associated with those technical debt things that are data related in our organization. So one of the reasons why, what are some of the reasons why technical debt accumulates? Well, because there's business pressures to get things done up and running quickly. There's a lack of understanding as to how these things are going to impact what's being delivered through the agile efforts that there's a lack of a test suite which encourages basically quick solutions to things or band-aids to apply to these big data issues rather than just applying those band-aids if we as the data governance practitioners help to take ownership of the technical debt then we're assuring that we're staying up or staying at least as much as we can in line with the steps of the process that are associated with the agile efforts. And reasons why technical debt accumulates that oftentimes there's the lack of the documentation which kind of brings me back to what I mentioned earlier, which is we want to make sure that the information that we're collecting from the agile teams and the business communities that are going to take advantage of these agile delivery efforts that we have the appropriate levels of documentation in our organization. Some additional reasons why technical debt accumulates is through lack of collaboration or not necessarily getting the right people involved at the right time. There's parallel development going on which could be contradictory to each of the efforts. There's a delayed set of refactoring as we know that the requirements for the projects evolved. It may be clear that parts of the code that have been written in earlier pieces of the agile delivery effort may need to be refactored if we don't do our due diligence and collect the requirements that we need when we need them in the project. So the lack of knowledge when the developers simply don't know how or when to write the code or what to write the code for because those requirements haven't been documented to the level of detail that they need to be documented. So we want to apply governance so that the technical debt does not accumulate. That's one of the ways that we as data governance practitioners can assist in the delivery of these agile efforts. So formal accountability for managing the technical debt might be a shared responsibility between the agile teams and the data governance teams. We want to make certain that we have formal escalation processes so that we can escalate issues that are associated with the technical debt and resolve the issues in a timely manner. Again, not to fold up the speedy delivery of the agile development teams. We want to have formal reporting of the technical debt status. Typically when the data governance team gets involved in the agile efforts, they may take on the responsibility of that formal reporting of what is on that technical debt list and how it's being addressed and who has the responsibility for addressing it. So the last of the four things where I think that we can look for some level of alignment between agile and data governance is that the requirements evolve. And this is basically one of the biggest challenges when we're justifying data governance for agile is that we want, from both perspectives, we need to evolve the requirements, but we have a limited amount of time to do it. So we're shooting at a moving target, but we want to make sure that we have at least the level of requirements that we need to assure that the data aspect of the agile-focused efforts are the way that they need to be in order to deliver effective agile projects. Requirements often mean that there's a gradual process in which the requirements are being collected. Oftentimes they change over time, and we need to make sure that we again govern the change to the requirements, govern the change to the agile efforts, and make sure that we have a process in place to make certain that as these requirements evolve over time, that we as the data folks are aware of how those requirements are changing and that we're involved in making certain that the data aspect of these agile-delivered efforts is appropriate for the organization. So let's break it down. We're just going to be a gradual process for collecting these requirements. They change into something different over time, so we need to make sure that we have people looking at that through the data magnifying glass to make sure that we're on top of how those requirements are changing, and oftentimes as the requirements change, they become more complex, they move into a better form, but it requires the constant governance of the gathering of those requirements and the use of those requirements to deliver effective agile efforts. So typically the management of the requirements requires some level of governance of the requirements, not only to document and record those, but to vet those and make sure that they're properly being addressed. Typically in a large project, they can tell you two things out of the three. The three things are how much time it takes to do something, the resources that are required, and the effort that's going to be required. Well, oftentimes they can tell you how much time you have and how many resources you have, and then you can describe to them from a data perspective what the effort is that's required, or they may give you the amount of time and the effort that's required, and you'll tell them how many resources are going to be required from a data perspective. But my point here is that they can't necessarily, even though they may try, to give you all three of these, typically in an organization to be successful, they can really only give you two of the three so that you can define the third. Again, if they tell you the resources that you have and the effort that's required, you can tell them how long it's going to take from a data perspective. So again, they provide two of the three, you can provide the third to make sure that the data aspect of the agile projects are being addressed. So agile itself is a form of project governance. It's just not data governance. And again, as we try to align data governance with agile efforts, we want to make sure that the data requirements are being addressed in a timely manner as to not slow down the agile delivery effort. So the bottom line is that governance is the execution and enforcement of authority, but the governance of the agile efforts is the same thing. It's the execution and the enforcement of authority to make sure that the projects get done in a timely manner and that they address the business requirements that are associated with the project. So we want to make sure that there's challenges to make sure that we're gathering the most important requirements, the most effective requirements and making that information available, making sure that we have those available requirements and that they are essential to the success of the agile efforts. And we want to make sure that the most essential requirements are in a timely period so that we don't prevent the agile projects from moving forward. So basically, real quickly, I talked today about who are the audiences for both agile and data governance and how can we bring them together and how can we get them to understand that even though we want both of these things, we've got to find a way of bringing these efforts together and having them cooperate. We talked about the governance aspects of agile and things that are in agile that require a level of governance. We talked about why data governance practitioners should embrace agile. What are some of the considerations for data governance that come from the agile principles that I shared with you? And last but not least, we talked about how to use agile to justify data governance as a form of making certain that we're delivering high quality data and we're delivering it quickly in these agile efforts. So that's what I have for you in the webinar today. Just a quick reminder that the upcoming webinars in January, February, and March are as you see here. And what I'd like to do is kind of turn it back to Shannon and see if we have any questions from today. Hi, Bob. We do have great questions coming in already and great presentation. Thank you so much. Just to answer some of the most popular questions that we get, I will be sending out a follow-up email by end of day Monday with links to the slides, links to the recording, and anything else requested throughout the webinar, including your bill of rights. So we do have that going out as well. So first question coming in here is, in my experience, it's not just the large efforts that senior leaders support going agile. They're supporting all software-related work going agile. Are you seeing more of that as well? No, I'm not actually. I'm seeing that agile is being at least the ones that the organizations are talking about. The agile efforts are associated with things that I mentioned, like I mentioned before, the ERP transformations where you've got a company that has been working under one, working under multiple ERPs, and they're looking to migrate to a single ERP, something like that, or redevelopment of a data warehouse effort. And to do that in an agile effort, it's just from my experience that I haven't seen agile being associated with the small software project. But then again, that's just my experience. But if you are in an organization where agile is being applied to these smaller efforts, the same thing still holds true. There's still data requirements that need to be gathered and need to be addressed. And again, we can limit those things that we're asking for from the agile teams in order to even make those smaller software projects successful. Sure. A lot of software questions come up and no surprise here. So can you share three to four technology solutions that can help assist in data lineage? Three to four technology solutions? Well, I can start with the one, which is the lineage metadata and knowing where the data came from and knowing where the data is going, because that's going to be one of the first questions that people have when they start to use these efforts that have been delivered to the agile delivery method. We need to know not only what did the data look like before it was moved over into this project, before it was transformed, what were the rules that were associated with transferring that data? So I would say that metadata is certainly one of the biggest technologies that can be applied. But we also need to have a place to be able to record that information and make that information available to people, including the agile development teams, to make sure that the appropriate level of documentation is there for the people that are expected to leverage and use the agile, the things that are being delivered to the agile method. Now, Bob, I know you remember the DGPO. So how do you see the DGPO and the PMO working together to enable an agile data governance model? That's very interesting. So the DGPO, the Data Governance Professionals Organization, is looking at best practices around data governance. And so I'm not enough involved with them to know that they have taken this on as a task. And one of the great things about the DGPO is that it's a community of interest of people around data governance. And I would say that the more people, the more of you that are involved in the DGPO, that go to them and say, one of the things that I really need, one of the areas where I'm lacking is in that coordination between data governance and agile, make it a point to the DGPO to make certain that there are these things that we need, and if they hear enough from enough people, they will, I'm sure, knowing the folks that run the organization, they will apply resources to providing you an answer of what can we do to, in a timely manner, assist with these agile efforts. That's a great answer. And in order to do the way you are talking about, the data governance practitioner has to be in the agile development team, basically the member of the agile team. If data governance committee is a separate entity and not part of the project team in an organization, how does it work? It doesn't. To be simple with the answer is that if you don't have people involved in the agile team, and let me share with you a quick anecdote about a company that was moving to do an ERP implementation for the first time in the organization, and there were no data focused people or people that were part of data governance that were involved in the team. Well, the way that we got started was we took individuals, data analyst type people, and we involved them in the efforts, basically as being a fly on the wall in the meeting. So they were instructed not necessarily to raise their hand or put the cup wash on things that were moving forward, but they were there to hear how the agile efforts and agile meetings were taking place. And so what we basically did was we instilled somebody from data governance, from a data team on the project team. If you have a problem with that, if your organization or if the agile teams are not willing to get data people involved, then that requires some level of escalation to somebody in the organization that says, look, we're not going to interfere with what you're doing, but we need to hear and make sure that we are collecting the appropriate level of data requirements in order for you to be successful with your project moving forward. So the easy answer is, no, you're not going to be successful if you don't have data people at least having an eyeball on what's happening with your agile efforts. And so what are the required steps in order to get somebody involved, whatever they are, get it to the appropriate person who says, yes, we can at least provide a resource to sit in and listen to what's going on during these agile meetings. Well, Bob, I'm afraid that's all we have time for. We do have a few quick questions coming in, but just let everybody know. Keep your questions coming. I'll be sure to get those questions to Bob and we'll get the answers to those questions unanswered in the follow-up email. Again, I'll send that follow-up email by end of day Monday with links to the slides, links to the recording of the conversation and additional information from Bob, including a link to his Bill of Rights as he heard about in the webinar today. So Bob, thank you so much for this great presentation and for wrapping up the webinar year so well. We certainly appreciate it and look forward to 2017. I am looking to it as well. Happy holidays, everybody. Have a safe and happy holiday. And happy new year. Hope everyone has a great day. Great. Thank you. Cheers.