cale up to the load or occasionally to replace an existing instance starting a new function instance allows loading runtime and your code requests that include function instance startup like cold starts can be slower than requests hitting existing function instances so if your function receives a steady load however then the number of cold starts is typically negligible unless of course your function frequently crashes and requires restarting of the function environment the environment running a function instance is typically resilient and reused by subsequent function invocations unless the number of instances is being scaled down say due to lack of ongoing traffic or your function crashes this means when one function execution ends another function invocation can be handled by the same instance therefore it is recommended to cache state across invocations in a global scope where possible your function should be still prepared to work without this cache available as there is no guarantee that the next invocation will reach the same function instance as we saw in the stateless function slide function scope versus global scope a single function invocation results in executing only the body of the function declared as the entry point the global scope in the function file which is expected to contain the function definition is executed on every cold start but not in the instance has already been initialized you can assume that the global scope has been executed exactly once before your function code is invoked in a new function instance and on every subsequent creation of a new function instance however you should not depend on the total number of or timing of global scope executions as they depend on the order scaling which is managed by Google now the function execution timeline a function has access to the resources requested CPU and memory here only for the duration of function execution code that is run outside of its execution periods is not guaranteed to execute and can be stopped at any time therefore you should always signal the end of your function execution and avoid running any code beyond it let's look at the execution guarantees your functions are typically invoked once for each incoming event however Google functions does not guarantee a single invocation in all cases because of differences in error scenarios the maximum or minimum number of times your function is going to be invoked on a single event depends on the type of your function HTTP functions are invoked at most once this is because of the synchronous nature of HTTP calls and it means that any error on handling function invocation will be returned without retrying the caller of an HTTP function is expected to handle these errors and retry if needed background functions are invoked at least once this is because of the asynchronous nature of handling events in which there is no caller that waits for the response and could retry on error an emitted event invokes the function with potential retries on failure if requested on function deployment and sporadic duplicate invocations for other reasons that is even if retries on failure were not requested when using multiple functions just to recap you want to make sure not to rely on shared memory variables file systems or state to share data across deployed functions you can use storage services like cloud data store cloud firestone or cloud storage alternatively you can invoke one function from another using their appropriate triggers or you can chain cloud functions so how do you build more complicated apps once you start to get to the world of 2, 3, 4, 5, 60, 70 different services and products how do you connect them how do you coordinate them so let's first talk about chaining functions so we got a bunch of these functions and we want to put them together here we have cloud pubsub pubsub is a publisher subscriber messaging system so in this example we have a new employees data getting inserted into firestore there is a cloud function listening for new employees as we want to trigger some other events when this happens a message is sent to cloud pubsub here is a new employee Brett location New York City now a new employee here needs a building business badge created for them so there is another cloud function that is listening to the new user topic and triggers the process to create a new badge for the new employee the same message can also trigger a third party system like an IT helpdesk system in this case and then for cloud monitoring and logging stack driver is built into this error logs and console logs are written automatically out of the box by default cloud functions for firebase is optimized for mobile developers cloud functions for firebase provides a way to extend the behavior of firebase and integrate firebase features through the addition of server side code it's optimized for specifically firebase SDK to configure your functions through code it's integrated with firebase console and firebase CLI the same triggers as google cloud functions are available plus in addition you have firebase realtime database firebase authentication and firebase analytics products can be used as triggers now let's take a look at the demo in this demo we will look at background services so here the objectives are to write and deploy a background cloud service and which is going to be in pubsub and then trigger that function by publishing a message to a cloud pubsub topic so these are the steps for this demo which we will go through on the UI now this is the code that we will be using in the demo and the command line in order to test the background service function in this demo we will create a google cloud function for a background service and that background service is going to be for the cloud pubsub and we will send a message to a topic on cloud pubsub and have a function triggered by that message so let's take a look at how we can do that so now we are back to our google cloud platform UI and as usual the first step is to make sure you have your billing set up and have a project created or select an existing project so we go to the project drop down here and in my case we are selecting the app engine test that we used for the previous demo so again we just want to take a note of the project ID which is concrete fusion 228 322 the project ID is unique across all google cloud platform products and services so now let's go to the navigation menu and find cloud functions so now let's go to create function and this time we will name this pub sub demo and for the trigger we will choose cloud pubsub and in this case we need to specify a topic since there are no topics set up here let's create one and now for a new pubsub topic the format is going to be projects and then the project ID topics and then the topic name so we will call this pubsub topic and create and we can choose to use the inline editor no js runtime and here we can write or update the code so instead of the boring code and the example code that's provided let's add our own code here so this code is provided in the slide previous to this demo so it's the same code that you can copy and paste in your projects so this code is a background cloud function that's to be triggered by pubsub so in this function it's looking for a message where the message represents someone's name a person's name and it's simply attaching the text hello comma and then appending the message which in this case it's expecting a name so we're going to send that message out so let's copy this code and paste it here ctrl v and we will keep this as the function to execute hello pubsub hello pubsub and create so now the function is being created and being deployed when this circle stops spinning and we get a green checkbox like here then the function has been successfully created and deployed so here we can see that it's done and we are ready to take a look at that let's just look at the details of that function by clicking on it and drilling down and it gives us the same dashboard for that specific function there's currently no data because that function hasn't been triggered yet let's check the trigger the trigger type is cloud pubsub and the topic is pubsub topic and we can take a look at the source code here and we can have a testing block here as well so now let's test our new function by publishing a topic and a message so for that instead of doing it in the block here let's launch our gcloud shell by clicking on the icon here we activate the cloud shell and the google cloud console is open here at the bottom now we just want to make sure that the correct project has been selected so in this case it's our concrete fusion 228332 project ID so now let's go back to our code here now this command gcloud pubsub topics publish the topic name and with the flag message and the person's name so this command will publish to this topic with this message so let's check the name of our topic here so this would be pubsub topic and let's say the name is james so let's copy this and paste it here ctrl v and hit enter so the message has been published and this is the unique message ID so now let's see if that actually worked can go back to general to our function and click here and check the logs now these logs are provided by stackdriver which is another google cloud platform product that provides monitoring and logging across all google cloud product and services so here we can see that the function execution has started for pubsub demo and then we can see the message here hello comma james that was sent so this has been a successful background function so just to recap what did we just do so we created a message on google cloud pubsub and then we created a function that should be triggered whenever someone publishes a message to that topic so every time someone publishes a message to that specific topic a function will be triggered so that's the background function that we created what does that function do that function takes the message from that topic and it appends hello comma to the message because we are expecting someone's name and then it sends out that message so that's what we just saw here so this concludes the demo options and database solutions that suit a hobbyist to a fortune 500 company I'll walk through a few of these options and provide a way to choose the right solution for your need almost all applications need persistent durable storage of some sort whether it's user accounts images events documents, logs or anything else different applications have different storage needs Google Cloud Platform offers 7 storage options Bigtable Cloud Datastore Cloud SQL Cloud Storage Cloud Spanner Persistent Disk and BigQuery BigQuery is a bit special because it's both a storage service and a powerful analysis tool so why Google Cloud Platform there are other options such as Microsoft Azure or Amazon Web Services or individual databases such as Microsoft SQL Server MongoDB etc the advantage here is a single API can be used across different storage classes it's scalable to exabytes of data and designed for a 99.999999999999% durability depending on the type of Datastore that's chosen there is very high availability across all storage classes and the time to first byte is in milliseconds as long as the time to first byte as long as the Datastores are on the Google Cloud Platform they all enjoy the same reliability and infrastructure that Google provides as well as strongly consistent listing now there are two categories of storage services structured and unstructured if the data you want to store can be organized into a table structure with columns and rows then it's structured data examples of structured data are user profiles event logs sensor measurements sales records and stock trade data structured data comes in many different shapes, sizes and usage patterns there's a great diversity in the way that you can store and interact with it Cloud SQL Cloud Datastore Cloud Bigtable Cloud Spanner Persistent Disk and BigQuery are all structured data unstructured data is also available in Google Cloud Platform this is simply called Google Cloud Storage you just provide a sequence of bytes to store a place to store it and a way to identify it and Google just stores them for you there's no way of knowing or there's no insight into how this data is actually internally stored all we know the way we can interact with the data is through buckets and objects an object is similar to a file however there is no file system here now let's take a look at some of these options in detail Persistent Disk is a fully managed block storage that's suitable for virtual machines and containers so if you have Google Compute Engine instances and for Google Kubernetes Persistent Disk is sort of like having your own hard drive or an SSD disk available for good for snapshots and data backup common workloads are for virtual machines and sharing read-only data across multiple virtual machines it provides a rapid durable backups of running virtual machines Cloud Bigtable is a scalable, fully managed NoSQL wide column database it's suitable for both time access and analytics workloads it provides a low latency read-write access a high throughput analytics and native time series support this is sort of similar to MongoDB where it's a NoSQL database that's great for IoT, finance, ad tech, geospatical datasets graphs and personalization Cloud Datastore is a scalable, fully managed NoSQL document database that's useful for web and mobile applications it stores semi-structured application data in a hierarchical manner and in a key value format it's great for user profiles product catalogs and to save game state Cloud SQL is a fully managed MySQL and PostgreSQL database service it also enjoys being built on the Google's infrastructure which provides reliability and scalability it's good for web frameworks structured data OLTP workloads it's commonly used for content management systems BI applications CRM and geospatical applications Cloud Spanner is a newer offering from Google that helps mission critical applications it provides a mission critical relational database service with transactional consistency global scale and high availability it's useful for financial services global supply chain retail and ad tech Google BigQuery is a scalable, fully managed enterprise data warehouse with SQL and fast response times it's good for exploring big data sets and petabytes scale applications and workloads the common workloads are analytical reporting on large data big data pricing using SQL and data science and advanced analysis now finally cloud storage is the unstructured data store that Google provides it's a scalable, fully managed, highly reliable and cost efficient object blob store it's good for storing images pictures and videos objects and blobs and any unstructured data common workloads include archives, backup disaster recovery storage for custom data analytics pipelines and storing and streaming multimedia mobile storage options are also a newer offering however Firebase and Firestore have been with Google for a while some of these specific products are still in beta cloud storage for Firebase is for mobile and web access to Google cloud storage with serverless third party authentication and authorization Firebase real time database provides a real time no-SQL JSON database Firebase hosting is a production grade web and mobile content hosting for developers and finally Cloud Firestore is a no-SQL document database that's for mobile applications with all these different options how do you choose what's the right product for your need to figure this out the first question you want to ask yourself is, is your data structured if the answer is no then is it a mobile application if it is then cloud storage for Firebase is the right product if not simply cloud storage if your data is structured then is your workload analytics if the answer is yes then do you need updates or low latency if yes Google cloud big table if not BigQuery if your workload is not analytics then is your data relational if it is then do you need horizontal scalability and if it's not is it a mobile application if it is a mobile application then cloud firestore for Firebase would be the right product if not cloud data store if you do need horizontal scalability then cloud spanner would be the right product if not cloud SQL the first part explores the basics of cloud storage and in the second part we'll look at the more advanced features so what is cloud storage it's a fully managed product that's provided by Google with its usual high reliability and cost efficient infrastructure that provides storage space for object and blob stores it's great for storing images, pictures videos any other sort of unstructured data typically it's also good for streaming multimedia storage for custom data analytics pipelines archive backup and disaster recovery integrated into all the apps and products and services offered by Google it's optimized by price performance across four storage classes you can manage that with the object lifecycle management you can access data instantly from any of these storage classes the durability here cloud goes up to 99.99999 that's up to 99 and the availability ranges from 99% to 99.9% SLA depending on the type of storage class that you choose and the amount of money you're willing to spend for your storage and also the scalability can go up to exabytes of data with a millisecond responsiveness so data in cloud storage is stored in buckets and data is stored as objects however unlike directories and folders you cannot nest buckets when you create a bucket you specify a name geographic location and a default storage class these are the only three properties that are required the location depends on where you want the bucket so store frequently access data such as data used for analytics as regional storage and you can use dual regional location when you want similar performance advantages as regional locations but with added geo-redundancy when you want to serve content outside of the google network and distributed in large geographic areas or when you need your data to be globally redundant you can use the multi region location bucket names that contain dots need to be valid the main names we will look into this in detail in the advanced lecture there are four types of storage classes multi regional storage near line storage and cold line storage multi regional guarantees two replicas by default at least 100 miles apart it's for the highest availability of frequently accessed data like website content and business continuity regional is for data accessed frequently within a region it's great for storing data for analytics or compute within a region it has the lowest latency possible within a region and it's cheaper than multi near line and cold line are for infrequently accessed content and archival storage this is comparable to glacier storage at amazon aws if you're familiar with that near line is typically for less than once a month usage and cold line is for less than once a year usage so each bucket has a default storage class that you select when you create a new bucket the objects that you add to this bucket are automatically assigned this storage class however this can be changed and only new objects added to the bucket will be assigned the updated storage class you can change the storage class for individual objects with an api you can't do this from the ui however there are some restrictions on what you can change it to based on the bucket location remember when creating a bucket there is a location and a storage class these are still two separate things now let's take a look at a demo on how to create storage buckets in this demo we'll learn how to create buckets so here we're back to our familiar google cloud platform user interface so before you begin as usual please make sure that you have your billing information set up as well as your project created or select the correct project so here I have my project selected if you need to you can create a new project by clicking on new project in this case I'm going to choose my existing project which is displayed here so now let's go to the navigation menu and look for cloud storage it's under storage and storage so we click on that and it takes us to the area where you'll see a list of buckets here here I have already three buckets from a previous project so let's create a bucket you can do that by pressing the create bucket button here click on that and as we have discussed in the lecture there are three main properties that you need to specify the name of the bucket the default storage class and the location so let's call our bucket name bucket 090 that's not taken already so it has to be a globally unique name and we'll just leave it as multi-regional and the location in the United States and press create that's it now we're in the bucket details so if we go back here by pressing the arrow key now we can see a list of all the buckets in our project in this case we can see we created bucket here let's look at a demo on how to get the bucket information now that we have created a bucket let's get some more information about that bucket through the command line so let's initiate and activate our cloud shell by clicking on this icon here this opens the google cloud console and we can write command lines here now the commands we'll be using here are listed in this file and these same commands can be seen on the slide previous to this demo so to list the buckets we can just simply use the ls command after the gsutil command so just copy that and paste it here ctrl v enter and now we can see that our bucket is listed here along with the other three buckets that are here so now let's take a look at the bucket size so for that this is the command gsutil du-s and then the name of the bucket ctrl v and then let's type in the bucket name which is bucket 190 hit enter and this will tell you that the bucket size is currently zero because there are no objects inside the bucket let's take another look at the next command so now we can change the default storage class of the bucket with this command which is gsutil def storage class set and the storage class that you want to set it to and then the bucket name so let me just copy this part here copy paste and let's so currently we can see that the default storage class is multi-regional so let's change it to let's say coldline and then we are to specify the name of the bucket we want to change that for which is bucket 190 so now that has been set let's see if that worked by refreshing the screen here by clicking on the refresh button and we can see that we have changed to coldline let's take a look at a demo on how to upload, download and list objects now let's add some objects to our bucket for that let me create a new bucket create bucket again and type call this bucket 191 and leave everything as default press create so now we have another bucket on the back button so now let's click on the bucket here and we can see the details of the bucket so currently there are no files here so if you want to add files or objects to your bucket you can directly drag files into this window here or you can also upload files by clicking the button here for example I click this and point to the file that I want to upload this case it's cat1.jpg open and the file is being uploaded here that's finished close this window and we can now see that there is one file in this bucket let's see that the object that has been uploaded also inherited the storage class the multi-regional storage class which is the default storage class of the bucket now we go back one level and we can also upload objects via the command line so in order to do that let's first create a file really quick in the command line so we can use the nano text editor type nano hit enter that opens the default text editor I'll just type in test as the text ctrl o to save the file let's select a name let's just say it's the test file dot txt and hit enter and then ctrl x to exit and now we're back to our console so now if we type ls we can list the files that are currently in this project so here we can see the test file dot txt that we just created so now let's upload this test file dot txt to our bucket in order to do that the command is gsutil cp the object location and then the destination bucket again these commands are on the slide where before this demo starts so let's copy just this part here copy ctrl v to paste and now we type the name of the file test file dot txt and then the destination bucket gs colon slash slash and the bucket name is bucket 191 and hit enter so copying one file completed let's see if that worked we click on the bucket 191 to see the details and now we have two files with the new file added here now we can also download objects from the bucket to our local directory here that it's the similar command cp and then we start with the bucket name slash object and then the destination so in this case let's just download test file dot txt back to our folder here so let's create a sample folder first the command is mkdir to create a new directory oops not here it's down here kdir let's say tstdir test directory let's see if it was actually created ls to list files and we can see that tstdir folder here now to copy our file from the bucket to local we use the same command here copy paste and gs colon slash slash bucket name bucket 191 slash test file dot txt and then we use tilde slash and the new directory that we created which is the tstdir and hit enter and now that now this file has been downloaded from the bucket 191 to our local directory testdir let's see if that actually worked so let's change directory to tstdir and then list the contents with ls and we can see the file here locally testfile.txt we can also delete objects from the bucket from the command line so for that we use the rm command so for that let's remove the testfile.txt object from the bucket bucket191 so I copy that command paste it here gsutil rm and gs colon slash slash bucket191 slash the name of the object which is testfile.txt and we hit enter and that object has been deleted so let's refresh this bucket here by clicking on refresh bucket and we can see that that file no longer exists let's take a look at a demo on moving and deleting buckets in this demo we'll see how we can move objects and delete buckets so currently we have our bucket191 with one object cat1.jpg so let's create another bucket here let's do it from the command line from the command line you only need to specify the bucket name you can optionally specify the storage class the default storage class as well as the location so in this case I'm only going to specify the bucket name so it's gsutil mb gs colon slash slash bucket name in this case I'm going to call it bucket192 hit enter now my bucket has been created so let's refresh this top panel by hitting the refresh button and we can see that our new bucket has been created with the same default storage class so now let's copy all the files and all the objects from bucket191 to bucket192 so in order to do that we can use the copy minus r flag so this is the command for the bucket so copy that and paste it here so to copy files from your old bucket to the new bucket the command is cp minus r and then the source bucket and then the destination bucket so you put in an asterisk here to specify that you want to copy all the files and all the objects from the source bucket to the destination bucket so let's copy this command and move our objects from bucket191 to bucket192 so copy that paste it here and then fill in the rest and fill in the rest which is gs colon slash slash bucket191 slash asterisk space gs colon slash bucket192 and we hit enter so we can see that the operation has completed and one object has been copied so let's see if this is true by refreshing the top panel and then clicking on bucket192 and now we can see that the file has been moved or actually copied here now we can delete the files from this old bucket so that it would actually be more like a move and not a copy so to delete files or objects from the old bucket we can use the command gsutil rm minus r and then the name of the bucket so let me copy that copy and then fill in the rest gs colon slash slash bucket191 the operation has completed and now let's go back and refresh our top panel and now we can see that bucket191 has been removed from our project now let's take a look at a demo on how to view edit object metadata in order to check the metadata of an object we first browse into the bucket identify where the object is and click on the more information icon that's all the way on the right hand side so we click on this and select edit metadata so here now we can edit the metadata for this specific object you can also add items here for your own custom key value pairs and save if you wish to you can also check this on the command line you can also do this from the command line using the gsutil ls minus l command bucket name and the object name so let's try that here going to copy this paste it here and fill in the bucket name which is bucket192 and the object cat1 dot jpg hit enter and now we can see the metadata for that object this can also be edited using the command here by specifying the key value pair of the metadata and then specifying the bucket name and the object name using the set meta minus h command where we'll discuss a few more advanced topics object lifecycle management to support common use cases like setting a time to live archiving older versions of objects or downgrading storage classes of objects to help manage costs cloud storage offers the object lifecycle management feature with this you can automate these tasks with the lifecycle management configuration the configuration consists of a set of rules which apply to the current and future objects in a bucket each rule has an action and a set of conditions if you have multiple conditions all of them have to be met in order for the action to take place actions may not necessarily be performed right away by cloud storage you can track lifecycle actions with logs and by enabling cloud pub sub notifications for cloud storage for your bucket examples of these actions include changing storage class to archive or delete an object after a period of time let's take a quick look at the UI on how to set these object lifecycle management options in this demo we'll look at how to enable object lifecycle management so we're now back at our familiar UI of Google cloud platform so let's make sure that we are in the correct project that we want to select or create a new project if you need to this we are in the app engine test project and let's go to the cloud storage section so we go to the navigation and under storage we have cloud storage now in order to enable now we can see the list of buckets that we have in this project and here we can see the details of the buckets and we can see this lifecycle column where it says none which means that it's not enabled here you can see an example of a bucket where the lifecycle management has been enabled so in order to enable it for our bucket bucket 192 we click on the none and that will open here where we can add a rule so when you click on add rule there are two parts here there are object conditions and there are actions so we can select any of these different object conditions for example age so that we can we can trigger this action if objects are of a certain number of days older or so or based on the creation date based on the storage class based on the number of new versions of that same object that have been created and it's live state or whether it's been archived or it's currently live and then you can select an option here to either set it to a different storage class or to have it deleted and then you press save and that would save and enable the object lifecycle management for that bucket object versioning allows for the retrieval of objects that are deleted or overwritten when you delete an object with it's called it gets archived as a different version you can restore archived versions or permanently delete them this is a feature that needs to be turned on for objects because it adds additional costs these costs can be managed and reduced with lifecycle management the versioning works by keeping track of the object's version and the version of the object's metadata you can add a retention policy to a bucket to specify a retention period if a bucket has a retention policy objects in the bucket can only be deleted or overwritten once their age is greater than the retention period a retention policy retroactively applies to existing objects in the bucket as well as the new ones that you add to it you can also lock a retention policy to permanently set it on the bucket this is done using a special account in order for compliance once the lock is done a regular administrator won't be able to remove or reduce the retention period you can't delete a bucket with the locked policy and you can't delete objects or remove them you can however increase the retention period or place additional holds on it this is again to comply with regulations you can use buckets to host static websites when you do that the name of the bucket needs to be the name of the C name record the objects inside the bucket are the web pages for the site so for example the URL is going to be http colon slash slash the bucket name slash the object name where the bucket name is for example dot com and the object name is the actual file index dot html you can make all the files in the bucket publicly accessible or you can separately set permissions to each file you can also set properties to the bucket to assign a page as a default like for example index dot html and then error pages like for example error 404 page not found when hosting static assets for a dynamic website you do not need to create a C name record and point to a bucket with the same name as you do for a static website in this case the URL will still be cloud dot google dot com and then your bucket name let's take a look at a demo where we create a bucket for a static website so let's host a static website using buckets so we are back to our familiar user interface for the google cloud platform and let's check and make sure that we are in our correct project or we can create a new project if we need to in this case I am in the app for this project now we open the cloud storage browser from the navigation menu let's click on the navigation menu go down to storage and click on cloud storage here we can see a list of our buckets in this project so now we create a new bucket create bucket press this button now for the name of the bucket in order to host a static website it has to be the domain name of your website including the www and the extension so in this case it's going to be www dot appcode world dot com now this domain has to already be verified and you have to prove that you are the owner of the domain before you can create a bucket with this name on google cloud platform so you press create and also if you want to have a dot in the name of a bucket it has to be a proper domain name press create and now we get to the bucket details so I have a few files that I have prepared before and this is just a basic index dot html file which is usually the first page in a website and just a content html file and an error 404 file so in order to upload these files and the two image files you can just drag and drop these files right here into the bucket and they get uploaded so now we can go and check the permissions for this bucket and so right now this bucket here that we just created is not available to the public so it can only be seen within the scope of this project and within the scope of google cloud platform so in order to change the permissions we go to the more menu and then edit bucket permissions so here we can add a member called all users and assign a role of storage and storage object viewer click on add and now you can see that public access has been granted to the bucket now let's assign specialty pages now we go back to the so now let's assign specialty pages to our website so for that we click on the more menu again and this time we click on edit website configuration so now here we have a chance to specify the main page as well as the 404 not found page so in our case it's the index.html page and error 404.html for the 404 page you press save and now we can test our site so when you go into the bucket details you can see all the files which are your objects in that bucket and we can also see that all the files have public access granted because they inherited the same permissions from the bucket and if we click on the link here this will open a new tab in the browser showing the website the index.html now you can see that this is still attached to storage.googleapis.com and that's because the CNAME has not been assigned to this bucket yet so we'll see that in the networking section of the lecture notifications there are three types of notifications that can be set so you can know when an object has changed this is usually useful to notify an application when an object is added updated or deleted in a bucket for example when you add a new picture to a bucket an application could be notified to create a thumbnail so to start watching a bucket for change notification events you can use this command watch bucket which would create a notification channel that sends notification events to the given application URL for that given bucket so you can see the workflow in the diagram below the application on the left hand side sends a watch request to Google cloud storage for a specific bucket when a user makes a change to the objects in that cloud storage bucket a notification change is triggered and through that notification channel a notification event is sent to the application for further action another way of doing this is through cloud pops up cloud pops up notifications send information about changes to objects in your buckets to cloud pops up the information is added to a cloud pops up topic of your choice in the form of messages it's implemented by adding a notification configuration rule to a bucket that specifies the topic trigger events and notification details this can take up to 30 seconds to begin sending notifications but then it guarantees at least once delivery to cloud pops up the third way of doing it is through cloud functions you can create a simple JavaScript function and make sure that it resides in the same project as your bucket this way you're able to create a lightweight standalone function that's independent of other products and services or MySQL and Postgres Google Cloud Platform offers a fully managed MySQL and Postgres database service that is built on Google's infrastructure this is specifically good for web frameworks structured data OLTP workloads and commonly used for WordPress, blogs websites content management systems CRMs e-commerce applications and business intelligence applications a simple and fully managed MySQL and Postgres database service is not a full replacement for a standalone one this means that it offloads mundane database management tasks to Google but a DBA replacement either for example it doesn't offer query optimization MySQL is the world's largest and most popular web database Postgres allows for a complex database design complex rule set use of stored procedures and procedural languages on the server side Cloud SQL offers MySQL and Postgres database services it doesn't mean that they're offering MySQL compatibility or Postgres compatibility they're offering the exact same community versions or the vanilla versions that you can download and set up yourself on your own local server Cloud SQL is easy to use it doesn't require any software installation and provides horizontal scalability out of the box we should note that the scalability is only on a single node here unlike Cloud Spanner which we will look at in a later lecture you can also access the database instance directly from within GCP via the Google Cloud console or via an IP or a proxy to run a database there's a lot going on under the hood even if you're using managed hosting you should have to worry about upgrades, patches, maintenance etc Cloud SQL automates all your backups, replication patches and updates while ensuring a greater than 99.95 availability anywhere in the world monitoring is done automatically and hooks are provided for external tools like Splunk whereas traditionally you would have to create APIs and do a lot of setup high availability clusterer a feature that's provided with Cloud SQL it provides data redundancy the configuration is made up of a primary instance the master and a secondary instance in a separate zone both of them have to be in the same region data is kept in sync between the instances via semi synchronous replication the failure is initiated if the primary instance is unresponsive for approximately 60 seconds or if the primary zone itself is failing there is such a thing as a replication health check that's done before and after there is something called a replication health that's constantly checked and if the replication lag is over 10 minutes then Cloud SQL will not initiate a failover automatically here in this diagram we can see what the typical setup looks like you can see the primary instance on the left and the failover instance on the right please take note that they're both in different zones but they're both in the same region daily automatic backups can be set up using a backup window this is to minimize any downtime or hinder any performance backups are also snapshots at block level instead of entire terror dumps backups are available on a rolling 7 days by default when you need to restore a backup you can do it on an hourly basis for granularity or you can enable binary logging in which case you can restore up to a point in time failure and up to a certain transaction now let's take a look at a demo in how to set up an instance where we'll see all the features that we just talked about let's take a look at how to create a cloud SQL database typically a MySQL database instance using Google Cloud Platform so for that we're now back on our familiar Google Cloud Platform UI so let's create or select a project first so in this case I have my project already created so I'm going to select Cloud SQL demo which is my project name and we take note of the project ID which is common for all products and services across Google Cloud Platform now we go to the navigation menu and find Cloud SQL which is under storage click on this and now we have a chance to create an instance, press create and you're offered two choices here MySQL and Postgres so we're going to choose MySQL here click on this and now we get a host of options that we can choose so let's start with the instance name it's going to call it demoDB now let's choose a root password for the instance and then we can choose the location of the data center so the region is the general area where you want to place your master database in and then the zone is the specific data center within that region so in this case let's choose East 1B and we can look at some more configuration options here when you click on advanced so here we can select to see if we want to use a different database version with this and we can also choose the type of machine that would be provisioning it so the VM behind it can be changed so currently it says that a one CPU machine has been selected with a memory of 3.75 GB let's select that to something a little more powerful let's say a 4 GB standard database sorry a CPU now we can see that there is a little more performance improvement here but if we look down here we can see that the disk out throughput and the number of IOPS the input output performance is really bad it's all the way down here so when you increase the GPUs you also want to update your storage capacity so if we add a little more let's say another 0 here we can see that the performance here has increased so google cloud platform provides this calculator for us so there are some formulas behind this calculation which we don't have to worry about and we can just play with the numbers here to make sure that we are getting the best possible performance so still that's not very good so if we choose 1000 here that might be a little too much so in this case let's try 400 so that looks a little bit better now the next step is the backups and high availability so as we have seen google cloud platform offers backup and failover services so we can have automatic backups during this specific window that we choose on a daily basis and we can also allow for binary logging so that we can get point in time recovery now high availability this is a feature for data redundancy that google cloud platform also offers so if we say we want to have failover instance also created that would be done at the same time our master is created now we do want to note that a failover database is actually a separate instance of the same size of the database and the specs that's being created in parallel to the master this means that the cost associated is also going to be double now we can also finally choose the maintenance schedule in order to allow google engineers to upgrade our server or apply patches to the operating system and do other server maintenance tasks we can leave this as it is or we can specify a certain day when the workload is going to be less on our database so we press create to have all these options and have our database set up now this will take a few minutes and when the spinning circle is spinning and there's a green check box that's when our database instance has been created we can see that our main database has been created and the failover is still being created now you can also take note that the failover is being created in the same region but in a different zone now we can see that the failover database has also been created you can take a look at the more options for our main database and we can see that from here we can also create a read replica if we want as we discussed in the lecture now if we click on the main database we can see the details of the instance that we just created and we can see the connection details here including the IP address and the instance connection name we can connect to our instance through the cloud shell by clicking on connect using cloud shell click on this and the Google cloud shell opens down here and a connection automatically gets established to our database so you can see that the command that's required here automatically provided to us so if we hit enter that automatically should get us connected to the database now since we are not using a proxy the IP address from where I'm connecting to the database from is being white listed automatically for 5 minutes so that during this time a firewall that's built in is giving me access to it now we press our password hit enter and then now we have our MySQL command prompt from here we can also manage the other aspects of our database instance like connections users databases backups replicas and other operations performance it's good to have one large database but it can get dominated by certain types of applications which demand a high number of reads for this you can create a read replica which is basically a read only copy of your master database the common use cases for this is if you have an analyst team or an analytics application that's creating reports read replicas use asynchronous communication compared to the failover which uses semi synchronous this is so as not to hinder the performance of the primary database so in practice the data on the read replica might be slightly out of date compared to the master you can also have external read replicas and replication from external servers cloud sql proxy let's say when you're working at a starbucks or a less secure public environment and want to connect to your instance you can do so by setting up a firewall and creating temporary rules or use cloud sql proxy the cloud sql proxy works by having a local client called the proxy running in the local environment your application then communicates with the proxy with the standard database protocol used by the database the proxy uses a secure tunnel to communicate sort of like a vpn with its companion process running in the server you can see in the diagram how this actually works it's extremely convenient if you'll have to constantly connect to your instance that offers transactional consistency at global scale what is cloud spanner so it's a relational database service that's provided by google cloud platform it's a sequel like database that provides sql commands and a relational table structure with tables columns rows etc it's specifically built for mission critical applications and high transaction services it offers managed instances with high availability to transparent synchronous built-in data replication the common examples for its use are financial services global supply chain retail and advertising so some of the features of cloud spanner it's a fully managed database service with global scale so it automates and provides synchronous replication with or across regions for availability and it's maintained by site reliability engineers from google it uses nodes for scalability so in our previous examples with cloud sequel and postgres and cloud sequel service that google cloud platform provides that's based purely on a single node where you can have a maximum of 32 cpus so here spanner is based on multiple nodes and each node provides up to 2 terabytes of storage however there's no backup solution available as of now so you'll have to back up spanner by perching all the data from the database spanner is being used and is still being used internally at google for really large products like adwords, google play google maps etc so let's take a look at some of the differences between a traditional database and spanner so even with the traditional databases there are mainly relational databases and non-relational databases which are more like non-sequel no-sequel databases so spanner forms a hybrid between these two between a traditional relational database and a no-sequel database so it provides the benefits of both of them like having the schema strong consistency sequel high availability and automatic replication so let's take a look at what the schema and the data model looks like so the tables look like relational database tables in that they're structured with rows columns and values and they contain primary keys and foreign keys and the tables are related to each other via the keys the data is strongly typed which means that you must define a schema for each database and that schema must specify the data types of each column of each table now here's where the difference comes in cloud spanner divides your data into chunks called splits which can move to different nodes or different servers these splits are automatically managed by cloud spanner and different parts of your table or data is handled by different servers another main difference in cloud spanner is the ability to interleave data an interleave table is a table that you declare to be a child of another table because you want the rows of the child table to be physically stored together with the associated parent so here they are literally interleaved the separate tables so in the diagram we can see three separate tables one called singers two called albums and three called songs so you can see them in with different color codes but here in the grid all three tables and all the data from those three tables are combined this kind of special way of storing data is what helps spanner become efficient and be able to scale horizontally so cloud spanner automatically creates replicas for each database split so there are three types of replicas read write, read only and witness so certain parts of the data in the replicas is owned by different nodes there is a custom algorithm that automatically manages data for any regional configuration cloud spanner maintains three read write replicas each within a different Google cloud platform zone in that region each read write replica contains a full copy of your operational database that is able to serve read write and read only requests cloud spanner uses replicas in different zones so that if a single zone failure occurs your database remains available in multi region configurations they contain two regions that are designated as read write regions each of which contains two read write replicas having more copies of your data makes the data more available to clients that want to read it now when it comes to nodes adding a node does not increase the number of replicas which are fixed for any given configuration even though cloud spanner increases the resources each replica has in that instance adding nodes gives each replica more cpu and ram which increases the replicas throughput that is more reads and writes per second can occur even though cloud spanner replicates across geographically distant locations you can still use cloud spanner as if it were a database and a single machine transactions are guaranteed to be serializable and the order of transactions within the database is the same as the order in which clients observe the transactions to have them committed this is what provides a single database experience now let's take a look at a demo where we create a cloud spanner in this demo we learn how to create a cloud spanner so we are now back here at our familiar google cloud platform interface so as usual the first step is to create or select our project so that's down here click and currently I have selected my project which is cloud spanner demo click on that and now let's navigate to cloud spanner click on the navigation menu and under storage we can see spanner click on that so now let's create an instance by pressing the create instance button let's give our instance a name spanner instance and we can choose either a regional or a multi-regional configuration and we can for now select regional and let's say europe north ok so now we can select the number of nodes and to provision our instance so in this case I'm just going to say three nodes and the cost here you can see the estimation per node and you can calculate that also if you want to know what the performance looks like you can also take a look at how the different number of nodes would matter so let's go ahead and create our instance press create and our instance has been created so the next step is to create a database so let's press the create database button let's have a name for the database demodb continue and now we can create the schema for our database so for that we can create tables so let's add a table our first table by pressing the add table button and here we can have a name for our table let's say singers as in our demo press continue and then we can add columns the first column which would be singer id and let's add another column singer name of type string now we continue and now we can set the primary key to single column and then singer id let's create that and now we have our first table so let's add a second table by pressing on the add table button again and this time let's choose albums and then now we can now we have another option here to interleave this table with another table so let's check this and then it automatically chooses the other table here which would be the parent so in this case the singers table and then we can also enable cascade delete so that if rows are deleted in this table then they will automatically be deleted in the related tables just continue and then we can add some columns here let's say album name or album id first and album name of type string continue and set the primary key and then press the create button so now we have created two tables here to create this schema so now our database has been set up with our two tables here so you can see there's a query tab and that is our albums table with the fields that we selected or created and our singers table with our fields that we created here and we can run queries SQL queries on these tables from the query tab here's a real world example Quizlet app is a popular app that provides quiz and notes for students it's a very demanding app and here's a comparison chart of using a MySQL database versus Cloud Spanner as we can see once the throughput with the number of queries per second reaches 8000 that's 8000 queries per second we can see that MySQL which was previously providing a very low latency suddenly starts to peak and it becomes almost infinite and crashes as with Cloud Spanner you can continue to maintain the latency that you want by just adding nodes so initially there were 9 nodes in green and then that was increased to 15 nodes and then to 30 nodes now this is a database that has of size now this is a database of size 700 gigabytes with 6 billion rows that's a pretty big table if you have 6 billion rows and in order to maintain that that's automatically done by Cloud Spanner using splits and the performance optimization across multiple zones TrueTime is a highly available distributed clock that is provided to applications on all Google servers TrueTime enables applications to generate monotonically increasing time stamps Cloud Spanner uses this to time stamp transactions allowing it to perform consistent reads across an entire database and across multiple cloud regions without blocking writes a traditional database that uses strict two phase locking provides external consistency in such a system every time your application wants to read the most current data which is called a strong read the system acquires a read lock on the data which blocks writes to the data being read with external consistency the system behaves as if all transactions were executed sequentially even though Cloud Spanner actually runs them across multiple servers the external consistency is a stronger property than strong consistency as it doesn't block data during the strong reads this is only available in the multi region configuration terabytes or even pedabytes of data you can use cloud big table to explore and query these types of data time series data such as CPU and memory usage over time for multiple servers marketing data such as purchase histories and customer preferences financial data such as transaction histories stock prices and currency exchange rates internet of things data such as usage reports from energy meters and home appliances or graph data such as information about how users are connected to one another cloud big table is good for low latency read write access it provides native time series support and a high throughput for analytics it's a fully managed no sequel solution that provides wide column database cloud big table is ideal for storing very large amounts of single key data with very low latency it supports high read and write throughput at low latency and is ideal data source for things like map reduce operation cloud big table compresses your data automatically using intelligent algorithm it provides incredible scalability for example it scales in direct proportion to the number of machines in your cluster it provides simple administration cloud big table handles upgrades and restarts transparently and it automatically maintains high data durability to replicate your data you simply have to add a second cluster to your instance and replication starts automatically we will look into this closer in a bit cluster resizing without downtime you can increase the size of a cloud big table cluster for a few hours to handle a large load then reduce the cluster size again all without any downtime after you change cluster size it typically takes just a few minutes under load for cloud big table to balance performance across all of the nodes in your cluster a cloud big table instance is mostly just a container for your clusters and nodes which do all of the real work tables belong to instances not to clusters or nodes the cluster represents the actual big table service each cluster belongs to a single cloud big table instance and an instance can have up to two clusters each cluster is located in a single zone in a production instance there are three or more nodes which are compute resources that big table uses to manage your data each cloud big table zone is managed by a master process which balances workload and data volume within clusters the master splits pizzeria larger tablets in half and merges less smaller tablets together redistributing them between nodes as needed the metrics you can monitor are typically the CPU average utilization utilization of the hardest node and then disk meaning storage utilization percentage per byte client requests go through a front end server before they're sent to a cloud big table node big table nodes are separate from the actual storage here we're not showing a load balancer that usually sits in between the client and the big table node multiple nodes map to data based on the metadata a cloud big table table is shorted into blocks of contiguous rows called tablets this is to help balance the workload of queries tablets are stored on colossus which is Google's file system in sst table format an sst table provides a persistent ordered immutable map from keys to values where both keys and values are arbitrary byte strings each tablet is associated with a specific cloud big table node in addition to the sst table files all writes are stored in colossus shared log as soon as they are acknowledged by cloud big table providing increased durability rebalancing these tablets from one node to another is very fast because the actual data is not being copied when a node fails no data is lost either throughput also scales linearly meaning that nodes add approximately 10,000 queries per second cloud big table stores data in massively scalable tables each of which is a sorted key value map the table is composed of rows each of which typically describes a single entity and columns which contain individual values for each row each row is indexed by a single row key columns that are related to one another are typically grouped together into a column family each column is identified by a combination of the column family and a column qualifier which is a unique name within the column family cloud big table tables are sparse if a cell does not contain any data it does not take up any space each table has only one index the row key there are no secondary indices rows are sorted lexiographically by row key meaning from the lowest to the highest byte string row keys are sorted in big indian or network byte order the binary equivalent of alphabetical order all operations are atomic row level meaning for example if you update two rows in a table it's possible that one row will be updated successfully and the other will fail ideally both reads and writes should be distributed evenly across the row space of the table so in general if you should keep all information for an entity in a single row entity that doesn't need atomic updates and reads can be split across multiple rows splitting across multiple rows is recommended if the entity data is large we will look at this closer in the performance section again cloud big table tables are sparse empty columns don't take up any space as a result it often makes sense to create a very large number of columns even if most columns are empty in most rows now let's take a look at cells each row column intersection can contain multiple cells at different time stamps providing a record of how the stored data has been altered over time so every cell is versioned with a time stamp by default cloud big table periodically writes your tables to remove deleted entries and to reorganize your data so that reads and writes are more efficient this process is known as compaction mutations or changes to a row take up extra storage space because cloud big table stores mutations sequentially so if you update the value in a cell both the original value will be stored on a disk for some amount of time until the data is compacted let's take a look at the schema design for time series data what is time series data whenever you measure something and you record the time together with the measurement you're building a time series examples include pressure over time plot of memory usage on your computer stock prices over time like day, month, year etc storing time series data in cloud big table is a good fit because big table stores data as unstructured columns in rows with the row keys sorted lexiographically retrieving data can be done by querying for a range of row keys so three design patterns for storing time series data so firstly choose meaningful names that are also as short as possible because the size of each name contributes to storage and RPC overhead for row key design in time series you should generally use tall and narrow tables storing one event per row makes it easier to run queries against your data also storing many events per row makes it more likely that the total row size will exceed the recommended maximum by default use new rows instead of column versions using multiple rows with a single version of an event in each row is the simplest way to represent understand and query your data then you have denormalization use two tables each with a row key appropriate to one of the queries this is a good solution because it results in a robust control system ensure that your row key avoids hard spotting when a row key for a time series includes a time stamp all of your writes will target a single note field promotion move fields from the column data into a row key to make writes non-contigious what's salting add an additional calculated element to a row key to artificially make writes non-contigious use reverse time stamps only when necessary prefer reverse time stamps only when your most common query is for the latest values this is because reversing time stamps makes every other query more complex indicates the overall schema so rows can be big but not infinite rows in cloud big table can contain about 100 column families and millions of columns with a 100 megabyte limit on each value stored in a column in general keep rows sizes below approximately 100 megabytes this is more of a guideline than a row rows can be larger than 100 megabytes however if you have many rows larger than this you're going to have performance issues when it comes to columns in general keep column values below approximately 10 megabytes again this is also more of a guideline keep related data in the same table and keep unrelated data in different tables cloud big table is a key value store it's not a relational store it does not support joints nor does it have support for transactions except within a single row so it's best to access data in individual rows or in a set of contiguous rows store data which you will access in a single query in a single column family all of the column qualifiers in a single column family are stored together access together and cache together as a result a query that accesses a single column family might execute more efficiently than a query spanning column families don't exploit a termicity of single rows only operations on a single row are transactional don't exploit a termicity of a single row only operations on on a single row are transactional as we talked about before transactions are also expensive meaning that a system that relies on transactions will not perform as well as it does not changes to data in an existing row should be stored as new separate row not changed in the existing row cloud big table stores data as unstructured columns in rows each row has a row key and row keys are sorted lexiographically there are two commonly used ways to retrieve data cloud big table you can get a single row by specifying the row key you can get multiple rows by specifying a range of row keys these methods are ideal for querying time series data since you often want data for a given time range for example all of the market data for the day or server CPU statistics for the last 15 minutes you can also increment values canonically append and do conditional updates now let's take a look at a demo on creating a big table instance so here we are back to our google cloud platform dashboard let's make sure that we select or create the correct project which is here down and I'm selecting my existing project here you can create a new project if you need to so now I'll go to the navigation menu and look for storage and then big table click on that so here we have no big table instances currently so we create instance click on that here we have some options the first thing we can see on the right here is the monthly cost estimate we can see that it's pretty expensive to create and run a big table instance it's almost $1600 per month for a basic one cluster one terabyte SSD big table instance so here first we can create the instance name let's call it BT instance and we can see the instance ID is also here which is permanent and then we have instance type we can choose either production or development production one will need at least three nodes and the development is a low cost option if you click on that we can see that the price here has changed to $644 from the previous $1600 and so if you choose to use a development version you can always upgrade it to a production version later on and the big difference is that with the development version you will only have one node and you can't increase the number of nodes and then you also can't have a replication cluster set up for a development instance so let's choose production and then you have a storage type we can choose either SSD or an HDD disk for your cluster in most cases the HDD is better because it provides a higher level of performance and this choice is permanent you can't change this later on so here we have the number of clusters and so this is the original cluster so we can select a region here choose Europe North 1 and we have three zones that we can select here from the nodes would be for replication purposes you choose a different zone in terms of nodes you start with three, three is the minimum number of nodes to create a production instance and here we have a choice to add a replication cluster if you need to so for now we just click done and create so now this creates our big table instance and it's created here with three nodes in the Europe North 1C zone when you create an instance with two clusters cloud big table immediately starts to synchronize your data between the zones where the clusters are located creating a separate independent copy of your data in each zone similarly when you add a new cluster to an existing instance cloud big table copies your existing data from the original cluster zone to the new cluster zone then synchronizes changes to your data between the two zones cloud big table replicates the changes automatically when you update data in existing tables new and deleted tables added and removed column families changes to column families garbage collection policy and also when you update the zones by default replication for cloud big table is eventually consistent what does that mean it means that when you write a change to one cluster you will eventually be able to read that change from the other cluster but only after the change is replicated between the clusters cloud big table can also provide read your writes consistency when replication is enabled which ensures that an application will never read data that is older than its most recent writes for read write for new consistencies the app profiles need to be configured for single cluster routing and for strong the second cluster cannot be used except for a failover if a cloud big table cluster becomes unresponsive the application makes it possible for incoming traffic to failover to the instances other cluster provides a detailed look at how you use data in your cloud big table tables it generates visual reports for your tables that break down your usage based on the row keys that you access key visualizer can help you complete the following tasks you can check whether your reads or writes are creating hot spots specific rows find rows that contain too much data look at whether your access patterns are balanced across all of the rows in a table key visualizer automatically generates hourly and daily scans for every table with a table size is over 30 gigabytes or the average reads or writes has at least 10,000 rows per second in the last 24 hours each scan includes a large heat map which you will see in a bit which shows access patterns for a group of row keys over time the core of a key visualizer scan is the heat map which is shown here this shows the value of a metric over time broken down into contiguous ranges of row keys the x-axis of the heat map represents time and the y-axis is the row keys if the metric had a low value for a group of row keys at a point in time the metric is cold and it appears in a dark color a high value which is hot appears into a bright color so white means hot and dark means cold a cloud big table table can have trillions of rows so it's not always practical to report metrics for each individual row instead key visualizer divides all of the row keys into 1,000 contiguous ranges with roughly the same number of row keys in each range these ranges are known as key buckets in general a cluster's performance increases linearly as you add nodes to the cluster for example if you create an SSD cluster with 10 nodes the cluster can support up to 100,000 rows per second for a typical read-only or write-only workload with a 6 millisecond latency for each read or write operation the two different strategies for optimizing data over time distributing the amount of data across nodes so if you have written only a small amount of data to a table cloud big table will store all of the tablets on a single node within your cluster as you can see in the first diagram as more tablets accumulate cloud big table will move some of them to other nodes in the cluster so that the amount of data is balanced more evenly across the cluster as you can see in the second diagram then you have distributing reads and writes evenly across nodes here you can see in the third diagram where there are some cases where you can't avoid accessing certain rows more frequently than others cloud big table helps you deal with these cases by taking reads and writes into account when it balances tablets across nodes for example in the third diagram we can see that 25% of the reads are going to a small number of tablets on the third node so here cloud big table will move those three tablets into a separate node so that they can get a higher amount of performance so when you are testing or troubleshooting performance some guidelines are make sure you are using a production instance with at least 300 gigabytes of data stay below the recommended storage utilization per node before you test run a heavy pre-test for several minutes this step gives cloud big table a chance to balance data across your nodes based on the access patterns it observes run your test for at least 10 minutes this step lets cloud big table further optimize your data it helps ensure that you will test reads from disk as well as cache reads from the memory make sure you are looking at the key visualizer scans for your table the key visualizer tool for big table provides daily scans that show the usage pattern for each table in a cluster key visualizer makes it easy to check whether your usage patterns are causing undesirable results such as hotspots on specific rows or excessive CPU utilization try commenting out the code that performs cloud big table reads and writes if the performance issue disappears then you are probably using cloud big table in a way that results in suboptimal performance ensure that you are creating as few clients as possible creating a client for cloud big table is a relatively expensive operation make sure you are reading and writing many different rows in your table cloud big table performs best when reads and writes are evenly distributed throughout your table which helps cloud big table distribute the workload across all the nodes in your cluster verify that you see approximately the same performance for reads and writes if you find that reads are much faster than writes you might be trying to read row keys that do not exist or a large number of row keys that contain only small number of rows let's take a look at another demo this time let's learn how we can manage tables table instance that we previously created in the previous demo we have seen how to create a big table instance now let's create actual tables and column families and so on we can see the big table instance that was created previously here if you click on this you can see the details of that instance and we can see that and if you click on tables we will see that there should be no tables found at this point so let's go ahead and activate the cloud shell so we can use the command line so we click on this icon activate cloud shell and this opens our gcloud command line tool whenever you are in the gcloud command line make sure that your project ID is correct and that you are currently in the project that you want to operate in so here we are in the correct one and so you can use the CBT command type CBT and hit enter and this will show you a list of commands that you can use for big table you can think of it as a command line for big table CBT and there is a list of different commands that you can use to manipulate your instance so for example you can type CBT list instances and hit enter and this will show you the list of big table instances that you have in this project so here we can see our BT instance that we just created previously so now to create a table we can use these commands here now these commands are also on the slide on the powerpoint previous to this demo so to create a table the command is CBT create table and then the table name let's type that in and I'll call it myTable and we hit enter and our table should be created so if we refresh the tab here we should be able to see our first table we can see here our table called myTable now in order to create a family it's CBT create family and then the table name and the family name so let's do that CBT create family table name and my family name we hit enter now let's create a second family family2 hit enter now to view the information about our table the command is CBT LS and then the table name so let's try that and hit enter and we can see that there are two families called family1 and family2 and we can also see that the GC is set to never this is the garbage collection policy which states how long before the data can be deleted meaning how many versions of the data should be stored so we can set that here as well with this command CBT set GC policy and in the table name and the family name and then max versions equals and then the number of versions we want to keep so let's try that now CBT set GC policy my table family1 max versions equals let's say 5 hit enter oh it should be max versions instead of version max versions equals 5 hit enter and that should do it now let's check the table information again with the previous command CBT LS my table and now we can see that the two families are shown here and the GC policy has been set for 5 only for family1 now in order to delete a family we can just use the CBT delete family table name and family name so let's delete family2 CBT delete delete family table name and family name now let's check the table information CBT LS my table now we can see that only one family is remaining here now when you delete a family it deletes all the data that's stored in that family as well now we can also check the monitoring to see the different statistics about our instance so if you click on monitoring as we have seen in the performance section we can see the CPU utilization CPU utilization of the hottest node and other metrics here that are constantly being checked including the cluster details and storage utilization and the error rates additionally you can check the metrics for instance tables and for the applications so if you click on tables you can specifically see the storage utilization the number of reads writes and the throughputs as well for the specific table and finally we can see the key visualizer by clicking on the key visualizer link here which opens an external link to stack driver so when stack driver loads you can see under resources here you can see the key visualizer for big table if you click on that then you get to the big table key visualizer tool however as it says here no tables in this project meet the minimum size activity requirement as we have seen in the slides that you need at least 30 gigabytes or 10,000 reads or writes for automatic scans and it takes up to 24 hours for new tables to be scanned automatically so you can't really see the heat map at this point but this is where you'd be able to see it now let's take a real world example I'm sure you all heard of Twitter Twitter has around 300 million monthly active users of those users they're generally generating around 100 million tweets per day which turns into something like 300 petabytes of data they use big table for their analytics product they can see things like spend breakdown impressions clicks and all these metrics in real time through UIs for the hundreds of thousands of active advertisers at any given time they also generally have around 5000 QPS queries per second which turns into around a gigabyte per second of data that they're processing at any given time just for query execution at a high SLO uptime of 9997 so things like give me all the impressions that my promoted tweets have generated for the last 7 days show me the spend that my campaigns have had over the last month some together or give me the impressions by day for only app installs by iOS users or how much do people in Japan spend on tweets as you can see big table can handle very demanding tasks when it comes to NoSQL unstructured data cloud big table when you create an instance with two clusters cloud big table immediately starts to synchronize your data between the zones where the clusters are located creating a separate independent copy of your data in each zone similarly when you add a new cluster to an existing instance cloud big table copies your existing data from the original clusters zone to the new clusters zone then synchronizes changes to your data between the two zones cloud big table replicates the changes automatically when you update data in existing tables new and deleted tables added and removed column families changes to column families garbage collection policy and also when you update the zones by default replication for cloud big table is eventually consistent what does that mean it means that when you write a change to one cluster you will eventually be able to read the change from the other cluster but only after the change is replicated between the clusters cloud big table can also provide read your writes consistency when replication is enabled which ensures that an application will never read data that is older than its most recent writes for read write and strong consistencies the app profiles need to be configured for single cluster routing and for strong the second cluster cannot be used except for a failover if a cloud big table cluster becomes unresponsive the application makes it possible for incoming traffic to failover to the instances other cluster key visualizer provides a detailed look at how you use data in your cloud big table tables it generates visual reports for your tables that break down your usage based on the row keys that you access key visualizer can help you complete the following tasks you can check whether your reads or writes are creating hotspots on specific rows find rows that contain too much data look at whether your access patterns are balanced across all of the rows in a table key visualizer automatically generates hourly and daily scans for every table where the table size is over 30 gigabytes or the average reads or writes has at least 10,000 rows per second in the last 24 hours each scan includes a large heat map which you'll see in a bit which shows access patterns for a group of row keys over time the core of a key visualizer scan is the heat map which is shown here this shows the value of a metric over time broken down into contiguous ranges of row keys the x-axis of the heat map represents time and the y-axis is the row keys if the metric had a low value for a group of row keys at a point in time the metric is cold and it appears in a dark color a high value which is hot appears into a bright color so white means hot and dark means cold a cloud big table table can have trillions of rows so it's not always practical to report metrics for each individual row instead key visualizer divides all of the row keys into 1,000 contiguous ranges with roughly the same number of row keys in each range these ranges are known as key buckets in general a cluster's performance increases linearly and adds nodes to the cluster for example if you create an SSD cluster with 10 nodes the cluster can support up to 100,000 rows per second for a typical read-only or write-only workload with a 6 millisecond latency for each read or write operation the two different strategies for optimizing data over time distributing the amount of data across nodes so if you have written only a small amount of data to a table cloud big table will store all of the tablets on a single node within your cluster as you can see in the first diagram as more tablets accumulate cloud big table will move some of them to other nodes in the cluster so that the amount of data is balanced more evenly across the cluster as you can see in the second diagram then you have distributing reads and writes evenly across nodes here you can see in the third diagram where there are some cases where you can't avoid accessing certain rows more frequently than others cloud big table helps you deal with these cases by taking reads and writes into account when it balances tablets across nodes for example in the third diagram we can see that 25% of the reads are going to a small number of tablets on the third node so here cloud big table will move those three tablets into a separate node so that they can get a higher amount of performance so when you're testing or troubleshooting performance some guidelines are make sure you're using a production instance with at least 300 gigabytes of data stay below the recommended storage utilization per node before you test run a heavy pre-test for several minutes this step gives cloud big table a chance to balance data across your nodes based on the access patterns it observes run your test for at least 10 minutes this step lets cloud big table further optimize your data and it helps ensure that you will test reads from disk as well as cached reads from the memory make sure you're looking at the key visualizer scans for your table the key visualizer tool for big table provides daily scans that show the usage pattern for each table in a cluster key visualizer makes it easy to check whether your usage patterns are causing undesirable results such as hot spots on specific rows or excessive CPU utilization try commenting out the code that performs cloud big table reads and writes if the performance issue disappears then you're probably using cloud big table in a way that results in sub-optimal performance ensure that you're creating as few clients as possible creating a client for cloud big table is a relatively expensive operation make sure you're reading and writing the rows in your table cloud big table performs best when reads and writes are evenly distributed throughout your table which helps cloud big table distribute the workload across all the nodes in your cluster verify that you see approximately the same performance for reads and writes if you find that reads are much faster than writes you might be trying to read row keys that do not exist or a large number of row keys that contain only small number of rows let's take a look at another demo this time let's learn how we can manage tables within the big table instance that we previously created in the previous demo we have seen how to create a big table instance now let's create actual tables and column families and so on we can see the big table instance that was created previously here if you click on this you can see the details of that instance and we can see that in if you click on tables we'll see that there should be no tables found at this point so let's go ahead and activate the cloud shell so we can use the command line so we click on this icon activate cloud shell and this opens our gcloud command line tool whenever you're in the gcloud command line make sure that your project ID is correct and that you're currently in the project that you want to operate in so here we're in the correct one and so you can use the cbt command type cbt and hit enter and this will show you a list of commands that you can use for big table you can think of it as command line for big table cbt and there's a list of different commands that you can use to manipulate your instance so for example you can type cbt list instances and hit enter and this will show you the list of big table instances that you have in this project so here we can see our bt instance that we just created previously so now to create a table we can use these commands here now these commands are also on the slide on the powerpoint previous to this demo so to create a table the command is cbt create table and then the table name stripe that in and I'll call it my table and we hit enter and our table should be created so if we refresh the tab here we should be able to see our first table we can see here our table called my table now in order to create a column family it's cbt create family and then the table name and the family name so let's do that cbt create family table name and my family name we hit enter now let's create a second family family 2 and we hit enter now to view the information about our table the command is cbt ls and then the table name so let's try that and hit enter and we can see that there are two families called family 1 and family 2 and we can also see that the GC policy is said to never this is the garbage collection policy which states how long before the data can be deleted and how many versions of the data should be stored so we can set that here as well with this command cbt set GC policy and in the table name and the family name and then max versions equals and then the number of versions we want to keep so let's try that now cbt set GC policy my table family 1 max versions equals let's say 5 hit enter it should be at max versions instead of version max versions equals 5 let's hit enter and that should do it now let's check the table information again with the previous command cbt ls my table and now we can see that the two families are shown here and the GC policy has been set for 5 only for family 1 now in order to delete a family we can just use the cbt delete family table name and family name so let's delete family 2 cbt delete delete family table name and family name now let's check the table information cbt ls my table now we can see that only one family is remaining here now if we delete a family it deletes all the data that's stored in that family as well now we can also check the monitoring to see the different statistics about our instance so if we click on monitoring as we have seen in the performance section we can see the cpu utilization cpu utilization of the hardest node and other metrics here that are constantly being checked including the cluster details and storage utilization and the error rates additionally you can check the metrics for instance tables and for the replications so if we click on tables you can specifically see the storage utilization the number of reads, writes and the throughputs as well for the specific table and finally we can see the key visualizer by clicking on the key visualizer link here which opens an external link to stack driver so when stack driver loads you can see under resources here you can see the key visualizer for big table if you click on that then you get to the big table key visualizer tool however as it says here no tables in this project meet the minimum size or activity requirement as we have seen in the slides that you need at least 30 gigabytes or 10,000 reads or writes for automatic scans and it takes up to 24 hours for new tables to be scanned automatically so you can't really see the heat map at this point but this is where you'd be able to see it now let's take a real world example I'm sure you all heard of Twitter Twitter has around 300 million monthly active users of those users they're generally generating around 100 million tweets per day which turns into something like 300 petabytes of data they use big table for their advertiser analytics product they can see things like spend breakdown, impressions clicks and all these metrics in real time through UIs for the hundreds of thousands of active advertisers at any given time they also generally have around 5000 QPS queries per second which turns into around a gigabyte a second of data that they're processing at any given time just for query execution at a high SLO uptime of 9997 so things like give me all the impressions that my promoted tweets have generated for the last 7 days show me the spend that my campaigns have had over the last month totaled up in sum together or give me the impressions by day for only app installs by iOS users or how much do people in Japan spend on tweets as you can see big table can handle very demanding tasks when it comes to NoSQL unstructured data on highly available structured data at scale it's a scalable fully managed NoSQL document database for your web and mobile applications it's good for semi structured application data hierarchical data and durable key value data you can use cloud data store to store and query these kinds of data product catalogs that provide real time inventory and product details for a retailer user profiles for a customized app experience based on the users past activities and preferences and transactions based on asset properties for example transferring funds from one bank account to another so what are some of the features of data store cloud data store can execute a set of operations where either all succeed or none occur high availability of reads and writes cloud data store runs in Google data centers which uses redundancy to minimize impact from points of failure massive scalability and high performance data store uses a distributed architecture to automatically manage scaling flexible storage and querying data store maps naturally to object oriented and scripting languages and it exposes to applications through multiple clients balance of strong and eventual consistency cloud data store ensures that entity lookups by key and ancestor queries always receives strongly consistent data all other queries are eventually consistent encryption at rest cloud data store automatically encrypts all data before it's written to disk and automatically decrypts the data when read by an authorized user while the cloud data store interface has many of the same features as traditional databases as a no-SQL database it differs from them in a way it describes here data store stores data in entities properties and kinds entities of the same kind can have different properties and different entities can have properties with the same name but different value types if you compare that to a traditional relational database kinds are tables entities are rows properties are columns and keys are the primary key equivalents data store entities of the same kind can have different properties and different entities can have properties with the same name but different value types cloud data store writes scale by automatically distributing data as necessary it reads scale because the only query supported are those performance scales with the size of the result set as opposed to the data set which traditional databases do cloud data store does not support for join operations inequality filtering on multiple properties or filtering on data based on results of a sub query cloud data store is schema less it doesn't require entities of the same kind to have a consistent set of properties when you create a new cloud data store instance you will have the option to create a cloud data store database or a cloud fire store database additionally a cloud fire store database can run in one mode native mode or data store mode cloud data store creates a cloud data store database with existing features and limitations whereas cloud fire store in data store mode uses cloud data store system behavior but accesses cloud fire store storage layer does removing some of the limitations like eventual consistency all cloud data store queries become strongly consistent now transactions are no longer limited to 25 entity groups and writes to an entity group are no longer limited to one per second we should note that these products are still in better so if you need an SLA then you should still opt for the original cloud data store until the cloud fire store in data store mode and the native modes become GA and remove the beta status now let's take a look at a demo where we can learn how to create a data store in this demo we will learn how to create a data store so now we are back to our Google cloud platform UI let's make sure we have selected the correct project or create one if necessary so in this case my project name is cloud data store demo click on that and I have my project name and project ID here which is the same across all Google products and services so now we go to the navigation menu and click on that and browse to storage and we see data store click on that and this takes us to the main cloud data store page so here we can see there are entities, dashboard indexes and admin so let's go ahead and create an entity by clicking on create entity we can leave that name space as default and let's create a kind called pens let's add a property called blue pen of type string and with the value big blue pen let's add a second property called on sale of type boolean and have the value as false so now let's go ahead and create our entity by pressing on the create button and now on the main page we can see that our entity has been created now let's go to the query query by gql click on this here we can query our data stores we can do this by using a sequel like language so we can do something like select star from pens now this has to be case sensitive here so you have to use a capital letter so and you click on run query and we can see one result here with the data that we specified we can also use a where clause to filter our results like for example say where on sale equals true now when you run the query we should get no results let's change the true to false and run the query again and we can see our result here so like we discussed in the lecture the kind which is pens here acts as the table in a real database the blue pen and on sale act as the column names and these are property names and you can see that the ID is the the primary key in this case now let's take a look at a real world example Pokemon Go by Niantic is a popular phone game it's a very demanding augmented reality game that requires GPS location and player data for every user at its peak the game had over 20 million daily active users these users were all demanding the servers for getting player data as well as their GPS location and the opponent's GPS location in order to do battles for this on the server side Niantic was using over 10,000 CPU cores in a Kubernetes solution in order for the servers to handle this traffic in real time and serve the GPS locations as well as keep track of player data a traditional relational database would not be sufficient the latency would be too high so in order for that to work you would need not just a no-SQL database but also a solution that would be able to distribute the load in real time and scale efficiently across multiple geographies so this is where Datastore came in and was the perfect fit Compute Engine, Virtual Machine Instances GKE Clusters and App Engine Flex Instances and other cloud based resources and services a VPC stands for a Virtual Private Cloud a Virtual Private Cloud is a logical isolation of a section of your cloud it's a cloud equivalent of your traditional network that you would run in and your data center but with the extremely important added benefits of running on infrastructure that is highly optimized for scale, performance and reduced network costs using Google Cloud means you're using the same online network infrastructure that powers all of Google services like Search, Maps, YouTube and Gmail VPC is a programmable interface for distributed networking functions on this interface it allows you the capability of creating virtual machines anywhere in the world in many regions and having private connectivity between all of these virtual machines by default some of the features include a single VPC across all regions no peering of VPCs is required and no cross region VPNs are required so you don't need to give your Virtual Private Machines public IPs in order for these to communicate with each other even though they are across different regions a single global VPC that is shareable in nature allows you to administer your network centrally as well while allowing thousands of developers and development teams in the organization to create separate projects and to create and manage compute resources the traffic that goes from the virtual machines to the storage for example would use the network backbone and would not use the internet so the data is encrypted while in transit there is a product called cloud armor that Google also provides which secures the VPC perimeter distributed firewalls and distributed networks with no choke points using cloud router high bandwidth and availability using the Andromeda control plane which supports Kubernetes via GKE here are some of the VPC concepts VPC networks and their associated routes firewalls, rules are global resources they are not associated with any particular region or zone a network must have at least one subnet before you can use it subnets are regional resources though each subnet defines a range of IP addresses when you create an instance or resource you select a zone a network and a subnet GCP assigns the instance an IP address from the range of available addresses in the subnet the traffic to and from instances can be controlled with the network firewall rules so here we can see on the left a traditional VPC and on the right what a Google VPC looks like so in the traditional version you can see two separate regions US west and US east each of these regions have a separate network and an application server on each of their networks you can see that in order for those two application servers to communicate there needs to be a VPN gateway that provides communication across the internet that's where the public transport happens and that's how traffic is routed between these two completely separate VPCs on the right you can see the Google VPC where you have two separate subnets and two separate VPCs with two separate application servers but they're both connected with the Google cloud platform a single Google cloud VPC can span multiple regions without communicating across the public internet so the containers where the VPC lives is a global construct for on-premises scenarios you can share a connection between VPC and on-premises resources with all regions in a single VPC so you don't need a connection in every region the Google cloud platform offers two types of VPC networks the auto mode and the custom mode when an auto mode network is created one subnet from each region is automatically created within its predefined IP ranges and new subnets are added when new regions become available each project starts with a default auto mode network the predefined IP ranges of the subnets do not overlap with IP ranges you would use for different purposes except for manually added ones when a custom mode network is created no subnets are automatically created this type of network provides you with a complete control over its subnets and IP ranges you get to decide which subnets to create in which regions to choose and using IP ranges you specify for example the price of creating a Kubernetes container cluster involves selecting a zone or region depending on the cluster type a network and a subnet the subnets available for selection are restricted to those in that selected region now let's do a demo where we create an auto mode network in this demo we'll look at how to create a VPC now we're back here at the Google Cloud Platform dashboard that should be familiar to all of us by now so let's select or create our project here by clicking down and in this case selecting my project here VPC and subnets demo or you can also create a new project if you need to so here I can go to the navigation menu and select VPC under the networking options which is here VPC network and VPC networks click on this so this brings us to the dashboard for VPC networks so on the left here we can see different tabs for external IP addresses firewall rules routes VPC network peering and shared VPC for now we'll create a VPC network by pressing this button here so to create a VPC network first we can choose a name we'll call it demo VPC description and for the subnets we can either choose custom or automatic so if we choose automatic then we get a message that says that the ranges are going to be assigned automatically based on the region here and if you do choose custom then we can actually create multiple subnets where we can choose a name for a subnet and then the region and then the IP address range we can also create secondary IP ranges here by clicking on this we can choose if we want to allow private google access if this is on that means all the VMs in this network will be able to communicate with each other without requiring external IP addresses for now we'll just choose automatic also when you choose custom you get to choose whether you want dynamic routing to be regional or global so for now let's just choose automatic and if you scroll down we get to firewall rules here some of these are already selected for us these can be either enabled or disabled and these two cannot be disabled these are the deny all incoming and allow all outgoing the ingress and egress rules that come with by default so here allowICMP is just for error reporting it's a network layer protocol this one is for internal communications and RDP is for remote protocol SSH is for secure shell and so we can choose to allow these and we don't need to provide a DNS server policy at this time so we can just press create to create our VPC so here you can see that our VPC is being created ok so now it's created and we can see that the different ranges are also created for the different regions if we click on the VPC you can get the details about our VPC network so here we can see there are multiple tabs here we can see the static internal IP addresses if there are any our firewall rules routes VPC network peering and if there are any private service connections subnets and IP ranges when you create a subnet you must define a primary IP address range and optionally up to 5 secondary IP ranges the primary address range can be used for virtual machine primary internal IP addresses VM alias IP addresses and IP addresses of internal load balancer the secondary IP ranges are used only for alias IP addresses each subnet has 4 reserved IP addresses in its primary IP range there is a network default gateway second to last reservation and a broadcast address let's take a look at a demo on subnets where we can learn how to add subnets and list and describe subnets GCP resources such as computer engine, VM instances forwarding rules GKE containers and app engine rely on IP addresses to communicate if you have multiple services running on a single VM instance you can give each service a different internal IP address using the alias IP ranges the VPC network forwards packets designed for each alias IP to the corresponding virtual machine we learn more in depth about IP addresses in a future lecture you can add multiple network interfaces to a VM instance each interface resides in a unique VPC network multiple network interfaces enables a network appliance VM to act as a gateway for securing traffic among different VPC networks or to and from the internet this diagram shows a situation where we can see the interplay between the two features in this case VM1 has a network interface in both network A and network B network B is paired with another network C IP1 is where the network interface with IP1 address from network A which is the primary interface IP2 is a network interface with IP2 address from network B which is a secondary interface so here IP3 can send traffic to IP2 because IP2 is a network B and network B routes are automatically propagated to network C when the two networks appeared we will look at peering in a bit but for IP2 to send network traffic to IP3 you will have to configure a policy routing on the IP2 interface IP1 flows are not installed in network C so therefore network C cannot access IP1 routes tell VM instances and VPC network how to send traffic from an instance to a destination either inside the network or outside of GCP each VPC network comes with some system generated routes to route traffic among its subnet and send traffic from eligible instances to the internet there are two types of system generated routes the default route which defines a path for traffic to leave the VPC network and provides general internet access to VMs and provides the typical path for the private Google access a subnet route is created for each of the IP ranges associated with a subnet every subnet has at least one subnet route for its primary IP range and an additional subnet routes created for a subnet if you add secondary IP ranges the subnet routes define paths for traffic for each VMs that use these subnets each VPC network also has an associated dynamic routing mode that controls the behavior of all its cloud routers each VPC network implements a distributed virtual firewall that you can configure firewall rules allow you to control which packets are allowed to travel to which destinations every VPC network has two implied firewall rules that block all incoming connections and allow all outgoing connections the default network includes a number of firewall rules in addition to the implied ones including the default allow internal rule which permits instance to instance communication within the network rules that come with the default network are also presented as options for you to apply to new automo networks that you create using the GCP console for instance to have outgoing internet access firewall rules must allow egress traffic from the instance and it must have an external IP address once the VPC is set up and secured the next thing to do is to connect your VPC running in the cloud with your on premises data center so there are multiple ways of doing this if bandwidth and the overhead of IPsec encryption is not a problem then you can just use a VPN you can either create static routes to your VPN tunnel to basically allow you to statically specify the prefixes that should be routed to your on-prem or you can use the cloud router which runs as a BGP daemon to exchange routes with your on-prem in a dynamic way so if bandwidth is a concern though you can use interconnects if you would like to directly peer with Google if you have a public ESN and you want to basically connect with them at one of the POPs or points of presence then you can use a dedicated interconnect if you're however doing the same thing using a carrier's infrastructure and you're paying the carrier for the service then you can use something called a partner interconnect Google Cloud Platform load balancing gives you the ability to distribute load balanced compute resources in single or multiple regions to meet your high availability requirements to put your resources behind a single anycast IP and to scale the resources up and down with order scaling there are three types of load balancers the global external load balancer for mostly HTTPS SSL proxy and TCP proxy and then two regional ones one for external network load balancing and one for internal load balancing shared VPC so you can share a VPC network from one project called a host project to another project in your GCP organization you can grant access to entire shared VPC networks or select subnets this allows you to provide centralized control over a common network while maintaining organizational flexibility shared VPC is especially useful in large organizations now a shared VPC was designed with the intention of having private connectivity between virtual machines in one organization but if you want to connect two VPCs in two different organizations then you can use VPC peering with VPC network peering all communication happens using private IP addresses these are subject to firewall rules and VM instances and each peer network can communicate with one another without using external IP addresses peer networks only share their subnet routes the network administration for each peer network is unchanged the network and security admins for one network do not automatically get those roles for the other network in the peering relationship if two networks from different projects appeared project owners editors and compute instance admins in one project do not automatically receive those roles in the project that contains the other network so let's take a look at an example here we have a shared VPC and we have four projects project one is the host project the VPC network peering allows peering with a shared VPC a shared VPC host project is a project that allows other projects to use one of its networks so the network shared VPC is in a shared VPC network in host project P1 service projects P3 and P4 are able to attach VM instances to network shared VPC the peers with network A as a result VM instances in shared VPC service projects that are using the network which is the VM3 and VM4 have private internal IP connectivity with all endpoints associated to network A VM instances associated to network A will have private internal IP connectivity with any endpoints associated to network SVPC regardless of whether those endpoints live in the host project or in a service project it's also possible to set up VPC network peering between two shared VPC networks and how they communicate with each other in GCP you can assign an IP address to certain resources for example you can assign an internal and external IP address to compute engine virtual machine instances similarly you can assign an internal or external IP address to a forwarding rule or external load balancing respectively to communicate between instances on the same virtual private cloud network you can use an instances internal IP address however to communicate with the internet you must always have the instances external IP address unless you have some kind of a proxy configured use the external IP address to connect to instances outside the same VPC network unless those networks are connected by a VPN both external and internal IP addresses can be either ephemeral or static a forwarding rule is required for network, global and internal load balancing the forwarding rule must have an external or internal IP address depending on the load balancer that you are using you can assign an external IP address to an instance or a forwarding rule if you need to communicate with the internet sources from outside the GCP VPC network can address a specific resource by the external IP address as long as firewall rules allow the connection only resources with an external IP address can send and receive traffic directly to and from outside the network Compute Engine supports two types of external IP addresses static external IP addresses and ephemeral external IP addresses you can reserve a static external IP address which assigns the address to your project indefinitely until you explicitly release it this is useful if you are dependent on a specific IP address for your service and need to prevent others from being able to use the address you can reserve a new static external IP address or promote an existing ephemeral external IP address to a static external IP address now within static external IP addresses they can either be regional or global a regional one allows resources of that region or resources of zones within that region to use the IP address in this case, VM instances and regional forwarding rules can use a regional static IP address now a global static external IP address these are available only to global forwarding rules these are used for global load balancing you can't assign these to regional or zonal resources now an ephemeral external IP address is one that does not persist beyond the life of the resource so when you create an instance or a forwarding rule without specifying an IP address the resource is automatically assigned an ephemeral external IP address so for VM instances if you stop the instance the IP address is also released so whether you restart it or stop it it's the same thing now every VM instance has one primary internal IP address that is unique to that VPC network you can assign a specific internal IP address when you create a VM instance or you can reserve a static internal IP address for your project and assign that address to your resources so if you do not specify an address compute engine assigns one automatically in either case the address must belong to the IP range of that subnet so if it's an auto mode VPC network the address comes from the region's subnet if it's custom you must specify which subnet the IP address will come from if your network is a legacy network then the IP address is assigned from the network's global internal IP range the engine supports two types of internal IP addresses static internal IP addresses and fmrl internal IP addresses static ones are assigned to a project long term until they are explicitly released and they remain attached to the resource until they are detached for VM instances static internal IP addresses remain attached to stopped instances until they are removed fmrl IP addresses remain attached to a VM instance only until the VM is stopped and restarted or the instance is terminated for internal load balancers you can assign a static internal IP address specify an explicit fmrl internal IP address or let GCP assign an internal IP address randomly now let's take a look at a demo in assigning a static external IP address to a VM instance in this demo we'll learn how to assign a static external IP address while creating a VM instance so we're back here in our google cloud platform dashboard so here first we select or create a new project by clicking down here in this case I'm choosing the project IP addresses demo click on that make sure we're in the right project so now we go to the navigation menu and select compute compute engine and VM instances click on that this takes us to the VM instances page so for the first time you would create a new VM instance you press create this takes us to our VM instance creation page so here we have we have several options so first we just have the typical VM name and the region machine type container access and so on down here we have a section called management security disks networking and sole tenancy so if you click on this we can see more options here and here we can see the networking tab so we click on that and we can see the networking details here so here we can see that the default network interface has this IP address and if you click on this we can make changes to the network interface that's being assigned to our VM so here we can see that the default subnet is slash twenty and the primary internal IP address is ephemeral that's automatically being assigned and the external IP address is also ephemeral because we haven't specified anything otherwise so in order to change this we can click on the down arrow here and press create new IP address now if you already have reserved IP addresses they will show up in this list so here I'm creating a new one so this opens a new window where I can assign or I can reserve a new static IP address so I'll call this VM IP1 and reserve so now we can see that an external IP address 34 73.80.7 has been assigned here so it has been reserved for us to be able to select this here and when we press create this creates our VM so here now our VM has been created with an external IP address of 34.73 80.7 and this will allow us to connect to the outside world now let's take a look at a demo on changing the external IP address in this demo we'll learn how to change the external IP address for a VM instance in the previous demo we have seen how we can assign an external static IP address to a VM instance while creating it now we have already created the VM instance now if we need to change this we can click on the VM instance so we can get the details of that instance and then press the edit button here now we can make changes here under network interfaces we can see the default network interface card Nick01 and we click on this to make changes here and here we can see the external IP address press the down key and we can set this to ephemeral if we need to so here we get a warning saying that the static IP address will be detached from the VM so now we scroll down and press save now if we go back to the VM details we can see that the IP address has changed here to 473.16.125 now we can see that specifically when you click on the details here that this is an ephemeral IP address meaning that it's a default that has been allocated by Google Cloud Platform and if you stop or restart this VM instance this IP address will be dropped and a new one will be assigned so it's not static anymore now let's take a look at a demo on new static internal IP addresses in this demo we'll learn how to create a static internal IP address and assign it to a new VM instance so we're back here in our Google Cloud Platform dashboard make sure you select your IP project or create a new one go to the navigation menu and we go to our compute engine VM instances so here we can see a list of VM instances in this project so let's create a new instance pressing create instance and here let's choose a region let's say Europe North One and the zone North One A scroll down and here under management security, disks, networking and soul tendency click on this now here we can see the networking details when we click on the networking tab and click on edit network interfaces back here from as we have seen in the previous demo and in this case we're going to choose the network here the demo VPC the demo VPC is a virtual private cloud network that was created in a previous lecture so this has an allocated IP addresses and a subnet and in this case we can choose a primary internal IP address here so here an affirmable one is automatically assigned when nothing is specified so you press down here and then reserve static internal IP address click on this so let's give this a name int VM and we'll allow this to be assigned automatically click on reserve and now the address is being reserved for us so now we have this IP address <IP_ADDRESS> now this is an internal IP address which means that only resources that are within the network can see and communicate with this VM with this IP address however it's static meaning that it won't be dropped or changed unless we specifically detach it from the resource so we can press done and create so now we have created a VM instance with a static internal IP address alias IP ranges Google cloud platform alias IP ranges let you assign ranges of internal IP addresses as aliases to a virtual machines network interfaces this is useful if you have multiple services running on a VM and you want to assign each service a different IP address for a VM network interface the alias IP must be from the same subnet resource that provides the IP address for the primary network interface you can't select a primary or secondary CIDR range from another subnet resource CIDR stands for classless inter domain routing and it is used for IP addresses and routing alias IP ranges also work with Kubernetes parts when an alias IP ranges are configured GCP automatically installs VPC network routes for primary and alias IP ranges for the subnet of the primary network interface card your container orchestrator does not need to specify VPC network connectivity for these routes so this simplifies the routing traffic managing your containers you don't have to add a route for every IP alias and you don't have to take route quotas into account either so using alias IP ranges container IP addresses can be allocated from a second CIDR range you can use this to separate the infrastructure from the services and you can use separate firewall controls for each of these ranges GCP automatically configures internal DNS for the primary IP of the primary interface of every VM instance GCP however does not associate alias IP addresses on the primary interface with the host name and it does not associate any IP addresses of secondary interfaces with the host name firewall source tags are not supported for alias IP addresses so when you configure source tags in firewall rules the source tag matches the VM primary IP address but not the alias IP addresses in a static route the next hop IP address must be the primary IP address of the virtual machine instance an alias IP address is not supported as the next hop IP address in VPC peering you're allowed to peer to VPC networks so that the VMs in the two networks can communicate via internal private IP addresses we can see in the diagram here of network peering both primary and secondary IP ranges of a subnet are reachable by VM instances in a peer network subnet overlap checks across peer networks ensure that primary and secondary ranges do not overlap with any peer ranges now when it comes to alias IP addresses within auto mode VPC networks you can allocate alias IP ranges from the automatically created primary CIDR range or you can add a secondary range to allocate the IP addresses from the alias IP ad