<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Notebook is a revised version of notebook from Sara Robinson and Ivan Chueng
E2E ML on GCP
Step1:  Restart the kernel
After you install the additional packages, you need to restart the notebook kernel so it can find the packages.
Step2:  Before you begin
GPU runtime
Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select Runtime > Change Runtime Type > GPU
Set up your Google Cloud project
The following steps are required, regardless of your notebook environment.
Select or create a Google Cloud project. When you first create an account, you get a $300 free credit towards your compute/storage costs.
Make sure that billing is enabled for your project.
Enable the following APIs
Step3:  Get your project number
Now that the project ID is set, you get your corresponding project number.
Step4:  Region
You can also change the REGION variable, which is used for operations
throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.
Americas
Step5:  Timestamp
If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial.
Step6:  Authenticate your Google Cloud account
If you are using Vertex AI Workbench Notebooks, your environment is already authenticated. Skip this step.
If you are using Colab, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.
Otherwise, follow these steps
Step7:  Create a Cloud Storage bucket
The following steps are required, regardless of your notebook environment.
When you initialize the Vertex AI SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.
Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.
Step8:  Only if your bucket doesn't already exist
Step9:  Finally, validate access to your Cloud Storage bucket by examining its contents
Step10:  Set up variables
Next, set up some variables used throughout the tutorial.
Import libraries and define constants
Step11:  Initialize Vertex AI SDK for Python
Initialize the Vertex AI SDK for Python for your project and corresponding bucket.
Step12:  Download and prepare the prebuilt GloVe embeddings
The GloVe embeddings consists of a set of pre-trained embeddings. The embeddings are split into a "train" and "test" splits.
You create a Vertex AI Matching Engine index from the "train" split, and use the embedding vectors in the "test" split as query vectors to test the index.
Note
Step13:  Load the embeddings into memory
Load the GloVe embeddings into memory from a HDF5 storage format.
Step14:  Save the train split in JSONL format
Next, you store the embeddings from the train split as a JSONL formatted file. Each embedding is stored as
Step15:  Store the JSONL formatted embeddings in Cloud Storage
Next, you upload the training data to your Cloud Storage bucket.
Step16:  Create Matching Engine Index
Next, you create the index for your embeddings. Currently, two indexing algorithms are supported
Step17:  Update the Index
Next, you update the index with a new embedding -- i.e., insertion.
Create update delta file
First, you make a JSONL file with the embeddings to update. You use synthetic data -- in this case, all zeros -- for existing embedding with id of 0. You then upload the JSONL file to a Cloud Storage location.
Step18:  Update the index
Next, you use the method update_embeddings() to incrementally update the index, with the following parameters
Step19:  Setup VPC peering network
To use a Matching Engine Index, you setup a VPC peering network between your project and the Vertex AI Matching Engine service project. This eliminates additional hops in network traffic and allows using efficient gRPC protocol.
Learn more about VPC peering.
IMPORTANT
Step20:  Create the VPC connection
Next, create the connection for VPC peering.
Note
Step21:  Check the status of your peering connections.
Step22:  Construct the full network name
You need to have the full network resource name when you subsequently create an Matching Engine Index Endpoint resource for VPC peering.
Step23:  Create an IndexEndpoint with VPC Network
Next, you create a Matching Engine Index Endpoint, similar to the concept of creating a Private Endpoint for prediction with a peer-to-peer network.
To create the Index Endpoint resource, you call the method create() with the following parameters
Step24:  Deploy the Matching Engine Index to the Index Endpoint resource
Next, deploy your index to the Index Endpoint using the method deploy_index() with the following parameters
Step25:  Create and execute an online query
Now that your index is deployed, you can make queries.
First, you construct a vector query using synthetic data, to use as the example to return matches for.
Next, you make the matching request using the method match(), with the following parameters
Step26:  Create brute force index for calibration
The brute force index uses a naive brute force method to find the nearest neighbors. This method uses a linear search and thus not efficient for large scale indexes. We recommend using the brute force index for calibrating the approximate nearest neighbor (ANN) index for recall, or for mission critical matches.
Create the brute force index
Now create the brute force index using the method create_brute_force_index().
To ensure an apples to apples comparison, the distanceMeasureType and featureNormType, dimensions of the brute force index should match those of the production indices being tuned.
Step27:  Update the index
For apples to apples comparison, you perform the same incremental update to the brute force index as you did for the Tree AH index.
Step28:  Deploy the brute force index to the IndexEndpoint resource
Next, you deploy the brute force index to the same IndexEndpoint.
Note
Step29:  Calibration
Now your ready to do calibration. The production version of the index uses an approxiamation method, which means it may have less than perfect recall when compared to the slower exact match (brute force) method.
Get test results for both indexes
First, using the GloVe test embeddings, you make the identical request to both indexes.
Step30:  Compute Recall
Finally, you determine from the results the percentage of exact matches are recalled from the production index. You can subsequently use this information to tune the deployment of the production index.
Step31:  Cleaning up
To clean up all Google Cloud resources used in this project, you can delete the Google Cloud
project you used for the tutorial.
You can also manually delete resources that you created by running the following code.
<ASSISTANT_TASK:>
Python Code:
import os
# The Vertex AI Workbench Notebook product has specific requirements
IS_WORKBENCH_NOTEBOOK = os.getenv("DL_ANACONDA_HOME")
IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(
    "/opt/deeplearning/metadata/env_version"
)
# Vertex AI Notebook requires dependencies to be installed with '--user'
USER_FLAG = ""
if IS_WORKBENCH_NOTEBOOK:
    USER_FLAG = "--user"
! pip3 install --upgrade google-cloud-aiplatform {USER_FLAG} -q
! pip3 install -U grpcio-tools {USER_FLAG} -q
! pip3 install -U h5py {USER_FLAG} -q
Explanation: Notebook is a revised version of notebook from Sara Robinson and Ivan Chueng
E2E ML on GCP: MLOps stage 6 : serving: get started with Vertex AI Matching Engine
<table align="left">
  <td>
    <a href="https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage6/get_started_with_matching_engine.ipynb">
      <img src="https://cloud.google.com/ml-engine/images/github-logo-32px.png" alt="GitHub logo">
      View on GitHub
    </a>
  </td>
    <td>
        <a href="https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/communityml_ops/stage6/get_started_with_matching_engine.ipynbb">
        <img src="https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt="Colab logo"> Run in Colab
        </a>
  </td>
  <td>
    <a href="https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/community/ml_ops/stage6/get_started_with_matching_engine.ipynbb">
      <img src="https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32" alt="Vertex AI logo">
      Run in Vertex Workbench
    </a>
  </td>
</table>
Overview
This tutorial demonstrates how to use Vertex AI Matching Engine service. This Cloud AI service is a appropriamate nearest neighbor (ANN) index and matching service for vectors (i.e., embeddings), with high scaling and low latency.
the GCP ANN Service.  The service is built upon Approximate Nearest Neighbor (ANN) technology developed by Google Research.
There are several levels of using this service.
no code
Demomstrated in this tutorial. The user brings their own embeddings for indexing and querying.
low code
The user constructs embeddings using a Vertex AI pre-built algorithm: Swivel and TwoTowers.
high code
The user configures the serving binary how to generate embeddings from the model, indexing and querying, using Vertex AI Explanations by Examples
Learn more about Vertex AI Matching Engine
Embeddings
The prebuilt embeddings used for this tutorial is the GloVe dataset.
"GloVe is an unsupervised learning algorithm for obtaining vector representations for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space."
Objective
In this notebook, you will learn how to create Approximate Nearest Neighbor (ANN) Index, query against indexes. 
The steps performed include:
Create ANN Index.
Create an IndexEndpoint with VPC Network
Deploy ANN Index
Perform online query
Deploy brute force Index.
Perform calibration between ANN and brute force index.
Costs
This tutorial uses billable components of Google Cloud:
Vertex AI
Cloud Storage
Learn about Vertex AI
pricing and Cloud Storage
pricing, and use the Pricing
Calculator
to generate a cost estimate based on your projected usage.
Installation
Install the packages required for executing this notebook.
End of explanation
# Automatically restart kernel after installs
import os
if not os.getenv("IS_TESTING"):
    # Automatically restart kernel after installs
    import IPython
    app = IPython.Application.instance()
    app.kernel.do_shutdown(True)
Explanation: Restart the kernel
After you install the additional packages, you need to restart the notebook kernel so it can find the packages.
End of explanation
PROJECT_ID = "[your-project-id]"  # @param {type:"string"}
if PROJECT_ID == "" or PROJECT_ID is None or PROJECT_ID == "[your-project-id]":
    # Get your GCP project id from gcloud
    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null
    PROJECT_ID = shell_output[0]
    print("Project ID:", PROJECT_ID)
! gcloud config set project $PROJECT_ID
Explanation: Before you begin
GPU runtime
Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select Runtime > Change Runtime Type > GPU
Set up your Google Cloud project
The following steps are required, regardless of your notebook environment.
Select or create a Google Cloud project. When you first create an account, you get a $300 free credit towards your compute/storage costs.
Make sure that billing is enabled for your project.
Enable the following APIs: Vertex AI APIs, Compute Engine APIs, and Cloud Storage.
Enable the Service Networking API.
Enable the Cloud DNS API.
If you are running this notebook locally, you will need to install the Cloud SDK.
Enter your project ID in the cell below. Then run the  cell to make sure the
Cloud SDK uses the right project for all the commands in this notebook.
Note: Jupyter runs lines prefixed with ! as shell commands, and it interpolates Python variables prefixed with $.
Set your project ID
If you don't know your project ID, you may be able to get your project ID using gcloud.
End of explanation
shell_output = ! gcloud projects list --filter="PROJECT_ID:'{PROJECT_ID}'" --format='value(PROJECT_NUMBER)'
PROJECT_NUMBER = shell_output[0]
print("Project Number:", PROJECT_NUMBER)
Explanation: Get your project number
Now that the project ID is set, you get your corresponding project number.
End of explanation
REGION = "[your-region]"  # @param {type: "string"}
if REGION == "[your-region]":
    REGION = "us-central1"
Explanation: Region
You can also change the REGION variable, which is used for operations
throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.
Americas: us-central1
Europe: europe-west4
Asia Pacific: asia-east1
You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.
Learn more about Vertex AI regions.
End of explanation
from datetime import datetime
TIMESTAMP = datetime.now().strftime("%Y%m%d%H%M%S")
Explanation: Timestamp
If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial.
End of explanation
# If you are running this notebook in Colab, run this cell and follow the
# instructions to authenticate your GCP account. This provides access to your
# Cloud Storage bucket and lets you submit training jobs and prediction
# requests.
import os
import sys
# If on Vertex AI Workbench, then don't execute this code
IS_COLAB = False
if not os.path.exists("/opt/deeplearning/metadata/env_version") and not os.getenv(
    "DL_ANACONDA_HOME"
):
    if "google.colab" in sys.modules:
        IS_COLAB = True
        from google.colab import auth as google_auth
        google_auth.authenticate_user()
    # If you are running this notebook locally, replace the string below with the
    # path to your service account key and run this cell to authenticate your GCP
    # account.
    elif not os.getenv("IS_TESTING"):
        %env GOOGLE_APPLICATION_CREDENTIALS ''
Explanation: Authenticate your Google Cloud account
If you are using Vertex AI Workbench Notebooks, your environment is already authenticated. Skip this step.
If you are using Colab, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.
Otherwise, follow these steps:
In the Cloud Console, go to the Create service account key page.
Click Create service account.
In the Service account name field, enter a name, and click Create.
In the Grant this service account access to project section, click the Role drop-down list. Type "Vertex" into the filter box, and select Vertex Administrator. Type "Storage Object Admin" into the filter box, and select Storage Object Admin.
Click Create. A JSON file that contains your key downloads to your local environment.
Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell.
End of explanation
BUCKET_NAME = "[your-bucket-name]"  # @param {type:"string"}
BUCKET_URI = f"gs://{BUCKET_NAME}"
if BUCKET_URI == "" or BUCKET_URI is None or BUCKET_URI == "gs://[your-bucket-name]":
    BUCKET_NAME = PROJECT_ID + "aip-" + TIMESTAMP
    BUCKET_URI = "gs://" + BUCKET_NAME
Explanation: Create a Cloud Storage bucket
The following steps are required, regardless of your notebook environment.
When you initialize the Vertex AI SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.
Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.
End of explanation
! gsutil mb -l $REGION $BUCKET_URI
Explanation: Only if your bucket doesn't already exist: Run the following cell to create your Cloud Storage bucket.
End of explanation
! gsutil ls -al $BUCKET_URI
Explanation: Finally, validate access to your Cloud Storage bucket by examining its contents:
End of explanation
import google.cloud.aiplatform as aiplatform
import h5py
Explanation: Set up variables
Next, set up some variables used throughout the tutorial.
Import libraries and define constants
End of explanation
aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)
Explanation: Initialize Vertex AI SDK for Python
Initialize the Vertex AI SDK for Python for your project and corresponding bucket.
End of explanation
! gsutil cp gs://cloud-samples-data/vertex-ai/matching_engine/glove-100-angular.hdf5 .
Explanation: Download and prepare the prebuilt GloVe embeddings
The GloVe embeddings consists of a set of pre-trained embeddings. The embeddings are split into a "train" and "test" splits.
You create a Vertex AI Matching Engine index from the "train" split, and use the embedding vectors in the "test" split as query vectors to test the index.
Note: While the data split uses the term "train", these are pre-trained embeddings and thus are ready to be indexed for search. The terms "train" and "test" split are used just to be consistent with usual machine learning terminology.
End of explanation
h5 = h5py.File("glove-100-angular.hdf5", "r")
train = h5["train"]
test = h5["test"]
print(train)
Explanation: Load the embeddings into memory
Load the GloVe embeddings into memory from a HDF5 storage format.
End of explanation
with open("glove100.json", "w") as f:
    for i in range(len(train)):
        f.write('{"id":"' + str(i) + '",')
        f.write('"embedding":[' + ",".join(str(x) for x in train[i]) + "]}")
        f.write("\n")
Explanation: Save the train split in JSONL format
Next, you store the embeddings from the train split as a JSONL formatted file. Each embedding is stored as:
{ 'id': .., 'embedding': [ ... ] }
The format of the embeddings for the index can be in either CSV, JSON, or Avro format.
Learn more about Embedding Formats for Indexing
End of explanation
EMBEDDINGS_INITIAL_URI = f"{BUCKET_URI}/matching_engine/initial/"
! gsutil cp glove100.json {EMBEDDINGS_INITIAL_URI}
Explanation: Store the JSONL formatted embeddings in Cloud Storage
Next, you upload the training data to your Cloud Storage bucket.
End of explanation
DIMENSIONS = 100
DISPLAY_NAME = "glove_100_1"
tree_ah_index = aiplatform.MatchingEngineIndex.create_tree_ah_index(
    display_name=DISPLAY_NAME,
    contents_delta_uri=EMBEDDINGS_INITIAL_URI,
    dimensions=DIMENSIONS,
    approximate_neighbors_count=150,
    distance_measure_type="DOT_PRODUCT_DISTANCE",
    description="Glove 100 ANN index",
    labels={"label_name": "label_value"},
    # TreeAH specific parameters
    leaf_node_embedding_count=500,
    leaf_nodes_to_search_percent=7,
)
INDEX_RESOURCE_NAME = tree_ah_index.resource_name
print(INDEX_RESOURCE_NAME)
Explanation: Create Matching Engine Index
Next, you create the index for your embeddings. Currently, two indexing algorithms are supported:
create_tree_ah_index():  Shallow tree + Asymmetric hashing.
create_brute_force_index(): Linear search.
In this tutorial, you use the create_tree_ah_index()for production scale. The method is called with the following parameters:
display_name: A human readable name for the index.
contents_delta_uri: A Cloud Storage location for the embeddings, which are either to be inserted, updated or deleted.
dimensions: The number of dimensions of the input vector
approximate_neighbors_count: (for Tree AH) The default number of neighbors to find via approximate search before exact reordering is performed. Exact reordering is a procedure where results returned by an approximate search algorithm are reordered via a more expensive distance computation.
distance_measure_type: The distance measure used in nearest neighbor search.
SQUARED_L2_DISTANCE: Euclidean (L2) Distance
L1_DISTANCE: Manhattan (L1) Distance
COSINE_DISTANCE: Cosine Distance. Defined as 1 - cosine similarity.
DOT_PRODUCT_DISTANCE: Default value. Defined as a negative of the dot product.
description: A human readble description of the index.
labels: User metadata in the form of a dictionary.
leaf_node_embedding_count: Number of embeddings on each leaf node. The default value is 1000 if not set.
leaf_nodes_to_search_percent: The default percentage of leaf nodes that any query may be searched. Must be in range 1-100, inclusive. The default value is 10 (means 10%) if not set.
This may take upto 30 minutes.
Learn more about Configuring Matching Engine Indexes.
End of explanation
with open("glove100_incremental.json", "w") as f:
    f.write(
        '{"id":"0","embedding":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]}\n'
    )
EMBEDDINGS_UPDATE_URI = f"{BUCKET_URI}/matching-engine/incremental/"
! gsutil cp glove100_incremental.json {EMBEDDINGS_UPDATE_URI}
Explanation: Update the Index
Next, you update the index with a new embedding -- i.e., insertion.
Create update delta file
First, you make a JSONL file with the embeddings to update. You use synthetic data -- in this case, all zeros -- for existing embedding with id of 0. You then upload the JSONL file to a Cloud Storage location.
End of explanation
tree_ah_index = tree_ah_index.update_embeddings(
    contents_delta_uri=EMBEDDINGS_UPDATE_URI,
)
INDEX_RESOURCE_NAME = tree_ah_index.resource_name
print(INDEX_RESOURCE_NAME)
Explanation: Update the index
Next, you use the method update_embeddings() to incrementally update the index, with the following parameters:
contents_delta_uri: A Cloud Storage location for the embeddings, which are either to be inserted or updated.
Optionally, the parameter is_complete_overwrite will replace the entire index.
End of explanation
# This is for display only; you can name the range anything.
PEERING_RANGE_NAME = "vertex-ai-prediction-peering-range"
NETWORK = "default"
# NOTE: `prefix-length=16` means a CIDR block with mask /16 will be
# reserved for use by Google services, such as Vertex AI.
! gcloud compute addresses create $PEERING_RANGE_NAME \
  --global \
  --prefix-length=16 \
  --description="peering range for Google service" \
  --network=$NETWORK \
  --purpose=VPC_PEERING
Explanation: Setup VPC peering network
To use a Matching Engine Index, you setup a VPC peering network between your project and the Vertex AI Matching Engine service project. This eliminates additional hops in network traffic and allows using efficient gRPC protocol.
Learn more about VPC peering.
IMPORTANT: you can only setup one VPC peering to servicenetworking.googleapis.com per project.
Create VPC peering for default network
For simplicity, we setup VPC peering to the default network. You can create a different network for your project.
If you setup VPC peering with any other network, make sure that the network already exists and that your VM is running on that network.
End of explanation
! gcloud services vpc-peerings connect \
  --service=servicenetworking.googleapis.com \
  --network=$NETWORK \
  --ranges=$PEERING_RANGE_NAME \
  --project=$PROJECT_ID
Explanation: Create the VPC connection
Next, create the connection for VPC peering.
Note: If you get a PERMISSION DENIED, you may not have the neccessary role 'Compute Network Admin' set for your default service account. In the Cloud Console, do the following steps.
Goto IAM &amp; Admin
Find your service account.
Click edit icon.
Select Add Another Role.
Enter 'Compute Network Admin'.
Select Save
End of explanation
! gcloud compute networks peerings list --network $NETWORK
Explanation: Check the status of your peering connections.
End of explanation
full_network_name = f"projects/{PROJECT_NUMBER}/global/networks/{NETWORK}"
Explanation: Construct the full network name
You need to have the full network resource name when you subsequently create an Matching Engine Index Endpoint resource for VPC peering.
End of explanation
index_endpoint = aiplatform.MatchingEngineIndexEndpoint.create(
    display_name="index_endpoint_for_demo",
    description="index endpoint description",
    network=full_network_name,
)
INDEX_ENDPOINT_NAME = index_endpoint.resource_name
print(INDEX_ENDPOINT_NAME)
Explanation: Create an IndexEndpoint with VPC Network
Next, you create a Matching Engine Index Endpoint, similar to the concept of creating a Private Endpoint for prediction with a peer-to-peer network.
To create the Index Endpoint resource, you call the method create() with the following parameters:
display_name: A human readable name for the Index Endpoint.
description: A description for the Index Endpoint.
network: The VPC network resource name.
End of explanation
DEPLOYED_INDEX_ID = "tree_ah_glove_deployed_" + TIMESTAMP
MIN_NODES = 1
MAX_NODES = 2
DEPLOY_COMPUTE = "n1-standard-16"
index_endpoint.deploy_index(
    display_name="deployed_index_for_demo",
    index=tree_ah_index,
    deployed_index_id=DEPLOYED_INDEX_ID,
    # machine_type=DEPLOY_COMPUTE,
    min_replica_count=MIN_NODES,
    max_replica_count=MAX_NODES,
)
print(index_endpoint.deployed_indexes)
Explanation: Deploy the Matching Engine Index to the Index Endpoint resource
Next, deploy your index to the Index Endpoint using the method deploy_index() with the following parameters:
display_name: A human readable name for the deployed index.
index: Your index.
deployed_index_id: A user assigned identifier for the deployed index.
machine_type: (optional) The VM instance type.
min_replica_count: (optional) Minimum number of VM instances for auto-scaling.
max_replica_count: (optional) Maximum number of VM instances for auto-scaling.
Learn more about Machine resources for Index Endpoint
End of explanation
# The number of nearest neighbors to be retrieved from database for each query.
NUM_NEIGHBOURS = 10
# Test query
queries = [
    [
        -0.11333,
        0.48402,
        0.090771,
        -0.22439,
        0.034206,
        -0.55831,
        0.041849,
        -0.53573,
        0.18809,
        -0.58722,
        0.015313,
        -0.014555,
        0.80842,
        -0.038519,
        0.75348,
        0.70502,
        -0.17863,
        0.3222,
        0.67575,
        0.67198,
        0.26044,
        0.4187,
        -0.34122,
        0.2286,
        -0.53529,
        1.2582,
        -0.091543,
        0.19716,
        -0.037454,
        -0.3336,
        0.31399,
        0.36488,
        0.71263,
        0.1307,
        -0.24654,
        -0.52445,
        -0.036091,
        0.55068,
        0.10017,
        0.48095,
        0.71104,
        -0.053462,
        0.22325,
        0.30917,
        -0.39926,
        0.036634,
        -0.35431,
        -0.42795,
        0.46444,
        0.25586,
        0.68257,
        -0.20821,
        0.38433,
        0.055773,
        -0.2539,
        -0.20804,
        0.52522,
        -0.11399,
        -0.3253,
        -0.44104,
        0.17528,
        0.62255,
        0.50237,
        -0.7607,
        -0.071786,
        0.0080131,
        -0.13286,
        0.50097,
        0.18824,
        -0.54722,
        -0.42664,
        0.4292,
        0.14877,
        -0.0072514,
        -0.16484,
        -0.059798,
        0.9895,
        -0.61738,
        0.054169,
        0.48424,
        -0.35084,
        -0.27053,
        0.37829,
        0.11503,
        -0.39613,
        0.24266,
        0.39147,
        -0.075256,
        0.65093,
        -0.20822,
        -0.17456,
        0.53571,
        -0.16537,
        0.13582,
        -0.56016,
        0.016964,
        0.1277,
        0.94071,
        -0.22608,
        -0.021106,
    ],
    [
        -0.99544,
        -2.3651,
        -0.24332,
        -1.0321,
        0.42052,
        -1.1817,
        -0.16451,
        -1.683,
        0.49673,
        -0.27258,
        -0.025397,
        0.34188,
        1.5523,
        1.3532,
        0.33297,
        -0.0056677,
        -0.76525,
        0.49587,
        1.2211,
        0.83394,
        -0.20031,
        -0.59657,
        0.38485,
        -0.23487,
        -1.0725,
        0.95856,
        0.16161,
        -1.2496,
        1.6751,
        0.73899,
        0.051347,
        -0.42702,
        0.16257,
        -0.16772,
        0.40146,
        0.29837,
        0.96204,
        -0.36232,
        -0.47848,
        0.78278,
        0.14834,
        1.3407,
        0.47834,
        -0.39083,
        -1.037,
        -0.24643,
        -0.75841,
        0.7669,
        -0.37363,
        0.52741,
        0.018563,
        -0.51301,
        0.97674,
        0.55232,
        1.1584,
        0.73715,
        1.3055,
        -0.44743,
        -0.15961,
        0.85006,
        -0.34092,
        -0.67667,
        0.2317,
        1.5582,
        1.2308,
        -0.62213,
        -0.032801,
        0.1206,
        -0.25899,
        -0.02756,
        -0.52814,
        -0.93523,
        0.58434,
        -0.24799,
        0.37692,
        0.86527,
        0.069626,
        1.3096,
        0.29975,
        -1.3651,
        -0.32048,
        -0.13741,
        0.33329,
        -1.9113,
        -0.60222,
        -0.23921,
        0.12664,
        -0.47961,
        -0.89531,
        0.62054,
        0.40869,
        -0.08503,
        0.6413,
        -0.84044,
        -0.74325,
        -0.19426,
        0.098722,
        0.32648,
        -0.67621,
        -0.62692,
    ],
]
matches = index_endpoint.match(
    deployed_index_id=DEPLOYED_INDEX_ID, queries=queries, num_neighbors=NUM_NEIGHBOURS
)
for instance in matches:
    print("INSTANCE")
    for match in instance:
        print(match)
Explanation: Create and execute an online query
Now that your index is deployed, you can make queries.
First, you construct a vector query using synthetic data, to use as the example to return matches for.
Next, you make the matching request using the method match(), with the following parameters:
deployed_index_id:  The identifier of the deployed index.
queries: A list of queries (instances).
num_neighbors: The number of closest matches to return.
End of explanation
brute_force_index = aiplatform.MatchingEngineIndex.create_brute_force_index(
    display_name=DISPLAY_NAME,
    contents_delta_uri=EMBEDDINGS_INITIAL_URI,
    dimensions=DIMENSIONS,
    distance_measure_type="DOT_PRODUCT_DISTANCE",
    description="Glove 100 index (brute force)",
    labels={"label_name": "label_value"},
)
INDEX_BRUTE_FORCE_RESOURCE_NAME = brute_force_index.resource_name
print(INDEX_BRUTE_FORCE_RESOURCE_NAME)
Explanation: Create brute force index for calibration
The brute force index uses a naive brute force method to find the nearest neighbors. This method uses a linear search and thus not efficient for large scale indexes. We recommend using the brute force index for calibrating the approximate nearest neighbor (ANN) index for recall, or for mission critical matches.
Create the brute force index
Now create the brute force index using the method create_brute_force_index().
To ensure an apples to apples comparison, the distanceMeasureType and featureNormType, dimensions of the brute force index should match those of the production indices being tuned.
End of explanation
brute_force_index = tree_ah_index.update_embeddings(
    contents_delta_uri=EMBEDDINGS_UPDATE_URI
)
Explanation: Update the index
For apples to apples comparison, you perform the same incremental update to the brute force index as you did for the Tree AH index.
End of explanation
DEPLOYED_BRUTE_FORCE_INDEX_ID = "glove_brute_force_deployed_" + TIMESTAMP
index_endpoint.deploy_index(
    index=brute_force_index, deployed_index_id=DEPLOYED_BRUTE_FORCE_INDEX_ID
)
Explanation: Deploy the brute force index to the IndexEndpoint resource
Next, you deploy the brute force index to the same IndexEndpoint.
Note: You can deploy multiple indexes to the same Index Endpoint resource.
End of explanation
prod_matches = index_endpoint.match(
    deployed_index_id=DEPLOYED_INDEX_ID,
    queries=list(test),
    num_neighbors=NUM_NEIGHBOURS,
)
exact_matches = index_endpoint.match(
    deployed_index_id=DEPLOYED_BRUTE_FORCE_INDEX_ID,
    queries=list(test),
    num_neighbors=NUM_NEIGHBOURS,
)
Explanation: Calibration
Now your ready to do calibration. The production version of the index uses an approxiamation method, which means it may have less than perfect recall when compared to the slower exact match (brute force) method.
Get test results for both indexes
First, using the GloVe test embeddings, you make the identical request to both indexes.
End of explanation
# Calculate recall by determining how many neighbors were correctly retrieved as compared to the brute-force option.
correct_neighbors = 0
for tree_ah_neighbors, brute_force_neighbors in zip(prod_matches, exact_matches):
    tree_ah_neighbor_ids = [neighbor.id for neighbor in tree_ah_neighbors]
    brute_force_neighbor_ids = [neighbor.id for neighbor in brute_force_neighbors]
    correct_neighbors += len(
        set(tree_ah_neighbor_ids).intersection(brute_force_neighbor_ids)
    )
recall = correct_neighbors / (len(test) * NUM_NEIGHBOURS)
print("Recall: {}".format(recall))
Explanation: Compute Recall
Finally, you determine from the results the percentage of exact matches are recalled from the production index. You can subsequently use this information to tune the deployment of the production index.
End of explanation
# Force undeployment of indexes and delete endpoint
try:
    index_endpoint.delete(force=True)
except Exception as e:
    print(e)
# Delete indexes
try:
    tree_ah_index.delete()
    brute_force_index.delete()
except Exception as e:
    print(e)
delete_bucket = False
if delete_bucket or os.getenv("IS_TESTING"):
    ! gsutil rm -rf {BUCKET_URI}
Explanation: Cleaning up
To clean up all Google Cloud resources used in this project, you can delete the Google Cloud
project you used for the tutorial.
You can also manually delete resources that you created by running the following code.
End of explanation
<END_TASK>