<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
Step1:  Form Parsing using Google Cloud Document AI
This notebook shows how to use Google Cloud Document AI to parse a campaign disclosure form.
It accompanies this Medium article
Step2:  Document
As an example, let's take this US election campaign disclosure form.
Step3:  Note
Step4:  Enable Document AI
First enable Document AI in your project by visiting
https
Step5:  Create a service account authorization by visiting
https
Step6:  Note
Step7:  Option 1
Step8:  We know that "Cash on Hand" is on Page 2.
Step9:  Cool, we are at the right part of the document! Let's get the next block, which should be the actual amount.
Step10:  Option 2
<ASSISTANT_TASK:>
Python Code:
!sudo chown -R jupyter:jupyter /home/jupyter/imported/formparsing.ipynb
from IPython.display import Markdown as md
### change to reflect your notebook
_nb_repo = 'training-data-analyst'
_nb_loc = "blogs/form_parser/formparsing.ipynb"
_nb_title = "Form Parsing Using Google Cloud Document AI"
### no need to change any of this
_nb_safeloc = _nb_loc.replace('/', '%2F')
_nb_safetitle = _nb_title.replace(' ', '+')
md(
<table class="tfo-notebook-buttons" align="left">
  <td>
    <a target="_blank" href="https://console.cloud.google.com/ai-platform/notebooks/deploy-notebook?name={1}&url=https%3A%2F%2Fgithub.com%2FGoogleCloudPlatform%2F{3}%2Fblob%2Fmaster%2F{2}&download_url=https%3A%2F%2Fgithub.com%2FGoogleCloudPlatform%2F{3}%2Fraw%2Fmaster%2F{2}">
    <img src="https://raw.githubusercontent.com/GoogleCloudPlatform/practical-ml-vision-book/master/logo-cloud.png"/> Run in AI Platform Notebook</a>
  </td>
  </td>
  <td>
    <a target="_blank" href="https://colab.research.google.com/github/GoogleCloudPlatform/{3}/blob/master/{0}">
    <img src="https://www.tensorflow.org/images/colab_logo_32px.png" />Run in Google Colab</a>
  </td>
  <td>
    <a target="_blank" href="https://github.com/GoogleCloudPlatform/{3}/blob/master/{0}">
    <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" />View source on GitHub</a>
  </td>
  <td>
    <a href="https://raw.githubusercontent.com/GoogleCloudPlatform/{3}/master/{0}">
    <img src="https://www.tensorflow.org/images/download_logo_32px.png" />Download notebook</a>
  </td>
</table>
.format(_nb_loc, _nb_safetitle, _nb_safeloc, _nb_repo))
Explanation: Form Parsing using Google Cloud Document AI
This notebook shows how to use Google Cloud Document AI to parse a campaign disclosure form.
It accompanies this Medium article:
https://medium.com/@lakshmanok/how-to-parse-forms-using-google-cloud-document-ai-68ad47e1c0ed
End of explanation
%%bash
if [ ! -f scott_walker.pdf ]; then
   curl -O https://storage.googleapis.com/practical-ml-vision-book/images/scott_walker.pdf
fi
!ls *.pdf
from IPython.display import IFrame
IFrame("./scott_walker.pdf", width=600, height=300)
Explanation: Document
As an example, let's take this US election campaign disclosure form.
End of explanation
BUCKET="ai-analytics-solutions-kfpdemo"  # CHANGE to a bucket that you own
!gsutil cp scott_walker.pdf gs://{BUCKET}/formparsing/scott_walker.pdf
!gsutil ls gs://{BUCKET}/formparsing/scott_walker.pdf
Explanation: Note: If the file is not visible, simply open the PDF file by double-clicking on it in the left hand menu.
Upload to Cloud Storage
Document AI works with documents on Cloud Storage, so let's upload the doc.
End of explanation
!gcloud auth list
Explanation: Enable Document AI
First enable Document AI in your project by visiting
https://console.developers.google.com/apis/api/documentai.googleapis.com/overview
Find out who you are running as:
End of explanation
%%bash
PDF="gs://ai-analytics-solutions-kfpdemo/formparsing/scott_walker.pdf" # CHANGE to your PDF file
REGION="us"  # change to EU if the bucket is in the EU
cat <<EOM > request.json
{
   "inputConfig":{
      "gcsSource":{
         "uri":"${PDF}"
      },
      "mimeType":"application/pdf"
   },
   "documentType":"general",
   "formExtractionParams":{
      "enabled":true
   }
}
EOM
# Send request to Document AI.
PROJECT=$(gcloud config get-value project)
echo "Sending the following request to Document AI in ${PROJECT} ($REGION region), saving to response.json"
cat request.json
curl -X POST \
  -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
  -H "Content-Type: application/json; charset=utf-8" \
  -d @request.json \
  https://${REGION}-documentai.googleapis.com/v1beta2/projects/${PROJECT}/locations/us/documents:process \
  > response.json
!tail response.json
Explanation: Create a service account authorization by visiting
https://console.cloud.google.com/iam-admin/serviceaccounts/create
Give this service account Document AI Core Service Account authorization
Give the above ACTIVE ACCOUNT the ability to use the service account you just created.
Call Document AI
End of explanation
import json
ifp = open('response.json')
response = json.load(ifp)
allText = response['text']
print(allText[:100])
Explanation: Note: If you get a 403 PERMISSION DENIED error, please re-run all the cells from the top.
Parse the response
Let's use Python to parse the response and pull out specific fields.
End of explanation
print(allText.index("CASH ON HAND"))
Explanation: Option 1: Parsing blocks of text
As an example, let's try to get the "Cash on Hand". This is in Page 2 and the answer is $75,931.36
All the data in the document is the allText field. we just need to find the right starting and ending index
for what we want to extract.
End of explanation
response['pages'][1]['blocks'][5]
response['pages'][1]['blocks'][5]['layout']['textAnchor']['textSegments'][0]
startIndex = int(response['pages'][1]['blocks'][5]['layout']['textAnchor']['textSegments'][0]['startIndex'])
endIndex = int(response['pages'][1]['blocks'][5]['layout']['textAnchor']['textSegments'][0]['endIndex'])
allText[startIndex:endIndex]
Explanation: We know that "Cash on Hand" is on Page 2.
End of explanation
def extractText(allText, elem):
    startIndex = int(elem['textAnchor']['textSegments'][0]['startIndex'])
    endIndex = int(elem['textAnchor']['textSegments'][0]['endIndex'])
    return allText[startIndex:endIndex].strip()
amount = float(extractText(allText, response['pages'][1]['blocks'][6]['layout']))
print(amount)
Explanation: Cool, we are at the right part of the document! Let's get the next block, which should be the actual amount.
End of explanation
response['pages'][1].keys()
response['pages'][1]['formFields'][2]
fieldName = extractText(allText, response['pages'][1]['formFields'][2]['fieldName'])
fieldValue = extractText(allText, response['pages'][1]['formFields'][2]['fieldValue'])
print('key={}\nvalue={}'.format(fieldName, fieldValue))
Explanation: Option 2: Parsing form fields
What we did with blocks of text was quite low-level. Document AI understands that forms tend to have key-value pairs, and part of the JSON response includes these extracted key-value pairs as well.
Besides FormField Document AI also supports getting Paragraph and Table from the document.
End of explanation
<END_TASK>