<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Indic NLP Library
The goal of the Indic NLP Library is to build Python based libraries for common text processing and Natural Language Processing in Indian languages. Indian languages share a lot of similarity in terms of script, phonology, language syntax, etc. and this library is an attempt to provide a general solution to very commonly required toolsets for Indian language text. 
The library provides the following functionalities
Step1:  Add Library to Python path
Step2:  Export environment variable  
export INDIC_RESOURCES_PATH=&lt;path&gt;
 OR
set it programmatically
We will use that method for this demo
Step3:  Initialize the Indic NLP library
Step4:  Let's actually try out some of the API methods in the Indic NLP library 
Many of the API functions require a language code. We use 2-letter ISO 639-1 codes. Some languages do not have assigned 2-letter codes. We use the following two-letter codes for such languages
Step5:  Script Conversion
Convert from one Indic script to another. This is a simple script which exploits the fact that Unicode points of various Indic scripts are at corresponding offsets from the base codepoint for that script. The following scripts are supported
Step6:  Romanization
Convert script text to Roman text in the ITRANS notation
Step7:  Indicization (ITRANS to Indic Script)
Let's call conversion of ITRANS-transliteration to an Indic script as Indicization!
Step8:  Script Information
Indic scripts have been designed keeping phonetic principles in nature and the design and organization of the scripts makes it easy to obtain phonetic information about the characters. 
Get Phonetic Feature Vector
With each script character, a phontic feature vector is associated, which encodes the phontic properties of the character. This is a bit vector which is can be obtained as shown below
Step9:  This fields in this bit vector are (from left to right)
Step10:  You can check the phonetic information database files in Indic NLP resources to know the definition of each of the bits. 
For Tamil Script
Step11:  Get Phonetic Similarity
Using the phonetic feature vectors, we can define phonetic similarity between the characters (and underlying phonemes). The library implements some  measures for phonetic similarity between the characters (and underlying phonemes). These can be defined using the phonetic feature vectors discussed earlier, so users can implement additional similarity measures. 
The implemented similarity measures are
Step12:  You may have figured out that you can also compute similarities of characters belonging to different scripts.
You can also get a similarity matrix which contains the similarities between all pairs of characters (within the same script or across scripts).
Let's see how we can compare the characters across Devanagari and Malayalam scripts
Step13:  Some similarity functions like sim do not generate values in the range [0,1] and it may be more convenient to have the similarity values in the range [0,1]. This can be achieved by setting the normalize paramter to True
Step14:  Orthographic Syllabification
Orthographic Syllabification is an approximate syllabification process for Indic scripts, where CV+ units are defined to be orthographic syllables.
See the following paper for details
Step15:  Tokenization
A trivial tokenizer which just tokenizes on the punctuation boundaries. This also includes punctuations for the Indian language scripts (the purna virama and the deergha virama). It returns a list of tokens.
Step16:  Word Segmentation
Unsupervised morphological analysers for various Indian language. Given a word, the analyzer returns the componenent morphemes. 
The analyzer can recognize inflectional and derivational morphemes. 
The following languages are supported
Step17:  Transliteration
We use the BrahmiNet REST API for transliteration.
<ASSISTANT_TASK:>
Python Code:
# The path to the local git repo for Indic NLP library
INDIC_NLP_LIB_HOME="/home/development/anoop/installs/indic_nlp_library"
# The path to the local git repo for Indic NLP Resources
INDIC_NLP_RESOURCES="/usr/local/bin/indicnlp/indic_nlp_resources"
Explanation: Indic NLP Library
The goal of the Indic NLP Library is to build Python based libraries for common text processing and Natural Language Processing in Indian languages. Indian languages share a lot of similarity in terms of script, phonology, language syntax, etc. and this library is an attempt to provide a general solution to very commonly required toolsets for Indian language text. 
The library provides the following functionalities: 
Text Normalization
Script Conversion
Romanization
Indicization
Script Information
Phonetic Similarity
Syllabification
Tokenization
Word Segmenation
Transliteration
Translation
The data resources required by the Indic NLP Library are hosted in a different repository. These resources are required for some modules. You can download from the Indic NLP Resources project.
Pre-requisites
Python 2.7+
Morfessor 2.0 Python Library
Getting Started
  ----- Set these variables -----
End of explanation
import sys
sys.path.append('{}/src'.format(INDIC_NLP_LIB_HOME))
Explanation: Add Library to Python path
End of explanation
from indicnlp import common
common.set_resources_path(INDIC_NLP_RESOURCES)
Explanation: Export environment variable  
export INDIC_RESOURCES_PATH=&lt;path&gt;
 OR
set it programmatically
We will use that method for this demo
End of explanation
from indicnlp import loader
loader.load()
Explanation: Initialize the Indic NLP library
End of explanation
from indicnlp.normalize.indic_normalize import IndicNormalizerFactory
input_text=u"\u0958 \u0915\u093c"
remove_nuktas=False
factory=IndicNormalizerFactory()
normalizer=factory.get_normalizer("hi",remove_nuktas)
output_text=normalizer.normalize(input_text)
print output_text
print 'Length before normalization: {}'.format(len(input_text))
print 'Length after normalization: {}'.format(len(output_text))
Explanation: Let's actually try out some of the API methods in the Indic NLP library 
Many of the API functions require a language code. We use 2-letter ISO 639-1 codes. Some languages do not have assigned 2-letter codes. We use the following two-letter codes for such languages: 
Konkani: kK
Manipuri: mP
Bodo: bD
Text Normalization
Text written in Indic scripts display a lot of quirky behaviour on account of varying input methods, multiple representations for the same character, etc. 
There is a need to canonicalize the representation of text so that NLP applications can handle the data in a consistent manner. The canonicalization primarily handles the following issues: 
- Non-spacing characters like ZWJ/ZWNL
- Multiple representations of Nukta based characters 
- Multiple representations of two part dependent vowel signs
- Typing inconsistencies: e.g. use of pipe (|) for poorna virama
End of explanation
from indicnlp.transliterate.unicode_transliterate import UnicodeIndicTransliterator
input_text=u'राजस्थान'
print UnicodeIndicTransliterator.transliterate(input_text,"hi","pa")
Explanation: Script Conversion
Convert from one Indic script to another. This is a simple script which exploits the fact that Unicode points of various Indic scripts are at corresponding offsets from the base codepoint for that script. The following scripts are supported:
Devanagari (Hindi,Marathi,Sanskrit,Konkani,Sindhi,Nepali), Assamese, Bengali, Oriya, Gujarati, Gurumukhi (Punjabi), Sindhi, Tamil, Telugu, Kannada, Malayalam
End of explanation
from indicnlp.transliterate.unicode_transliterate import ItransTransliterator
input_text=u'राजस्थान'
lang='hi'
print ItransTransliterator.to_itrans(input_text,lang)
Explanation: Romanization
Convert script text to Roman text in the ITRANS notation
End of explanation
from indicnlp.transliterate.unicode_transliterate import ItransTransliterator
# input_text=u'rajasthAna'
input_text=u'pitL^In'
lang='hi'
x=ItransTransliterator.from_itrans(input_text,lang)
print x
for y in x:
    print '{:x}'.format(ord(y))
Explanation: Indicization (ITRANS to Indic Script)
Let's call conversion of ITRANS-transliteration to an Indic script as Indicization!
End of explanation
from indicnlp.script import  indic_scripts as isc
c=u'क'
lang='hi'
isc.get_phonetic_feature_vector(c,lang)
Explanation: Script Information
Indic scripts have been designed keeping phonetic principles in nature and the design and organization of the scripts makes it easy to obtain phonetic information about the characters. 
Get Phonetic Feature Vector
With each script character, a phontic feature vector is associated, which encodes the phontic properties of the character. This is a bit vector which is can be obtained as shown below:
End of explanation
sorted(isc.PV_PROP_RANGES.iteritems(),key=lambda x:x[1][0])
Explanation: This fields in this bit vector are (from left to right):
End of explanation
from indicnlp.langinfo import *
c=u'क'
lang='hi'
print 'Is vowel?:  {}'.format(is_vowel(c,lang))
print 'Is consonant?:  {}'.format(is_consonant(c,lang))
print 'Is velar?:  {}'.format(is_velar(c,lang))
print 'Is palatal?:  {}'.format(is_palatal(c,lang))
print 'Is aspirated?:  {}'.format(is_aspirated(c,lang))
print 'Is unvoiced?:  {}'.format(is_unvoiced(c,lang))
print 'Is nasal?:  {}'.format(is_nasal(c,lang))
Explanation: You can check the phonetic information database files in Indic NLP resources to know the definition of each of the bits. 
For Tamil Script: database 
For other Indic Scripts: database
Query Phonetic Properties
Note: This interface below will be deprecated soon and a new interface will be available soon.
End of explanation
from indicnlp.script import  indic_scripts as isc
from indicnlp.script import  phonetic_sim as psim
c1=u'क'
c2=u'ख'
c3=u'भ'
lang='hi'
print u'Similarity between {} and {}'.format(c1,c2)
print psim.cosine(
    isc.get_phonetic_feature_vector(c1,lang),
    isc.get_phonetic_feature_vector(c2,lang)
    )
print
print u'Similarity between {} and {}'.format(c1,c3)
print psim.cosine(
    isc.get_phonetic_feature_vector(c1,lang),
    isc.get_phonetic_feature_vector(c3,lang)
    )
Explanation: Get Phonetic Similarity
Using the phonetic feature vectors, we can define phonetic similarity between the characters (and underlying phonemes). The library implements some  measures for phonetic similarity between the characters (and underlying phonemes). These can be defined using the phonetic feature vectors discussed earlier, so users can implement additional similarity measures. 
The implemented similarity measures are: 
cosine
dice
jaccard
dot_product
sim1 (Kunchukuttan et al., 2016)
softmax
 References 
Anoop Kunchukuttan, Pushpak Bhattacharyya, Mitesh Khapra. Substring-based unsupervised transliteration with phonetic and contextual knowledge. SIGNLL Conference on Computational Natural Language Learning  (CoNLL 2016) . 2016.
End of explanation
from indicnlp.script import  indic_scripts as isc
from indicnlp.script import  phonetic_sim as psim
slang='hi'
tlang='ml'
sim_mat=psim.create_similarity_matrix(psim.cosine,slang,tlang,normalize=False)
c1=u'क'
c2=u'ഖ'
print u'Similarity between {} and {}'.format(c1,c2)
print sim_mat[isc.get_offset(c1,slang),isc.get_offset(c2,tlang)]
Explanation: You may have figured out that you can also compute similarities of characters belonging to different scripts.
You can also get a similarity matrix which contains the similarities between all pairs of characters (within the same script or across scripts).
Let's see how we can compare the characters across Devanagari and Malayalam scripts
End of explanation
slang='hi'
tlang='ml'
sim_mat=psim.create_similarity_matrix(psim.sim1,slang,tlang,normalize=True)
c1=u'क'
c2=u'ഖ'
print u'Similarity between {} and {}'.format(c1,c2)
print sim_mat[isc.get_offset(c1,slang),isc.get_offset(c2,tlang)]
Explanation: Some similarity functions like sim do not generate values in the range [0,1] and it may be more convenient to have the similarity values in the range [0,1]. This can be achieved by setting the normalize paramter to True
End of explanation
from indicnlp.syllable import  syllabifier
w=u'जगदीशचंद्र'
lang='hi'
print u' '.join(syllabifier.orthographic_syllabify(w,lang))
Explanation: Orthographic Syllabification
Orthographic Syllabification is an approximate syllabification process for Indic scripts, where CV+ units are defined to be orthographic syllables.
See the following paper for details:
Anoop Kunchukuttan, Pushpak Bhattacharyya. Orthographic Syllable as basic unit for SMT between Related Languages. Conference on Empirical Methods in Natural Language Processing (EMNLP 2016). 2016.
End of explanation
from indicnlp.tokenize import indic_tokenize  
indic_string=u'अनूप,अनूप?।फोन'
print u'Input String: {}'.format(indic_string)
print u'Tokens: '
for t in indic_tokenize.trivial_tokenize(indic_string): 
    print t
Explanation: Tokenization
A trivial tokenizer which just tokenizes on the punctuation boundaries. This also includes punctuations for the Indian language scripts (the purna virama and the deergha virama). It returns a list of tokens.
End of explanation
from indicnlp.morph import unsupervised_morph 
from indicnlp import common
analyzer=unsupervised_morph.UnsupervisedMorphAnalyzer('mr')
indic_string=u'आपल्या हिरड्यांच्या आणि दातांच्यामध्ये जीवाणू असतात .'
analyzes_tokens=analyzer.morph_analyze_document(indic_string.split(' '))
for w in analyzes_tokens: 
    print w
Explanation: Word Segmentation
Unsupervised morphological analysers for various Indian language. Given a word, the analyzer returns the componenent morphemes. 
The analyzer can recognize inflectional and derivational morphemes. 
The following languages are supported:
Hindi, Punjabi, Marathi, Konkani, Gujarati, Bengali, Kannada, Tamil, Telugu, Malayalam
Support for more languages will be added soon.
End of explanation
import urllib2
from django.utils.encoding import * 
from django.utils.http import * 
text=iri_to_uri(urlquote('anoop, ratish kal fone par baat karenge'))
url=u'http://www.cfilt.iitb.ac.in/indicnlpweb/indicnlpws/transliterate_bulk/en/hi/{}/statistical'.format(text)
response=urllib2.urlopen(url).read()
print response
Explanation: Transliteration
We use the BrahmiNet REST API for transliteration.
End of explanation
<END_TASK>