<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Import Blockchain
Step1:  Change the headers
Step2:  NODES
Convert txt files to csv
http
Step3:  -----------------------------
EDGES
Step4:  Import script - LOAD CSV?? - STOPPED HERE
graphdb lives here
Step5:  Original Blockchain API
Step6:  OLD CODE
Step7:  benford's law test for ticker data
applies the law to a couple of days of BTC-E non-zero price returns.
https
Step8:  Python Drivers
https
<ASSISTANT_TASK:>
Python Code:
import errno
import os
import shutil
import zipfile
import numpy as np
import pandas as pd
# In[22]:
# TARGETDIR = '../btc/graphs_njp.zip'
# In[23]:
# with open(doc, "rb") as zipsrc:
#     zfile = zipfile.ZipFile(zipsrc)
#     for member in zfile.infolist():
#        target_path = os.path.join(TARGETDIR, member.filename)
#        if target_path.endswith('/'):  # folder entry, create
#            try:
#                os.makedirs(target_path)
#            except (OSError, IOError) as err:
#                # Windows may complain if the folders already exist
#                if err.errno != errno.EEXIST:
#                    raise
#            continue
#        with open(target_path, 'wb') as outfile, zfile.open(member) as infile:
#            shutil.copyfileobj(infile, outfile)
Explanation: Import Blockchain
End of explanation
!ls
!unzip bitcoin_dataset.zip
Explanation: Change the headers
End of explanation
# addresses
addr = pd.read_csv('addresses.txt', header = None, names = ['user:ID', 'addr'], delimiter= '\t')
addr[':LABEL'] = 'address'
addr.to_csv('addresses.csv', index = False)
addr.head()
# blocks
blks = pd.read_csv('blockhash.txt', header = None, names = ['block:ID', 'bhash', 'btime', 'txs' ], delimiter= '\t')
blks[':LABEL'] = 'blockchain'
blks.to_csv('blockhash.csv', index = False)
blks.head()
!rm addresses.txt 
!rm blockhash.txt
# transactions + transaction time
txns = pd.read_csv('txhash.txt', header = None, names = ['tx:ID', 'txhash'], delimiter= '\t')
txns2 = pd.read_csv('tx.txt', header = None, names = ['tx:ID', 'block' ,'n_inputs', 'n_outputs'], delimiter= '\t')
# txns = txns.drop(['Unnamed: 0', ':LABEL'], axis = 1)
txnstime = pd.read_csv('txtime.txt', header = None, names = ['tx:ID', 'unixtime'], delimiter= '\t')
txnstime['time'] = pd.to_datetime(txnstime['unixtime'],unit='s')
txns = txns.merge(txnstime, how = 'left', on = 'tx:ID')
txns = txns.merge(txns2, how = 'left', on = 'tx:ID')
txns[':LABEL'] = 'transaction'
txns.head()
txns.to_csv('tx.csv', index = False)
print(txns.columns)
!rm txhash.txt
!rm txtime.txt
!rm tx.txt
# txnsin
txnsin = pd.read_csv('txin.txt', header = None, names = ['tx:ID', 'user' ,'value'], delimiter= '\t')
txnsin[':LABEL'] = 'incoming_payment'
txnsin.to_csv('txin.csv', index = False)
txnsin.head()
# txnsout
txnsout = pd.read_csv('txout.txt', header = None, names = ['tx:ID', 'user' ,'value'], delimiter= '\t')
txnsout[':LABEL'] = 'sent_coins'
txnsout.to_csv('txout.csv', index = False)
txnsout.head()
!rm txin.txt
!rm txout.txt
Explanation: NODES
Convert txt files to csv
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html
nodes first
https://neo4j.com/docs/opera
End of explanation
import numpy as np
import pandas as pd
!ls
rels_txns_to_block = pd.read_csv('tx.csv') #, compression='zip')
rels_txns_to_block.columns
rels_txns_to_block = rels_txns_to_block.drop(['txhash', 'unixtime', 'time', 'n_inputs', 'n_outputs',
       ':LABEL'], axis = 1)
rels_txns_to_block.head()
rels_txns_to_block[':TYPE'] = 'part_of_block'
rels_txns_to_block.to_csv('relsPaymentsToAddress.csv', index = False)
txnsout.head()
Explanation: -----------------------------
EDGES
End of explanation
# In[ ]:
file = open("poc_transaction.txt", "w")
fileIn = open("poc_transactionsIn.txt", "w")
fileOut = open("poc_transactionsOut.txt", "w")
fileTransList = open("poc_transactionList.txt","w")
fileAddressList = open ("poc_addressList.txt","w")
fileBlockList = open("poc_blockdata.txt", "w")
file.write(":ID,Transaction_ID,Hash,Time,V_In,V_Out,:LABEL" + "\n")
fileIn.write(":ID,Transaction_ID,Transaction_Hash,Address,Spent,Value,:LABEL" + "\n")
fileOut.write(":ID,Transaction_ID,Transaction_Hash,Type,:LABEL" + "\n")
fileAddressList.write(":ID,AddressID,:LABEL" + "\n")
fileBlockList.write(":ID,BlockID,Hash,Received_Time,Previous_Block_Hash,Transaction_Count,Height,:LABEL"  + "\n")
fileBlockTrans = open("poc_relsBlocksTransactions.txt","w")
fileTransOut = open("poc_relsTransactionsOut.txt","w")
fileTransInRels = open("poc_relsTransactionsIn.txt","w")
filePaymentOutAddr = open("poc_relsPaymentsToAddress.txt","w")
filePaymentInAddr = open("poc_relsRedeemedFromAddress.txt","w")
fileBlockTrans.write(":START_ID,:END_ID,:TYPE" + "\n")
fileTransOut.write(":START_ID,:END_ID,Spent,Value,:TYPE" + "\n")
fileTransInRels.write(":START_ID,:END_ID,Spent,Value,:TYPE" + "\n")
filePaymentOutAddr.write(":START_ID,:END_ID,Spent,Value,:TYPE" + "\n")
filePaymentInAddr.write(":START_ID,:END_ID,Spent,Value,:TYPE" + "\n")
fileBlockTrans.write(str(hash) + ',' + str('trans' + str(item["tx_index"])) + ',' + 'PART_OF_BLOCK' + "\n")
fileTransOut.write('trans' + str(xx["tx_index"]) + ',' + 'out_' + str(tout) + ',' + str(xx["spent"]) + ','+ str( Decimal(xx["value"]) / Decimal(100000000.0)) + ',' + 'SENT_COINS' + "\n")
fileTransInRels.write('transin_' + str(tin) + ',' + 'trans' + str(tx_index) + ',' + str(nn["prev_out"]["spent"]) + ','+ str( Decimal(nn["prev_out"]["value"]) / Decimal(100000000.0)) + ',' + 'INCOMING_PAYMENT' + "\n")
filePaymentOutAddr.write('out_' + str(tout) + ',' + rec + ',' + str(xx["spent"]) + ','+ str( Decimal(xx["value"]) / Decimal(100000000.0)) + ','+ 'WAS_SENT_TO' + "\n")
filePaymentInAddr.write(strAddr + ',' + 'transin_' + str(tin) + ',' + str(nn["prev_out"]["spent"]) + ','+ str( Decimal(nn["prev_out"]["value"])/ Decimal(100000000.0)) + ','+ 'REDEEMED' + "\n")
# tout = 1
# tin = 1
# f = open('blockList.txt', 'r')
# temp = f.read().splitlines()
# for line in temp:
# 	words = line.split("|")
# 	s = words[1]
	
# 	url = "https://blockchain.info/rawblock/" + str(s)
# 	print url
# 	usock = urllib2.urlopen(url)
# 	data = usock.read()
# 	result = json.loads(data)
# 	hash = result['hash']
# 	block_index = result['block_index']
# 	height = result['height']
# 	size = result['size']
# 	main_chain = result['main_chain']
# 	prev_block = result['prev_block']
# 	try: 
# 		received_time = result['received_time']
# 	except KeyError:
# 		received_time = 'NA'
# 	n_tx = result['n_tx']
# 	fileBlockList.write(str(hash) + ',' + str(block_index) + ',' +str(hash) + ',' + str(received_time) + ',' + str(prev_block) + ',' + str(n_tx) + ',' + str(height) + ",BlockChain" + "\n");
# 	parent =  result["tx"]
# 	for item in parent:
# 		tx_index = str(item["tx_index"])
# 		tx_hash = str(item["hash"])
# 		file.write(str('trans' + str(item["tx_index"])) + ',' + str(item["tx_index"]) + ',' +str(item["hash"]) + ',' + str(item["time"]) + ',' + str(item["vin_sz"]) + ',' + str(item["vout_sz"]) + ",Transaction"+ "\n");
# 		fileBlockTrans.write(str(hash) + ',' + str('trans' + str(item["tx_index"])) + ',' + 'PART_OF_BLOCK' + "\n")
# 		if 'inputs' in item :
# 			for nn in item["inputs"]:
# #				print nn["sequence"]
# 				if 'prev_out' in nn :
# #					print nn["prev_out"]["addr"]
# #					print nn["prev_out"]["spent"]
# #					print nn["prev_out"]["value"]
# 					try: 
# 						strAddr = str(nn["prev_out"]["addr"])
# 					except KeyError:
# 						strAddr = 'NA'
# 					fileIn.write('transin_' + str(tin) + ',' + tx_index + ',' + str(tx_hash) + ',' + strAddr + ',' +str(nn["prev_out"]["spent"]) + ',' + str( Decimal(nn["prev_out"]["value"]) / Decimal(100000000.0))  + ",IncomingPayment" + "\n");
# 					fileTransInRels.write('transin_' + str(tin) + ',' + 'trans' + str(tx_index) + ',' + str(nn["prev_out"]["spent"]) + ','+ str( Decimal(nn["prev_out"]["value"]) / Decimal(100000000.0)) + ',' + 'INCOMING_PAYMENT' + "\n")
# 					filePaymentInAddr.write(strAddr + ',' + 'transin_' + str(tin) + ',' + str(nn["prev_out"]["spent"]) + ','+ str( Decimal(nn["prev_out"]["value"])/ Decimal(100000000.0)) + ','+ 'REDEEMED' + "\n")
# 					fileTransList.write('transin_' + str(tin) + ',' + tx_hash + "\n")
# 					fileAddressList.write(strAddr + ',' + strAddr + ',Address' + "\n")
# 					tin = tin + 1
					
# 		if 'out' in item :
# 			for xx in item["out"]:
# #							print xx["tx_index"]
# #							print xx["type"]
# #							print xx["addr"]
# #							print xx["spent"]
# #							print xx["value"]
# 				try: 
# 					rec = str(xx["addr"])
# 					fileOut.write('out_' + str(tout) + ',' + str(xx["tx_index"]) + ',' + str(tx_hash) + ',' + str(xx["type"])  + ",OutgoingPayment"+ "\n");
# 					fileTransOut.write('trans' + str(xx["tx_index"]) + ',' + 'out_' + str(tout) + ',' + str(xx["spent"]) + ','+ str( Decimal(xx["value"]) / Decimal(100000000.0)) + ',' + 'SENT_COINS' + "\n")
# 					filePaymentOutAddr.write('out_' + str(tout) + ',' + rec + ',' + str(xx["spent"]) + ','+ str( Decimal(xx["value"]) / Decimal(100000000.0)) + ','+ 'WAS_SENT_TO' + "\n")
# 					fileAddressList.write(str(rec) + ',' + str(rec) + ',Address' + "\n")
# 					tout = tout+1
# 				except KeyError:
# 					rec = 'Unavailable'
	usock.close()
file.close()
fileIn.close()
fileOut.close()
fileTransList.close()
fileBlockTrans.close()
fileTransInRels.close()
filePaymentOutAddr.close()
filePaymentInAddr.close()
fileAddressList.close()
fileBlockList.close()
f.close();
print "Done"
Explanation: Import script - LOAD CSV?? - STOPPED HERE
graphdb lives here: /Users/eastblue/Documents/Neo4j/default.graphdb
```bash
./bin/neo4j-import --into /Users/eastblue/Documents/Neo4j/default.graphdb
--nodes /Users/eastblue/ds/metis/challenges/Proj_Kojak/btc/addresses.csv
--nodes /Users/eastblue/ds/metis/challenges/Proj_Kojak/btc/blockhash.csv
--nodes /Users/eastblue/ds/metis/challenges/Proj_Kojak/btc/tx.csv
--nodes /Users/eastblue/ds/metis/challenges/Proj_Kojak/btc/txin.csv
--nodes /Users/eastblue/ds/metis/challenges/Proj_Kojak/btc/txout.csv 
--relationships /Users/eastblue/ds/metis/challenges/Proj_Kojak/btc/relsPaymentsToAddress.csv 
--relationships /Users/davidfauth/neo4j_nyc_graphday/bitcoinData/poc_relsBlocksTransactions.txt 
--relationships /Users/davidfauth/neo4j_nyc_graphday/bitcoinData/poc_relsBlockData.txt
--relationships /Users/davidfauth/neo4j_nyc_graphday/bitcoinData/poc_relsRedeemedFromAddress.txt 
--relationships /Users/davidfauth/neo4j_nyc_graphday/bitcoinData/poc_relsTransactionsOut.txt 
--relationships /Users/davidfauth/neo4j_nyc_graphday/bitcoinData/poc_relsTransactionsIn.txt
```
--------------
REFERENCE
End of explanation
#!/usr/bin/env python
import simplejson as json
import httplib
import urllib2
from httplib import HTTPConnection, HTTPS_PORT
import ssl
from decimal import *
file = open("poc_transaction.txt", "w")
fileIn = open("poc_transactionsIn.txt", "w")
fileOut = open("poc_transactionsOut.txt", "w")
fileTransList = open("poc_transactionList.txt","w")
fileBlockTrans = open("poc_relsBlocksTransactions.txt","w")
fileTransOut = open("poc_relsTransactionsOut.txt","w")
fileTransInRels = open("poc_relsTransactionsIn.txt","w")
filePaymentOutAddr = open("poc_relsPaymentsToAddress.txt","w")
filePaymentInAddr = open("poc_relsRedeemedFromAddress.txt","w")
fileAddressList = open ("poc_addressList.txt","w")
fileBlockList = open("poc_blockdata.txt", "w")
file.write(":ID,Transaction_ID,Hash,Time,V_In,V_Out,:LABEL" + "\n")
fileIn.write(":ID,Transaction_ID,Transaction_Hash,Address,Spent,Value,:LABEL" + "\n")
fileOut.write(":ID,Transaction_ID,Transaction_Hash,Type,:LABEL" + "\n")
fileBlockTrans.write(":START_ID,:END_ID,:TYPE" + "\n")
fileTransOut.write(":START_ID,:END_ID,Spent,Value,:TYPE" + "\n")
fileTransInRels.write(":START_ID,:END_ID,Spent,Value,:TYPE" + "\n")
filePaymentOutAddr.write(":START_ID,:END_ID,Spent,Value,:TYPE" + "\n")
filePaymentInAddr.write(":START_ID,:END_ID,Spent,Value,:TYPE" + "\n")
fileAddressList.write(":ID,AddressID,:LABEL" + "\n")
fileBlockList.write(":ID,BlockID,Hash,Received_Time,Previous_Block_Hash,Transaction_Count,Height,:LABEL"  + "\n")
tout = 1
tin = 1
f = open('blockList.txt', 'r')
temp = f.read().splitlines()
for line in temp:
	words = line.split("|")
	s = words[1]
	
	url = "https://blockchain.info/rawblock/" + str(s)
	print url
	usock = urllib2.urlopen(url)
	data = usock.read()
	result = json.loads(data)
	hash = result['hash']
	block_index = result['block_index']
	height = result['height']
	size = result['size']
	main_chain = result['main_chain']
	prev_block = result['prev_block']
	try: 
		received_time = result['received_time']
	except KeyError:
		received_time = 'NA'
	n_tx = result['n_tx']
	fileBlockList.write(str(hash) + ',' + str(block_index) + ',' +str(hash) + ',' + str(received_time) + ',' + str(prev_block) + ',' + str(n_tx) + ',' + str(height) + ",BlockChain" + "\n");
	parent =  result["tx"]
	for item in parent:
		tx_index = str(item["tx_index"])
		tx_hash = str(item["hash"])
		file.write(str('trans' + str(item["tx_index"])) + ',' + str(item["tx_index"]) + ',' +str(item["hash"]) + ',' + str(item["time"]) + ',' + str(item["vin_sz"]) + ',' + str(item["vout_sz"]) + ",Transaction"+ "\n");
		fileBlockTrans.write(str(hash) + ',' + str('trans' + str(item["tx_index"])) + ',' + 'PART_OF_BLOCK' + "\n")
		if 'inputs' in item :
			for nn in item["inputs"]:
#				print nn["sequence"]
				if 'prev_out' in nn :
#					print nn["prev_out"]["addr"]
#					print nn["prev_out"]["spent"]
#					print nn["prev_out"]["value"]
					try: 
						strAddr = str(nn["prev_out"]["addr"])
					except KeyError:
						strAddr = 'NA'
					fileIn.write('transin_' + str(tin) + ',' + tx_index + ',' + str(tx_hash) + ',' + strAddr + ',' +str(nn["prev_out"]["spent"]) + ',' + str( Decimal(nn["prev_out"]["value"]) / Decimal(100000000.0))  + ",IncomingPayment" + "\n");
					fileTransInRels.write('transin_' + str(tin) + ',' + 'trans' + str(tx_index) + ',' + str(nn["prev_out"]["spent"]) + ','+ str( Decimal(nn["prev_out"]["value"]) / Decimal(100000000.0)) + ',' + 'INCOMING_PAYMENT' + "\n")
					filePaymentInAddr.write(strAddr + ',' + 'transin_' + str(tin) + ',' + str(nn["prev_out"]["spent"]) + ','+ str( Decimal(nn["prev_out"]["value"])/ Decimal(100000000.0)) + ','+ 'REDEEMED' + "\n")
					fileTransList.write('transin_' + str(tin) + ',' + tx_hash + "\n")
					fileAddressList.write(strAddr + ',' + strAddr + ',Address' + "\n")
					tin = tin + 1
					
		if 'out' in item :
			for xx in item["out"]:
#							print xx["tx_index"]
#							print xx["type"]
#							print xx["addr"]
#							print xx["spent"]
#							print xx["value"]
				try: 
					rec = str(xx["addr"])
					fileOut.write('out_' + str(tout) + ',' + str(xx["tx_index"]) + ',' + str(tx_hash) + ',' + str(xx["type"])  + ",OutgoingPayment"+ "\n");
					fileTransOut.write('trans' + str(xx["tx_index"]) + ',' + 'out_' + str(tout) + ',' + str(xx["spent"]) + ','+ str( Decimal(xx["value"]) / Decimal(100000000.0)) + ',' + 'SENT_COINS' + "\n")
					filePaymentOutAddr.write('out_' + str(tout) + ',' + rec + ',' + str(xx["spent"]) + ','+ str( Decimal(xx["value"]) / Decimal(100000000.0)) + ','+ 'WAS_SENT_TO' + "\n")
					fileAddressList.write(str(rec) + ',' + str(rec) + ',Address' + "\n")
					tout = tout+1
				except KeyError:
					rec = 'Unavailable'
	usock.close()
file.close()
fileIn.close()
fileOut.close()
fileTransList.close()
fileBlockTrans.close()
fileTransInRels.close()
filePaymentOutAddr.close()
filePaymentInAddr.close()
fileAddressList.close()
fileBlockList.close()
f.close();
print "Done"
Explanation: Original Blockchain API
End of explanation
# # NODES 
# - Convert txt files to csv
#   - http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html
# - nodes first
# - https://neo4j.com/docs/operations-manual/3.2/tools/import/file-header-format/#import-tool-id-spaces
# In[10]:
test = pd.read_csv('graph_addresses.txt', header = None, names = ['user:ID', ':LABEL'], delimiter= '\t')
# In[11]:
test.head()
# In[12]:
test2 = test.groupby('user:ID').sum()
# In[13]:
test2.head()
# In[27]:
test.to_csv('graph_addresses.csv')
# In[39]:
test.shape  # number of nodes
# # EDGES
# - Specify start, end, and type for each edge
# In[ ]:
test = pd.read_csv('txedgeunique.txt', header = None, delimiter= '\t')
# In[29]:
test1 = pd.read_csv('../btc/au_graph.txt', header = None, names = [':START_ID', 'stop', 'unixtime'], delimiter= '\t')
# In[30]:
test1[':END_ID'] = test1['stop']
# In[35]:
test1 = test1.drop('stop', axis = 1)
# test[':TYPE'] = 'TXN'
# In[36]:
test1.head()
# In[37]:
test1.to_csv('au_graph.csv')
# In[40]:
test1.shape # number of transactions
Explanation: OLD CODE
End of explanation
import pandas
from math import log10, floor
from scipy.constants import codata
 
def most_significant_digit(x):
    e = floor(log10(x))
    return int(x*10**-e)
def f(x):
    return most_significant_digit(abs(x))
# read in the ticker data
tick = pandas.read_csv('./your_ticker_data.csv')
tick_ret = tick.diff()
 
# count leading digits
data = tick_ret[tick_ret!=0]
counts = data.fillna(method='bfill').apply(f).value_counts()
total = counts.sum()
 
# expected number of each leading digit per Benford's law
benford = [total*log10(1 + 1./i) for i in range(1, 10)]
# plot actual vs expected
bins = np.arange(9)
error_config = {'ecolor': '0.3'}
r1 = plt.bar(bins, counts.values, 0.35, alpha=0.4, color='b', error_kw=error_config, label = 'actual')
r2 = plt.bar(bins + 0.35, benford, 0.35, alpha=0.4, color='r', error_kw=error_config, label = 'expected')
plt.xlabel('Most significant digit')
plt.ylabel('Occurence count')
plt.title('Leading digits in BTC-E ticker volume')
plt.xticks(bins + 0.35, bins+1)
plt.legend()
plt.show()
Explanation: benford's law test for ticker data
applies the law to a couple of days of BTC-E non-zero price returns.
https://plus.maths.org/content/looking-out-number-one
End of explanation
from neo4j.v1 import GraphDatabase, basic_auth
# In[ ]:
driver = GraphDatabase.driver("bolt://localhost:7687", auth=basic_auth("neo4j", "neo4j"))
session = driver.session()
session.run("CREATE (a:Person {name: {name}, title: {title}})",
          {"name": "Arthur", "title": "King"})
result = session.run("MATCH (a:Person) WHERE a.name = {name} "
                   "RETURN a.name AS name, a.title AS title",
                   {"name": "Arthur"})
for record in result:
  print("%s %s" % (record["title"], record["name"]))
session.close()
# In[ ]:
from py2neo import Graph, Path
graph = Graph()
tx = graph.cypher.begin()
for name in ["Alice", "Bob", "Carol"]:
    tx.append("CREATE (person:Person {name:{name}}) RETURN person", name=name)
alice, bob, carol = [result.one for result in tx.commit()]
friends = Path(alice, "KNOWS", bob, "KNOWS", carol)
graph.create(friends)
Explanation: Python Drivers
https://marcobonzanini.com/2015/04/06/getting-started-with-neo4j-and-python/
https://neo4j.com/developer/python/
End of explanation
<END_TASK>