<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Import and tabular transformation of the json file
Step1:  Inspecting unique values in each columns
Step2:  Selecting only relevant columns
Step3:  still needs to be done 
Step4:  Saving selection
Step5:  Selecting a subset of the data
Step6:  New column with all votes summary with parlementary id as key
Step7:  Preparing data for d3 
Step8:  Selecting a subset of the data to export
<ASSISTANT_TASK:>
Python Code:
with open('d3/mapHemicycle/data/scrutins.json', 'r') as f:
    json_data = json.load(f)
json_data.keys()
json_data['scrutins'].keys()
df = pd.io.json.json_normalize(json_data['scrutins']['scrutin'])
Explanation: Import and tabular transformation of the json file
End of explanation
for col in df.columns:
    print ('____________________')
    print (col)
    try:
        print (df[col].unique())
    except:
        print ('error col ' + col)
Explanation: Inspecting unique values in each columns
End of explanation
df_sel = df[['dateScrutin', 'demandeur.texte',
       'miseAuPoint.abstentions.votant',
       'miseAuPoint.abstentions.votant.acteurRef',
       'miseAuPoint.abstentions.votant.mandatRef',
       'miseAuPoint.contres.votant', 'miseAuPoint.contres.votant.acteurRef',
       'miseAuPoint.contres.votant.mandatRef',
       'miseAuPoint.nonVotantsVolontaires.votant',
       'miseAuPoint.pours.votant', 'miseAuPoint.pours.votant.acteurRef',
       'miseAuPoint.pours.votant.mandatRef', 'modePublicationDesVotes',
       'numero', 'objet.libelle',
       'quantiemeJourSeance', 'seanceRef', 'sessionRef', 'sort.code',
       'sort.libelle', 'syntheseVote.annonce',
       'syntheseVote.decompte.abstention', 'syntheseVote.decompte.contre',
       'syntheseVote.decompte.nonVotant', 'syntheseVote.decompte.pour',
       'syntheseVote.nbrSuffragesRequis', 'syntheseVote.nombreVotants',
       'syntheseVote.suffragesExprimes', 'titre', 'typeVote.codeTypeVote',
       'typeVote.libelleTypeVote', 'typeVote.typeMajorite', 'uid',
       'ventilationVotes.organe.groupes.groupe']]
df_sel.head()
Explanation: Selecting only relevant columns
End of explanation
def map_all_votes(all_votes):
    dict_votes = {'pours':[], 'contres':[], 'nonVotants':[], 'abstentions':[]}
    for parl_group in all_votes:
        decompte_nominatif = parl_group['vote']['decompteNominatif']
        for voteposition in decompte_nominatif.keys():
            try:
                dict_votes[voteposition] += [vote['acteurRef'] for vote in decompte_nominatif[voteposition]['votant']]
            except:
                dict_votes[voteposition] += []
    return [dict_votes['pours'], dict_votes['contres'], dict_votes['nonVotants'], dict_votes['abstentions'] ]
df_sel['votes_pour'], df_sel['votes_contre'], df_sel['nonVotants'], df_sel['abstentions'] = zip(*df_sel['ventilationVotes.organe.groupes.groupe'].map(map_all_votes))
df_sel.head()
Explanation: still needs to be done : consolidating with these columns :
miseAuPoint.abstentions.votant
miseAuPoint.contres.votant
miseAuPoint.nonVotantsVolontaires.votant
miseAuPoint.pours.votant
Summary columns with list of votes
End of explanation
df_sel.to_pickle('data/scrutins/tabular_data_scrutin_sel.pkl')
df_sel.to_csv('data/scrutins/tabular_data_scrutin_sel.csv')
Explanation: Saving selection
End of explanation
df_short = df_sel[['dateScrutin', 'demandeur.texte', 'modePublicationDesVotes', 'numero', 'objet.libelle', 'quantiemeJourSeance', 'sort.code', 'syntheseVote.decompte.abstention',
         'syntheseVote.decompte.contre', 'syntheseVote.decompte.nonVotant', 'syntheseVote.decompte.pour',
        'syntheseVote.nbrSuffragesRequis', 'syntheseVote.nombreVotants',
       'syntheseVote.suffragesExprimes', 'titre', 'typeVote.libelleTypeVote', 'typeVote.typeMajorite', 'votes_pour', 'votes_contre',
       'nonVotants', 'abstentions']]
df_short = df_short.rename(columns={'demandeur.texte':'demandeur', 'sort.code': 'resultat','syntheseVote.decompte.abstention': 'nombre_abstentions',
                            'syntheseVote.decompte.pour': 'nombre_pour',
                            'syntheseVote.decompte.contre': 'nombre_contre',
                            'syntheseVote.decompte.nonVotant': 'nombre_nonVotant',
                            'syntheseVote.nbrSuffragesRequis': 'nombre_suffrages_requis',
                            'syntheseVote.nombreVotants': 'nombre_votants',
                            'syntheseVote.suffragesExprimes': 'nombre_suffrages_exprimes',
                            'typeVote.libelleTypeVote': 'type_vote',
                            'typeVote.typeMajorite': 'type_majorite',
                                   'objet.libelle':'libelle'})
df_short.info()
Explanation: Selecting a subset of the data
End of explanation
typevote_dict = {"votes_pour":"Pour", "votes_contre":"Contre", "nonVotants":"Non-votant", "abstentions":"Abstention"}
def populate_summary_col(row):
    new_dict = {}
    for keyvote, valuevote in typevote_dict.items():
        for vote in row[keyvote]:
            new_dict[vote] = valuevote
    return new_dict
df_short['all_votes'] = df_short.apply(populate_summary_col, axis=1)
df_short.to_pickle('data/scrutins/tabular_data_scrutin_sh.pkl')
df_short.to_csv('data/scrutins/tabular_data_scrutin_sh.csv')
df_short_for_d3 = df_short.copy()
Explanation: New column with all votes summary with parlementary id as key
End of explanation
df_short_for_d3["votes_pour"] = df_short_for_d3.votes_pour.astype('str').map(lambda x: x.replace("'", '"'))
df_short_for_d3["votes_contre"] = df_short_for_d3.votes_contre.astype('str').map(lambda x: x.replace("'", '"'))
df_short_for_d3["nonVotants"] = df_short_for_d3.nonVotants.astype('str').map(lambda x: x.replace("'", '"'))
df_short_for_d3["abstentions"] = df_short_for_d3.abstentions.astype('str').map(lambda x: x.replace("'", '"'))
df_short_for_d3['all_votes'] = df_short_for_d3.all_votes.astype('str').map(lambda x: x.replace("'", '"'))
Explanation: Preparing data for d3 : changing dict column to valid json
End of explanation
df_short_for_d3 = df_short_for_d3[['dateScrutin', 'demandeur', 'modePublicationDesVotes', 'numero',
       'libelle', 'quantiemeJourSeance', 'resultat', 'nombre_abstentions',
       'nombre_contre', 'nombre_nonVotant', 'nombre_pour',
       'nombre_suffrages_requis', 'nombre_votants',
       'nombre_suffrages_exprimes', 'titre', 'type_vote', 'type_majorite',
       'all_votes']]
df_short_for_d3.to_csv('d3/mapHemicycle/data/scrutins_tabular.csv')
Explanation: Selecting a subset of the data to export
End of explanation
<END_TASK>