<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Author
Step1:  First let's check if there are new or deleted files (only matching by file names).
Step2:  So we have the same set of files in both versions
Step3:  Let's make sure the structure hasn't changed
Step4:  All files have the same columns as before
Step5:  One important change is the one added to referentiel_code_rome, adding it might be the reason of all the other changes as it's adding a new job group and all other files would need to propagate that change.
New Job Group
Identify the New Job Group
Let's check it out. First let's make sure than no job groups were removed
Step6:  Alright, so the only change was the job group added
Step7:  Let's see if this is a different grouping of existing jobs or if it's entirely new jobs. First let's check the jobs in this new job group.
Step8:  Now let's see if those jobs were already there, and if so which were there job groups
Step9:  Alright, it seems that these are entirely new jobs. Just to make sure let's check with a keyword.
Step10:  What? Wait a minute! what happened to this job that looks almost exactly like the new one `Animatrice 2D - films d'animation'.
Step11:  OK, this one did not move at all. What is this other job group that seems so close to ours?
Step12:  Ouch, it's indeed quite close and might have fooled more than one jobseeker…
So we have an entirely new job group L1510 which stands for Films d'animation et effets spéciaux. It's quite close to E1205 (Réalisation de contenus multimédias) and by the past many jobs of the new job groups might have defaulted to similar jobs of E1205.
Let's check now the impact on the rest of the ROME datasets, especially to identify other changes that might have not be related to adding this job group.
Impact on ROME
Let's first check the ROME mobility (there were 8 new lines)
Step13:  Cool, we found our 8 new rows, and as expected it's linking to closeby job groups. We can see that the two job groups E1104 and E1205 are especially close as there are some mobility in both ways to and from the new job group.
Step14:  Let's seek the skills related to that new job group
Step15:  Some of the skills already existed (e.g. Technique de dessin), others have been added with this release specially for this job group (e.g. Logiciel de motion capture).
OK I think this is enough scrutiny for this new job group. Let's check out the other changes.
Other Changes
Let's first check the job names dataset. We've seen that some jobs were added for this job group but there might be others
Step16:  Those looks legitimate. New jobs are added regularly to ROME and this release makes no exception.
What about the skills?
<ASSISTANT_TASK:>
Python Code:
import collections
import glob
import os
from os import path
import matplotlib_venn
import pandas
rome_path = path.join(os.getenv('DATA_FOLDER'), 'rome/csv')
OLD_VERSION = '331'
NEW_VERSION = '332'
old_version_files = frozenset(glob.glob(rome_path + '/*{}*'.format(OLD_VERSION)))
new_version_files = frozenset(glob.glob(rome_path + '/*{}*'.format(NEW_VERSION)))
Explanation: Author: Pascal, pascal@bayesimpact.org
Date: 2016-06-28
ROME update from v331 to v332
In June 2017 a new version of the ROME was realeased. I want to investigate what changed and whether we need to do anything about it.
You might not be able to reproduce this notebook, mostly because it requires to have the two versions of the ROME in your data/rome/csv folder which happens only just before we switch to v332. You will have to trust me on the results ;-)
Skip the run test because it requires older versions of the ROME.
End of explanation
new_files = new_version_files - frozenset(f.replace(OLD_VERSION, NEW_VERSION) for f in old_version_files)
deleted_files = old_version_files - frozenset(f.replace(NEW_VERSION, OLD_VERSION) for f in new_version_files)
print('{:d} new files'.format(len(new_files)))
print('{:d} deleted files'.format(len(deleted_files)))
Explanation: First let's check if there are new or deleted files (only matching by file names).
End of explanation
new_to_old = dict((f, f.replace(NEW_VERSION, OLD_VERSION)) for f in new_version_files)
# Load all ROME datasets for the two versions we compare.
VersionedDataset = collections.namedtuple('VersionedDataset', ['basename', 'old', 'new'])
rome_data = [VersionedDataset(
        basename=path.basename(f),
        old=pandas.read_csv(f.replace(NEW_VERSION, OLD_VERSION)),
        new=pandas.read_csv(f))
    for f in sorted(new_version_files)]
def find_rome_dataset_by_name(data, partial_name):
    for dataset in data:
        if 'unix_{}_v{}_utf8.csv'.format(partial_name, NEW_VERSION) == dataset.basename:
            return dataset
    raise ValueError('No dataset named {}, the list is\n{}'.format(partial_name, [dataset.basename for d in data]))
Explanation: So we have the same set of files in both versions: good start.
Now let's set up a dataset that, for each table, links the old file and the new file.
End of explanation
for dataset in rome_data:
    if set(dataset.old.columns) != set(dataset.new.columns):
        print('Columns of {} have changed.'.format(dataset.basename))
Explanation: Let's make sure the structure hasn't changed:
End of explanation
same_row_count_files = 0
for dataset in rome_data:
    diff = len(dataset.new.index) - len(dataset.old.index)
    if diff > 0:
        print('{:d} values added in {}'.format(diff, dataset.basename))
    elif diff < 0:
        print('{:d} values removed in {}'.format(diff, dataset.basename))
    else:
        same_row_count_files += 1
print('{:d}/{:d} files with the same number of rows'.format(same_row_count_files, len(rome_data)))
Explanation: All files have the same columns as before: still good.
Now let's see for each file if they more or less rows.
End of explanation
job_groups = find_rome_dataset_by_name(rome_data, 'referentiel_code_rome')
obsolete_job_groups = set(job_groups.old.code_rome) - set(job_groups.new.code_rome)
obsolete_job_groups
Explanation: One important change is the one added to referentiel_code_rome, adding it might be the reason of all the other changes as it's adding a new job group and all other files would need to propagate that change.
New Job Group
Identify the New Job Group
Let's check it out. First let's make sure than no job groups were removed:
End of explanation
new_job_groups_codes = set(job_groups.new.code_rome) - set(job_groups.old.code_rome)
new_job_groups = job_groups.new[job_groups.new.code_rome.isin(new_job_groups_codes)]
new_job_groups                          
Explanation: Alright, so the only change was the job group added:
End of explanation
jobs = find_rome_dataset_by_name(rome_data, 'referentiel_appellation')
jobs.new[jobs.new.code_rome == 'L1510'].head()
Explanation: Let's see if this is a different grouping of existing jobs or if it's entirely new jobs. First let's check the jobs in this new job group.
End of explanation
jobs.old[jobs.old.code_ogr.isin(jobs.new[jobs.new.code_rome == 'L1510'].code_ogr)]
Explanation: Now let's see if those jobs were already there, and if so which were there job groups:
End of explanation
jobs.old[jobs.old.libelle_appellation_court.str.contains('Animatrice 2D', case=False)]
Explanation: Alright, it seems that these are entirely new jobs. Just to make sure let's check with a keyword.
End of explanation
jobs.new[jobs.new.code_ogr == 10969]
Explanation: What? Wait a minute! what happened to this job that looks almost exactly like the new one `Animatrice 2D - films d'animation'.
End of explanation
job_groups.new[job_groups.new.code_rome == 'E1205']
Explanation: OK, this one did not move at all. What is this other job group that seems so close to ours?
End of explanation
mobility = find_rome_dataset_by_name(rome_data, 'rubrique_mobilite')
mobility.new[(mobility.new.code_rome == 'L1510') | (mobility.new.code_rome_cible == 'L1510')]
Explanation: Ouch, it's indeed quite close and might have fooled more than one jobseeker…
So we have an entirely new job group L1510 which stands for Films d'animation et effets spéciaux. It's quite close to E1205 (Réalisation de contenus multimédias) and by the past many jobs of the new job groups might have defaulted to similar jobs of E1205.
Let's check now the impact on the rest of the ROME datasets, especially to identify other changes that might have not be related to adding this job group.
Impact on ROME
Let's first check the ROME mobility (there were 8 new lines):
End of explanation
job_groups.new[job_groups.new.code_rome.isin(('E1205', 'E1104'))]
Explanation: Cool, we found our 8 new rows, and as expected it's linking to closeby job groups. We can see that the two job groups E1104 and E1205 are especially close as there are some mobility in both ways to and from the new job group.
End of explanation
skills = find_rome_dataset_by_name(rome_data, 'referentiel_competence')
link = find_rome_dataset_by_name(rome_data, 'liens_rome_referentiels')
new_linked_skills = link.new.join(skills.new.set_index('code_ogr'), 'code_ogr')[
    ['code_rome', 'code_ogr', 'libelle_competence', 'libelle_type_competence']]
new_linked_skills[new_linked_skills.code_rome == 'L1510'].dropna()
Explanation: Let's seek the skills related to that new job group:
End of explanation
new_jobs = set(jobs.new.code_ogr) - set(jobs.old.code_ogr)
new_linked_skills[new_linked_skills.code_rome == 'L1510'].dropna()
Explanation: Some of the skills already existed (e.g. Technique de dessin), others have been added with this release specially for this job group (e.g. Logiciel de motion capture).
OK I think this is enough scrutiny for this new job group. Let's check out the other changes.
Other Changes
Let's first check the job names dataset. We've seen that some jobs were added for this job group but there might be others:
End of explanation
new_skills = set(skills.new.code_ogr) - set(skills.old.code_ogr)
skills_for_new_job_group = new_linked_skills[new_linked_skills.code_rome == 'L1510'].code_ogr
skills.new[skills.new.code_ogr.isin(new_skills) & (~skills.new.code_ogr.isin(skills_for_new_job_group))]
Explanation: Those looks legitimate. New jobs are added regularly to ROME and this release makes no exception.
What about the skills?
End of explanation
<END_TASK>