<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Figure 4
Step1:  Transfer learning vs. nightlights
In these experiments, we compare the performance of the transfer learning model based on satellite imagery with the performance of a model that uses nightlights.
The parameters needed to produce the plots for Panels A and B are as follows
Step2:  Panel B
Step3:  Randomization tests
In these experiments, we randomly reassign daytime imagery to survey locations and retrain the model on incorrect images (see SM 1.7).
The parameters needed to produce the plots for Panels C and D are as follows
Step4:  Panel D
<ASSISTANT_TASK:>
Python Code:
from fig_utils import *
import matplotlib.pyplot as plt
import time
%matplotlib inline
Explanation: Figure 4: Evaluation of model performance
This notebook generates individual panels of Figure 4 in "Combining satellite imagery and machine learning to predict poverty".
End of explanation
country_path = '../data/output/LSMS/pooled/'
percentiles = [0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.35,
               0.40, 0.45, 0.50, 0.55, 0.60, 0.65, 0.70,
               0.75, 0.80, 0.85, 0.90, 0.95, 1.00]
survey = 'lsms'
dimension = 10
k = 10
k_inner = 5
trials = 5
poverty_line = 1.90
multiples = [1, 2, 3]
t0 = time.time()
compare_models(country_path, survey, percentiles, dimension, k, k_inner,
             trials, poverty_line, multiples)
t1 = time.time()
print 'Finished in {} seconds'.format(t1-t0)
Explanation: Transfer learning vs. nightlights
In these experiments, we compare the performance of the transfer learning model based on satellite imagery with the performance of a model that uses nightlights.
The parameters needed to produce the plots for Panels A and B are as follows:
country_paths: Paths of directories containing pooled survey data
percentiles: Wealth percentiles to evaluate
survey: Either 'lsms' or 'dhs'
dimension: Number of dimensions to reduce image features to using PCA
k: Number of cross validation folds
k_inner: Number of inner cross validation folds for selection of regularization parameter
trials: Number of trials to average over
poverty_line: International poverty line ($1.90/capita/day)
multiples: Multiples of the poverty line to plot
For many trials, it will take several minutes or more to produce the plots. For 100 trials, it should take 40-60 minutes for LSMS and longer for DHS.
Each data directory should contain the following 4 files:
conv_features.npy: (n, 4096) array containing image features corresponding to n clusters
nightlights.npy: (n,) vector containing the average nightlights value for each cluster
households.npy: (n,) vector containing the number of households for each cluster
image_counts.npy: (n,) vector containing the number of images available for each cluster
Each data directory should also contain one of the following:
consumptions.npy: (n,) vector containing average cluster consumption expenditures for LSMS surveys
assets.npy: (n,) vector containing average cluster asset index for DHS surveys
Exact results may differ slightly with each run due to randomly splitting data into training and test sets.
Panel A: Pooled LSMS
End of explanation
country_path = '../data/output/DHS/pooled/'
percentiles = [0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.35,
               0.40, 0.45, 0.50, 0.55, 0.60, 0.65, 0.70,
               0.75, 0.80, 0.85, 0.90, 0.95, 1.00]
survey = 'dhs'
dimension = 10
k = 5
k_inner = 3
trials = 3
t0 = time.time()
compare_models(country_path, survey, percentiles, dimension, k, k_inner,
             trials, poverty_line, multiples)
t1 = time.time()
print 'Finished in {} seconds'.format(t1-t0)
Explanation: Panel B: Pooled DHS
End of explanation
# Parameters
country_names = ['nigeria', 'tanzania', 'uganda', 'malawi', 'pooled']
country_paths = ['../data/output/LSMS/nigeria/',
                '../data/output/LSMS/tanzania/',
                '../data/output/LSMS/uganda/',
                '../data/output/LSMS/malawi/',
                '../data/output/LSMS/pooled/']
survey = 'lsms'
dimension = 100
k = 3
k_inner = 3
points = 10
alpha_low = 0
alpha_high = 3
trials = 100
run_randomization_test(country_names, country_paths, survey, dimension,
                       k, k_inner, points, alpha_low, alpha_high, trials)
Explanation: Randomization tests
In these experiments, we randomly reassign daytime imagery to survey locations and retrain the model on incorrect images (see SM 1.7).
The parameters needed to produce the plots for Panels C and D are as follows:
country_names: Names of countries as a list of lower-case strings
country_paths: Paths of directories containing survey data corresponding to specified countries
survey: Either 'lsms' or 'dhs'
dimension: Number of dimensions to reduce image features to using PCA
k: Number of cross validation folds
k_inner: Number of inner cross validation folds for selection of regularization parameter
points: Number of regularization parameters to try
alpha_low: Log of smallest regularization parameter to try
alpha_high: Log of largest regularization parameter to try
trials: Number of trials for shuffled distribution
If trials is large (>100), producing the plots will take more than a couple of minutes.
Each data directory should contain the following 3 files:
cluster_conv_features.npy: (n, 4096) array containing image features corresponding to n clusters
cluster_households.npy: (n,) vector containing the number of households for each cluster
cluster_image_counts.npy: (n,) vector containing the number of images available for each cluster
Each data directory should also contain one of the following:
cluster_consumptions.npy: (n,) vector containing average cluster consumption expenditures for LSMS surveys
cluster_assets.npy: (n,) vector containing average cluster asset index for DHS surveys
Exact results may differ slightly with each run due to randomly splitting data into training and test sets.
Panel C: LSMS consumption expenditures
End of explanation
# Parameters
country_names = ['nigeria', 'tanzania', 'uganda', 'malawi', 'rwanda', 'pooled']
country_paths = ['../data/output/DHS/nigeria/',
                '../data/output/DHS/tanzania/',
                '../data/output/DHS/uganda/',
                '../data/output/DHS/malawi/',
                '../data/output/DHS/rwanda/',
                '../data/output/DHS/pooled/']
survey = 'dhs'
dimension = 100
k = 3
k_inner = 3
points = 10
alpha_low = 0
alpha_high = 3
trials = 100
run_randomization_test(country_names, country_paths, survey, dimension,
                       k, k_inner, points, alpha_low, alpha_high, trials)
Explanation: Panel D: DHS assets
End of explanation
<END_TASK>