<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 2018-02-16 / FMA sub-sampling mk2
In case entrofy is too slow / doesn't work out, we'll make a backup sampler
Requirements
Step1:  Algorithm
For each instrument, split at the QUANTILE by likelihood
Skip tracks that have already appeared
Take BALANCE samples uniformly from above the quantile, and 1-BALANCE from below.
Handle instruments by increasing order of median likelihood
UPDATES
Step2:  Figures
Step3:  Cruft
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
from tqdm import tqdm_notebook as tqdm
import gzip
import seaborn as sns
%matplotlib inline
# Filter down to the set with valid licenses
#df_valid = pd.read_csv('/home/bmcfee/data/cc_tracks.csv.gz', usecols=[0])['track_id']
df_valid = pd.read_csv('/home/bmcfee/data/cc_tracks.csv.gz', usecols=[0], nrows=1000000)['track_id']
df_valid = df_valid.apply(lambda x: '{:06d}'.format(x))
valid_set = set(df_valid)
#df = pd.read_csv('/home/bmcfee/data/vggish-likelihoods-a226b3-maxagg10.csv.gz', index_col=0)
df = pd.read_csv('/home/bmcfee/data/vggish-likelihoods-a226b3-maxagg10.csv.gz', nrows=1000000, index_col=0)
df = df.drop([x for x in tqdm(df.index) if x[:6] not in valid_set])
df.head(5)
Explanation: 2018-02-16 / FMA sub-sampling mk2
In case entrofy is too slow / doesn't work out, we'll make a backup sampler
Requirements:
subsample of size N
for each instrument, at least K positives
2018-03-15 / Updates
License filtering: CC only
Sampling: 50/50 from 80/20 per instrument
2018-03-26 / Resample
Sample per instrument category
No explicit negatives
Save categorical samples independently
End of explanation
K = 1000
N = K * 23
#BALANCE = 0.5
#QUANTILE = 0.8
BALANCE = 1.0
QUANTILE = 0.9
dfm = df.median(axis=0)
inst_order = dfm.sort_values().index
inst_order
idx_inst = df[inst_order[0]].sort_values(ascending=False).index
len(idx_inst)
rng = np.random.RandomState(20180326)
from collections import defaultdict
idx_set = {k: set() for k in inst_order}
track_set = set()
N_ABOVE = int(BALANCE * K)
N_BELOW = int(K * (1 - BALANCE))
pos_thresh = int((1 - QUANTILE) * len(idx_inst))
for col in tqdm(inst_order):
    print(col)
    # This gives a descending sort of the instruments by likelihood
    idx_inst = df[col].sort_values(ascending=False).index
    
    # Get the top quantile
    idx_pos = idx_inst[:pos_thresh]
    # Randomly permute
    idx_pos = rng.permutation(idx_pos)
    
    count = 0
    
    # Make a uniform selection within the top 
    for v in idx_pos:
        track, seg = v.split('_', maxsplit=1)
        
        if track not in track_set:
            idx_set[col].add(v)
            track_set.add(track)
            count += 1
        
        if count == N_ABOVE:
            break
            
    # Get the bottom quantile
    #idx_neg = idx_inst[pos_thresh:]
    # Randomly permute
    #idx_neg = rng.permutation(idx_neg)
    
    #count = 0
    
    # Make a uniform selection within the bottom 
    #for v in idx_neg:
    #    track, seg = v.split('_', maxsplit=1)
        
    #    if track not in track_set:
    #        idx_set[col].add(v)
    #        track_set.add(track)
    #        count += 1
        
    #    if count == N_BELOW:
    #        break
N_ABOVE
dict(idx_set)
df_sample = df.loc[idx_set['accordion']]
df_sample
import json
for inst in idx_set:
    with open('../notebooks/subsample_idx_{}.json'.format(inst), 'w') as fd:
        json.dump(list(idx_set[inst]), fd, indent=2)
#with open('../notebooks/subsample_idx_greedy_balanced.json', 'w') as fd:
#    json.dump(list(idx_set), fd, indent=2)
Explanation: Algorithm
For each instrument, split at the QUANTILE by likelihood
Skip tracks that have already appeared
Take BALANCE samples uniformly from above the quantile, and 1-BALANCE from below.
Handle instruments by increasing order of median likelihood
UPDATES: 2018-03-26
End of explanation
Xs = df_sample.values
np.cov(Xs.T)
import matplotlib.pyplot as plt
%matplotlib nbagg
#C = np.cov(Xs.T)
C = df_sample.corr()
Co = df.corr()
plt.figure(figsize=(12, 5))
plt.subplot(1,2,1)
#sns.pointplot(data=df_sample, orient='h', join=False, markers='o', capsize=1, )
sns.boxplot(data=df_sample, orient='h', fliersize=1)
plt.title('Instrument likelihood')
plt.ylim([-0.5, 22.5])
plt.gca().invert_yaxis()
plt.subplot(1,2,2)
mask = np.zeros_like(C, dtype=np.bool)
mask[np.triu_indices_from(mask, k=1)] = True
cmap = sns.diverging_palette(220, 10, as_cmap=True)
sns.heatmap(C, mask=mask, cmap=cmap, vmax=.3, center=0,
            square=True, linewidths=.5, cbar_kws={"shrink": .5})
plt.title('Subsample instrument correlations')
plt.tight_layout()
plt.figure(figsize=(12, 5))
plt.subplot(1,2,1)
#sns.pointplot(data=df, orient='h', join=False, markers='o', capsize=1, )
sns.boxplot(data=df, orient='h', fliersize=1)
plt.title('Instrument likelihood')
plt.ylim([-0.5, 22.5])
plt.gca().invert_yaxis()
plt.subplot(1,2,2)
mask = np.zeros_like(Co, dtype=np.bool)
mask[np.triu_indices_from(mask, k=1)] = True
cmap = sns.diverging_palette(220, 10, as_cmap=True)
sns.heatmap(Co, mask=mask, cmap=cmap, vmax=.3, center=0,
            square=True, linewidths=.5, cbar_kws={"shrink": .5})
plt.title('Full instrument correlations')
plt.tight_layout()
Explanation: Figures
End of explanation
idx_set = set()
track_set = set()
for col in tqdm(dfm.sort_values().index):
    # This gives a descending sort of the instruments by likelihood
    idx_inst = df[col].sort_values(ascending=False).index
    
    count = 0
    
    # Make a uniform selection within the top 
    for v in idx_inst:
        track, seg = v.split('_', maxsplit=1)
        
        if track in track_set:
            continue
            
        idx_set.add(v)
        track_set.add(track)
        count += 1
        
        if count == K:
            break
Explanation: Cruft
End of explanation
<END_TASK>