<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Experiment
Step1:  Load and check data
Step2:  ## Analysis
Experiment Details
Step3:  Which level of hebbian pruning was better?
Step4:  No evidence of hebbian learning improving performance. Actually the opposite behavior, there is a clear reduction in acc as hebbian prune percentage increase from 0 to 0.5
Did hebbian grow help?
Step5:  Random Growth outperforms growing connections by using coactivations (hebbian grow) by 0.2%
Step6:  Which is better, kwinners or ReLU?
<ASSISTANT_TASK:>
Python Code:
%load_ext autoreload
%autoreload 2
import sys
sys.path.append("../../")
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import os
import glob
import tabulate
import pprint
import click
import numpy as np
import pandas as pd
from ray.tune.commands import *
from dynamic_sparse.common.browser import *
Explanation: Experiment:
Analyze Hebbian Learning with different choices of hyperparameters
- Different prune perc values
- Hebbian Grow or not
- Kwinners or RELU
Motivation.
Verify if pruning by coactivations is better than pruning by magnitude
Verify if adding weights by coactivations is better than random
Conclusion
Negative correlation between accuracy and hebbian pruning percentage - 0.4% diff in acc between base model (0.976) and pruning 30% every epoch by hebbian learning (0.972). In contrast to what is seen in the magnitude based pruning
Random growth outperforms hebbian pruning by ~ 0.2%
ReLU better than KWinners (with 25% on perc), by ~ 0.2%
End of explanation
exps = ['neurips_debug_test2', ]
paths = [os.path.expanduser("~/nta/results/{}".format(e)) for e in exps]
df = load_many(paths)
df.head(5)
df.columns
df.shape
df.iloc[1]
df.groupby('model')['model'].count()
Explanation: Load and check data
End of explanation
# Did any  trials failed?
df[df["epochs"]<30]["epochs"].count()
# Removing failed or incomplete trials
df_origin = df.copy()
df = df_origin[df_origin["epochs"]>=30]
df.shape
# which ones failed?
# failed, or still ongoing?
df_origin['failed'] = df_origin["epochs"]<30
df_origin[df_origin['failed']]['epochs']
# helper functions
def mean_and_std(s):
    return "{:.3f} ± {:.3f}".format(s.mean(), s.std())
def round_mean(s):
    return "{:.0f}".format(round(s.mean()))
stats = ['min', 'max', 'mean', 'std']
def agg(columns, filter=None, round=3):
    if filter is None:
        return (df.groupby(columns)
             .agg({'val_acc_max_epoch': round_mean,
                   'val_acc_max': stats,                
                   'model': ['count']})).round(round)
    else:
        return (df[filter].groupby(columns)
             .agg({'val_acc_max_epoch': round_mean,
                   'val_acc_max': stats,                
                   'model': ['count']})).round(round)
Explanation: ## Analysis
Experiment Details
End of explanation
agg(['hebbian_prune_perc'])
relu_only = (df['use_kwinners'] == False)
agg(['hebbian_prune_perc'], relu_only)
kwinners_only = (df['use_kwinners'] == True)
agg(['hebbian_prune_perc'], kwinners_only)
Explanation: Which level of hebbian pruning was better?
End of explanation
with_pruning = (df['hebbian_prune_perc'] > 0)
agg(['hebbian_grow'], with_pruning)
Explanation: No evidence of hebbian learning improving performance. Actually the opposite behavior, there is a clear reduction in acc as hebbian prune percentage increase from 0 to 0.5
Did hebbian grow help?
End of explanation
with_pruning = (df['hebbian_prune_perc'] > 0)
agg(['hebbian_prune_perc', 'hebbian_grow'])
Explanation: Random Growth outperforms growing connections by using coactivations (hebbian grow) by 0.2%
End of explanation
agg(['use_kwinners'])
agg(['hebbian_prune_perc', 'use_kwinners'])
Explanation: Which is better, kwinners or ReLU?
End of explanation
<END_TASK>