<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 This is a follow-along with How to write custom omdels with fast.ai
How to Write Custom Models with Fast.AI
Example on how to modify fastai to use a custom pretrained network.
Step2:  Custom ConvnetBuilder
This ConvnetBuilderVGG is similar to ConvnetBuilder. It's modifying VGG16. The default vgg16 in FastAI 'cuts' all the fully connected layers. In this example we keep all layers but the last one. There's a very small dataset so the intention is to train the last 2 fully connected layers.
Step3:  NOTE
Step4:  So, the numbers in the top layers are different from Yannet's, but the Conv layer pars are exactly the same. Pretty sure this is because when you finetune a model, the FC layer/s you stack ontop are randomly initialized, but the Conv layers coming from a pretrained network (PyTorch VGG 16), are necessarily exactly the same, given you got the same pretrained weights.
Cross Validation
Step5:  Here's code to do cross-validation
Step6:  How to train is a W.I.P. --- great..
Step7:  Predictions
Step8:  This'll actually take forever on an i5 CPU, but it does start training so, that's good.
Step9:  Alrighty, that gave me some idea of how a custom model is added to FastAI.
Getting length of dataset. Dataset is my custom-built set for my G-LOC-Detector.
<ASSISTANT_TASK:>
Python Code:
%reload_ext autoreload
%autoreload 2
%matplotlib inline
from fastai.transforms import *
from fastai.conv_learner import *
from fastai.model import *
from fastai.dataset import *
from fastai.sgdr import *
from fastai.plots import *
import pandas as pd
import numpy as np
path = 'data/gloc/'
model_path = path + 'results/'
Explanation: This is a follow-along with How to write custom omdels with fast.ai
How to Write Custom Models with Fast.AI
Example on how to modify fastai to use a custom pretrained network.
End of explanation
# to override the fastai vgg16 function
from torchvision.models import vgg16
# Creates a ConvnetBuilder with all pretrained layers from vgg16 but the last fully connected layer
class ConvnetBuilderVGG():
    
    Class representing a convolutional network.
    Arguments:
        c (int): size of the last layer
        is_multi (bool): is multilabel classification
        is_reg (bool): is a regression
        ps (float): dropout parameter for last layer
    
    
    def __init__(self, c, is_multi, is_reg, ps=None):
        self.c, self.is_multi, self.is_reg = c, is_multi, is_reg
        self.ps = ps or 0.5
        vgg = vgg16(True) # NOTE: okay so I need to study how PyTorch does this
        self.lr_cut = 30
        layers = children(vgg.features)
        layers += [Flatten()] + children(vgg.classifier)[:5]
        #self.nf = 4096
        # here top model is everything but the last layer
        self.top_model = nn.Sequential(*layers) # NOTE: I need to find out what the fn(*arg) syntax is
        
        fc_layers = self.create_fc_layer(4096, c, p=None)
        self.n_fc = len(fc_layers)
        self.fc_model = to_gpu(nn.Sequential(*fc_layers))
        apply_init(self.fc_model, kaiming_normal)
        self.model = to_gpu(nn.Sequential(*(layers+fc_layers)))
        
    def create_fc_layer(self, ni, nf, p, actn=None):
        res=[]
        if p: res.append(nn.Dropout(p=p))
        res.append(nn.Linear(in_features=ni, out_features=nf))
        if actn: res.append(actn())
        return res
    @property # NOTE: I also need to learn Python Static Method syntax --> https://stackoverflow.com/questions/400739/what-does-asterisk-mean-in-python
    def name(self): return "vgg16"
    
    def get_layer_groups(self, do_fc=False):
        if do_fc:
            m, idxs = self.fc_model, []
        else:
            m, idxs = self.model, [self.lr_cut, -self.n_fc]
        lgs = list(split_by_idxs(children(m), idxs))
        return lgs
bs=32; sz=224
f_model = vgg16
n = 7637
val_idxs = get_cv_idxs(n, 0, val_pct=0.2)
tfms = tfms_from_model(f_model, sz) # NOTE: how would it know, if this is a custom/PyTorch model?
data = ImageClassifierData.from_csv(path, 'train', f'{path}train.csv', bs, tfms, 
                                    val_idxs=val_idxs, continuous=True)
# note precompute=False
models = ConvnetBuilderVGG(data.c, data.is_multi, data.is_reg)
models.model
class ConvLearnerVGG(ConvLearner):
    
    # rewriting pretrained
    @classmethod
    def pretrained(cls, data, ps=None, **kwargs):
        models = ConvnetBuilderVGG(data.c, data.is_multi, data.is_reg, ps=ps)
        return cls(data, models, **kwargs)
    
    # redefining freeze to freeze everything but last layer
    def freeze(self):
        layers = children(self.model)
        n = len(layers)
        for λ in layers:
            λ.trainable=False
            for p in λ.parameters(): p.requires_grad=False
        λ = layers[n-1]
        λ.trainable=True
        for p in λ.parameters(): p.requires_grad=True
    
    def unfreeze_prev_layer(self):
        layers = children(self.model)
        λ = layers[35]
        λ.trainable=True
        for p in λ.parameters(): p.requires_grad=True
bs = 32; sz = 224
f_model = vgg16
n = 7637
val_idxs = get_cv_idxs(n, 0, val_pct=0.2)
tfms = tfms_from_model(f_model, sz)
data = ImageClassifierData.from_csv(path, 'train', f'{path}train.csv', bs, tfms, 
                                    val_idxs=val_idxs, continuous=True)
learn = ConvLearnerVGG.pretrained(data, ps=0.0, precompute=False)
Explanation: Custom ConvnetBuilder
This ConvnetBuilderVGG is similar to ConvnetBuilder. It's modifying VGG16. The default vgg16 in FastAI 'cuts' all the fully connected layers. In this example we keep all layers but the last one. There's a very small dataset so the intention is to train the last 2 fully connected layers.
End of explanation
m = learn.models.model
trainable_params_(m)
learn.unfreeze_prev_layer()
trainable_params_(m)
Explanation: NOTE: this is on my MacBook w/o a GPU.
End of explanation
bs=32; sz=224
n = 7637
transforms_basic = [RandomRotateXY(10), RandomDihedralXY()]
transforms_basic = [RandomRotateXY(10)]
Explanation: So, the numbers in the top layers are different from Yannet's, but the Conv layer pars are exactly the same. Pretty sure this is because when you finetune a model, the FC layer/s you stack ontop are randomly initialized, but the Conv layers coming from a pretrained network (PyTorch VGG 16), are necessarily exactly the same, given you got the same pretrained weights.
Cross Validation
End of explanation
def get_model_i(i=0):
    val_idxs = get_cv_idxs(n, i, val_pct=0.1)
    tfms = tfms_from_model(f_model, sz, aug_tfms=transforms_basic, max_zoom=1.05)
    data = ImageClassifierData.from_csv(path, 'train', f'{path}train.csv', bs, tfms, 
                                        val_idxs=val_idxs, suffix='.jpg', continuous=True)
    learn = ConvLearnerVGG.pretrained(data, ps=0.0, precompute=False)
    return learn
Explanation: Here's code to do cross-validation:
End of explanation
def fit_and_predict(learn):
    learn.fit(1e-3, 3)
    learn.fit(1e-4, 4)
    print("unfreezing")
    learn.unfreeze_prev_layer()
    #learn.fit(1e-5, 3, cycle_len=1, cycle_mult=2)
    learn.fit(1e-5, 3)
    return learn.TTA()
Explanation: How to train is a W.I.P. --- great..
End of explanation
preds = []
for i in range(11):
    print("iteration ", i)
    learn = get_model_i(i)
    preds.append(fit_and_predict(learn))
Explanation: Predictions
End of explanation
def reshape_preds(preds):
    predictions = [preds[i][0] for i in range(11)]
    y = [preds[i][1] for i in range(11)]
    pp = np.vstack(predictions)
    yy = np.vstack(y)
    print(yy.shape)
    pp = np.maximum(pp, 0.0)
    err = np.abs(pp - yy).mean()
    print("err", err)
reshape_preds(preds)
Explanation: This'll actually take forever on an i5 CPU, but it does start training so, that's good.
End of explanation
df = pd.read_csv(path + 'train.csv')
len(df['id'])
Explanation: Alrighty, that gave me some idea of how a custom model is added to FastAI.
Getting length of dataset. Dataset is my custom-built set for my G-LOC-Detector.
End of explanation
<END_TASK>