<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 2018/9/15-16 WNixalo
https
Step1:  the torch.Tensor.sum(dim) call takes an integer argument as the axis along which to sum. This applies to NumPy arrays as well.
In this case xb.sum(-1) will turn a 64x784 tensor into a size 64 tensor. This creates a tensor with each element being the total sum of its corresponding size 784 (28x28 flattened) image from the minibatch.
Step2:  torch.unsqueeze returns a tensor with a dimension of size 1 inserted at the specified position.
the returned tensor shares the smae underlying data with this tensor.
Step3:  taking a look at what .unsqueeze does; what does the tensor look like right before unsqueeze is applied to it?
Step4:  making sure I didn't need parentheses there
Step5:  Okay so .unsqueeze turns the size 64 tensor into a 64x1 tensor, so it's nicely packaged up with the first element being the 64-long vector ... or something like that right?
Step6:  The unsqueezed tensor doesn't look as 'nice'.. I guess. So it's packaged into a single column vector because we'll need that for the linear algebra we'll do to it later yeah?
Step7:  Oh this is cool. I was wondering how .unsqeeze worked for tensors with multiple items in multiple dimensions (ie
Step8:  So .unsqueeze turns our size 64x10 ... ohhhhhhhh I misread
Step9:  Oh this is where I was confused. I'm not throwing xb into Log Softmax. I'm throwing xb • w + bias. The shape going into the log softmax function is not 64x784, it's 64x10. Yeah that makes sense. well duh it has to. Each value in the tensor is an activation for a class, for each image in the minibatch. So by the magic of machine learning, each activation encapsulates the effect of the weights and biases on that input element with respect to that class.
So that means that the .unsqueeze oepration is not going to be giving a 64x784 vector.
Step10:  Note the loss equals that in cell Out[25] above as it should.
Back to teasing this apart by hand.
The minibatch
Step11:  The minibatch's activations as they head into the Log Softmax
Step12:  The minibatch activations after the Log Softmax and before heading into Negative Log Likelihood
Step13:  The loss value computed via NLL on the Log Softmax activations
Step14:  Okay. Now questions. What is indexing input by [range(target.shape[0]), target] supposed to be doing? I established before that A[range(n)] is valid if n ≤ A.shape[0]. So what's going on is I'm range-indexing the 1st dimension of the LogSoftmax activations with the length of the target tensor, and the rest of the dimension indices being the ..target tensor itself?
That means the index is this
Step15:  Okay. What does it look like when I index a tensor – forget range-idx for now – with another tensor?
Step16:  Okay..
Step17:  Uh, moment of truth
Step18:  Oof course. What happened. Is it.. yes. I'm indexing the wrong array. Also no value in target is greater than the number of classes ... oh... oh ffs. Okay.
I range index by the length of target's first dim to get the entire first dim of the LogSoftmax activations, and each vector in that index is itself indexed by the value of the target.
Less-shitty English
Step19:  When the activations are activating, only the weights and biases are having a say. Right?
Step20:  Right.
Now what about the Log Softmax operation itself? Well okay I can simulate this by hand
Step21:  umm....
...
So it's torch.tensor not torch.Tensor? Got a lot of errors trying to specify a datatype with capital T. Alright then.
Step22:  Good it works. Now to change things. The question was if any of the dropped values (non-target index) had any effect on the loss - since the loss was only calculated on error from the correct target. Basically
<ASSISTANT_TASK:>
Python Code:
from pathlib import Path
import requests
data_path = Path('data')
path = data_path/'mnist'
path.mkdir(parents=True, exist_ok=True)
url = 'http://deeplearning.net/data/mnist/'
filename = 'mnist.pkl.gz'
(path/filename)
if not (path/filename).exists():
    content = requests.get(url+filename).content
    (path/filename).open('wb').write(content)
import pickle, gzip
with gzip.open(path/filename, 'rb') as f:
    ((x_train, y_train), (x_valid, y_valid), _) = pickle.load(f, encoding='latin-1')
%matplotlib inline
from matplotlib import pyplot
import numpy as np
pyplot.imshow(x_train[0].reshape((28,28)), cmap="gray")
x_train.shape
import torch
x_train,y_train,x_valid,y_valid = map(torch.tensor, (x_train,y_train,x_valid,y_valid))
n,c = x_train.shape
x_train, x_train.shape, y_train.min(), y_train.max()
import math
weights = torch.rand(784, 10)/math.sqrt(784)
weights.requires_grad_()
bias = torch.zeros(10, requires_grad=True)
def log_softmax(x): return x - x.exp().sum(-1).log().unsqueeze(-1)
def model(xb): return log_softmax(xb @ weights + bias)
xb.shape, xb.sum(-1).shape
Explanation: 2018/9/15-16 WNixalo
https://github.com/fastai/fastai_v1/blob/master/dev_nb/001a_nn_basics.ipynb
End of explanation
bs = 64
xb = x_train[0:bs] # a mini-batch from x
preds = model(xb)
preds[0], preds.shape
def nll(input, target): return -input[range(target.shape[0]), target].mean()
loss_func = nll
yb = y_train[0:bs]
loss_func(preds, yb)
preds[0]
((x_train[0:bs]@weights+bias) - (x_train[0:bs]@weights+bias).exp().sum(-1).log().unsqueeze(-1))[0]
preds[0]
nll(preds, yb)
-preds[range(yb.shape[0]), yb].mean()
type(preds)
preds[range(0)]
preds[0]
preds[range(1)]
preds[range(2)]
preds[:2]
type(preds)
np.array([[range(10)]])[range(1)]
A = np.array([[range(10)]])
A.shape
A[range(2)]
A.shape
len(A[0])
A.shape[0]
A[0]
A[range(1)]
xb.sum()
xb.numpy().sum(-1)
xb.sum(-1)
Explanation: the torch.Tensor.sum(dim) call takes an integer argument as the axis along which to sum. This applies to NumPy arrays as well.
In this case xb.sum(-1) will turn a 64x784 tensor into a size 64 tensor. This creates a tensor with each element being the total sum of its corresponding size 784 (28x28 flattened) image from the minibatch.
End of explanation
xb.sum(-1)
xb[0].sum()
Explanation: torch.unsqueeze returns a tensor with a dimension of size 1 inserted at the specified position.
the returned tensor shares the smae underlying data with this tensor.
End of explanation
xb.exp().sum(-1).log()
xb.exp().sum(-1).log()[0]
Explanation: taking a look at what .unsqueeze does; what does the tensor look like right before unsqueeze is applied to it?
End of explanation
(xb.exp().sum(-1).log())[0]
xb.exp().sum(-1).log().unsqueeze(-1)[:10]
np.array([i for i in range(10)]).shape
torch.Tensor([i for i in range(10)]).shape
xb.exp().sum(-1).log().unsqueeze(-1).numpy().shape
Explanation: making sure I didn't need parentheses there
End of explanation
xb.exp().sum(-1).log()[:10]
Explanation: Okay so .unsqueeze turns the size 64 tensor into a 64x1 tensor, so it's nicely packaged up with the first element being the 64-long vector ... or something like that right?
End of explanation
preds.unsqueeze(-1).shape
Explanation: The unsqueezed tensor doesn't look as 'nice'.. I guess. So it's packaged into a single column vector because we'll need that for the linear algebra we'll do to it later yeah?
End of explanation
preds.unsqueeze(-1)[:2]
Explanation: Oh this is cool. I was wondering how .unsqeeze worked for tensors with multiple items in multiple dimensions (ie: not just a single row vector). Well this is what it does:
End of explanation
# logsoftmax(xb)
ls_xb = log_softmax(xb)
log_softmax(xb@weights+bias)[0]
(xb@weights).shape
xb.shape
(xb@weights).shape
Explanation: So .unsqueeze turns our size 64x10 ... ohhhhhhhh I misread:
torch.unsqueeze returns a tensor with a dimension of size 1 inserted at the specified position.
doesn't mean it repackages the original tensor into a 1-dimensional tensor. I was wonder how it knew how long to make it (you'd have to just concatenate everything, but then in what order?).
No, a size-1 dimension is inserted where you tell it. So if it's an (X,Y) matrix, you go and give it a Z dimension, but that Z only contains the original (X,Y), ie: the only thing added is a dimension.
Okay, interesting. Not exactly sure yet why we want 3 dimensions, but I kinda get it. Is it related to our data being 28x28x1? Wait isn't PyTorch's ordering N x [C x H x W] ? So it's unrelated then? Or useful for returning 64x784 to 64x28x28? I think that's not the case? Don't know.
So what's up with the input[range(.. thing?:
End of explanation
# for reference:
xb = x_train[0:bs]
yb = y_train[0:bs]
def log_softmax(x): return x - x.exp().sum(-1).log().unsqueeze(-1)
def model(xb): return log_softmax(xb @ weights + bias)
preds = model(xb)
def nll(input, target): return -input[range(target.shape[0]), target].mean()
loss = nll(preds, yb)
loss
Explanation: Oh this is where I was confused. I'm not throwing xb into Log Softmax. I'm throwing xb • w + bias. The shape going into the log softmax function is not 64x784, it's 64x10. Yeah that makes sense. well duh it has to. Each value in the tensor is an activation for a class, for each image in the minibatch. So by the magic of machine learning, each activation encapsulates the effect of the weights and biases on that input element with respect to that class.
So that means that the .unsqueeze oepration is not going to be giving a 64x784 vector.
End of explanation
xb, xb.shape
Explanation: Note the loss equals that in cell Out[25] above as it should.
Back to teasing this apart by hand.
The minibatch:
End of explanation
(xb @ weights + bias)[:2]
(xb @ weights + bias).shape
Explanation: The minibatch's activations as they head into the Log Softmax:
End of explanation
log_softmax(xb@weights+bias)[:2]
log_softmax(xb@weights+bias).shape
Explanation: The minibatch activations after the Log Softmax and before heading into Negative Log Likelihood:
End of explanation
nll(log_softmax(xb@weights+bias), yb)
Explanation: The loss value computed via NLL on the Log Softmax activations:
End of explanation
[range(yb.shape[0]), yb]
Explanation: Okay. Now questions. What is indexing input by [range(target.shape[0]), target] supposed to be doing? I established before that A[range(n)] is valid if n ≤ A.shape[0]. So what's going on is I'm range-indexing the 1st dimension of the LogSoftmax activations with the length of the target tensor, and the rest of the dimension indices being the ..target tensor itself?
That means the index is this:
End of explanation
xb[yb]
Explanation: Okay. What does it look like when I index a tensor – forget range-idx for now – with another tensor?
End of explanation
xb.shape, yb.shape
array_1 = np.array([[str(j)+str(i) for i in range(10)] for j in range(5)])
array_1
array_2 = np.array([i for i in range(len(array_1[0]))])
array_2
Explanation: Okay..
End of explanation
array_1[range(array_2.shape[0]), array_2]
Explanation: Uh, moment of truth:
End of explanation
# for reference (again):
xb = x_train[0:bs]
yb = y_train[0:bs]
def log_softmax(x): return x - x.exp().sum(-1).log().unsqueeze(-1)
def model(xb): return log_softmax(xb @ weights + bias)
preds = model(xb)
def nll(input, target): return -input[range(target.shape[0]), target].mean()
loss = nll(preds, yb)
Explanation: Oof course. What happened. Is it.. yes. I'm indexing the wrong array. Also no value in target is greater than the number of classes ... oh... oh ffs. Okay.
I range index by the length of target's first dim to get the entire first dim of the LogSoftmax activations, and each vector in that index is itself indexed by the value of the target.
Less-shitty English: take the first dimension of the activations; that should be batch_size x num_classes activations; so: num_classes values in each of batch_size vectors; Now for each of those vectors, pull out the value indexed by the corresponding index-value in the target tensor.
Oh I see. So just now I was confused that there was redundant work being done. yeah kinda. It's Linear Algebra. See, the weights and biases produce the entire output-activations tensor. Meaning: the dot-product & addition operation creates probabilities for every class for every image in the minibatch. Yeah that can be a lot; linalg exists in a block-like world & it's easy to get carried away (I think).
And that answers another question: the loss function here only cares about how wrong the correct class was. Looks like the incorrect classes are totally ignored (hence a bit of mental hesitation for me because it looks like 90% of the information is being thrown away (it is)). Now, that's not what's going on when the Log Softmax is being computed. Gotta think about that a moment..
could activations for non-target classes affect the target-activations during the Log Softmax step, before they're disgarded in the NLL?
xb - xb.exp().sum(-1).log().unsqueeze(-1)
is the magic line (xb is x in the definition).
End of explanation
xb.shape, weights.shape
np.array([[1,1,1],[2,2,2],[3,3,3]]) @ np.array([[1],[2],[3]])
np.array([[1,1,1],[2,2,2],[-11,0,3]]) @ np.array([[1],[2],[3]])
Explanation: When the activations are activating, only the weights and biases are having a say. Right?
End of explanation
yb.type()
# batch size of 3
xb_tmp = np.array([[1,1,1,1,1],[2,2,2,2,2],[3,3,3,3,3]])
yb_tmp = np.array([0,1,2])
# 4 classes
c = 4
w_tmp  = np.array([[i for i in range(c)] for j in range(xb_tmp.shape[1])])
xb_tmp = torch.Tensor(xb_tmp)
yb_tmp = torch.tensor(yb_tmp, dtype=torch.int64) # see: https://pytorch.org/docs/stable/tensors.html#torch-tensor
w_tmp  = torch.Tensor(w_tmp)
Explanation: Right.
Now what about the Log Softmax operation itself? Well okay I can simulate this by hand:
End of explanation
torch.tensor([[1, 2, 3]],dtype=torch.int32)
xb_tmp.shape, yb_tmp.shape, w_tmp.shape
xb.shape, yb.shape, weights.shape
actv_tmp = log_softmax(xb_tmp @ w_tmp)
actv_tmp
nll(actv_tmp, yb_tmp)
Explanation: umm....
...
So it's torch.tensor not torch.Tensor? Got a lot of errors trying to specify a datatype with capital T. Alright then.
End of explanation
# batch size of 3
xb_tmp = np.array([[0,1,1,0,0]])
yb_tmp = np.array([1])
# 4 classes
c = 4
w_tmp  = np.array([[i for i in range(c)] for j in range(xb_tmp.shape[1])])
xb_tmp = torch.Tensor(xb_tmp)
yb_tmp = torch.tensor(yb_tmp, dtype=torch.int64) # see: https://pytorch.org/docs/stable/tensors.html#torch-tensor
w_tmp  = torch.Tensor(w_tmp)
xb_tmp @ w_tmp
# LogSoftmax(activations)
actv_tmp = log_softmax(xb_tmp @ w_tmp)
actv_tmp
# NLL Loss
loss = nll(actv_tmp, yb_tmp)
loss
def cross_test(x, y):
    # batch size of 3
    xb_tmp = np.array(x)
    yb_tmp = np.array(y)
    # 4 classes
    c = 4
    w_tmp  = np.array([[i for i in range(c)] for j in range(xb_tmp.shape[1])])
    xb_tmp = torch.Tensor(xb_tmp)
    yb_tmp = torch.tensor(yb_tmp, dtype=torch.int64) # see: https://pytorch.org/docs/stable/tensors.html#torch-tensor
    w_tmp  = torch.Tensor(w_tmp)
    print(f'Activation:  {xb_tmp @ w_tmp}')
    
    # LogSoftmax(activations)
    actv_tmp = log_softmax(xb_tmp @ w_tmp)
    print(f'Log Softmax: {actv_tmp}')
    
    # NLL Loss
    loss = nll(actv_tmp, yb_tmp)
    print(f'NLL Loss:    {loss}')
w_tmp
cross_test([[1,1,1,1,1]], [1])
cross_test([[1,1,1,1,0]], [1])
cross_test([[1,1,1,0,0]], [1])
cross_test([[1,1,1,1,0]], [1])
cross_test([[1,1,0,0,0]], [1])
Explanation: Good it works. Now to change things. The question was if any of the dropped values (non-target index) had any effect on the loss - since the loss was only calculated on error from the correct target. Basically: is there any lateral flow of information?
So I'll check this by editing values in the softmax activation that are not of the correct index.
Wait that shouldn't have an effect anyway. No the question is if information earlier in the stream had an effect later on. It is 4:12 am.. 
Aha. My question was if the activations that created the non-target class probabilities had any effect on target classes. Which is asking if there is crossing of information in the ... oh.
I confused myself with the minibatches. Ignore those, there'd be something very wrong if there was cross-talk between them. I want to know if there is cross-talk within an individual tensor as it travels through the model.
End of explanation
<END_TASK>