<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Regresja liniowa
Zamiast
Step1:  Macierz $A$ dla regresji liniowej wynosi
Step2:  Współczynniki dokładnie będą wynosiły
Step3:  Optymalizacja metodą iteracyjną,
Nie zakładamy, że mamy problem regresji liniowej.
Używamy
Step4:  Tensor flow - gradient descend
<ASSISTANT_TASK:>
Python Code:
%matplotlib notebook
import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
import numpy as np 
import matplotlib.pyplot as plt
learning_rate = 0.01
training_epochs = 1000
display_step = 50
train_X = np.asarray([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167,
                         7.042,10.791,5.313,7.997,5.654,9.27,3.1])
train_Y = np.asarray([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221,
                         2.827,3.465,1.65,2.904,2.42,2.94,1.3])
n_samples = train_X.shape[0]
Explanation: Regresja liniowa
Zamiast:
<p>$$  A x =  b$$</p>
<p>rozwiązujemy</p>
<p>$$ A^T A x = A^T b.$$</p>
<p>Inaczej mówiąc, niech błąd:</p>
<p>$$  r=b-A x$$</p>
<p>leży w lewym jądrze operatora A:</p>
<p>$$A^T r =A^T( b-Ax) = A^T b-A^TAx = 0.$$</p>
<p> </p>
<p>$$ A^T A x = A^T b.$$</p>
<ol> </ol>
Można też zarządać znikania gradientu kwadratu odchylenia:
$$ \frac{\partial}{\partial x_k} (A_{ij} x_j - b_i) (A_{il} x_l - b_i) = 0$$ 
$$  A_{ij} \delta_{jk} (A_{il} x_l - b_i) + A_{il} \delta_{lk} (A_{ij} x_j - b_i) =0 $$
$$ A^TAx - A^T b + A^TAx - A^T b =0 $$
$$ A^TAx - A^T b  =0 $$
End of explanation
import numpy as np 
M = np.vstack([np.ones_like(train_X),train_X]).T
M
print (np.dot(M.T,M))
print(np.dot(M.T,train_Y))
Explanation: Macierz $A$ dla regresji liniowej wynosi:
End of explanation
c = np.linalg.solve(np.dot(M.T,M),np.dot(M.T,train_Y))
c
plt.plot(train_X, train_Y, 'ro', label='Original data')
plt.plot(train_X, c[1] * train_X + c[0], label='Fitted line')
plt.legend()
plt.close()
Explanation: Współczynniki dokładnie będą wynosiły:
End of explanation
from scipy.optimize import minimize
def cost(c,x=train_X,y=train_Y):
    return sum( (c[0]+x_*c[1]-y_)**2  for (x_,y_) in zip(x,y) )
cost([1,2])
res = minimize(cost, [1,1], method='nelder-mead', options={'xtol': 1e-8, 'disp': True})
res.x
x = np.linspace(-2,2,77)
y = np.linspace(-2,2,77)
X,Y = np.meshgrid(x,y)
cost([X,Y]).shape
plt.contourf( X,Y,np.log(cost([X,Y])),cmap='gray')
plt.plot(res.x[0],res.x[1],'o')
np.min(cost([X,Y]))
px=[]
py=[]
for i in range(20):
    res = minimize(cost, [1,1], options={ 'maxiter':i})
    px.append(res.x[0])
    py.append(res.x[1])
    print(res.x)
plt.plot(px,py,'ro-')
import sympy
from sympy.abc import x,y
sympy.init_printing(use_latex='mathjax')
f_symb = cost([x,y]).expand()
f_symb.diff(x)
F = sympy.lambdify((x,y),f_symb,np)
Fx = sympy.lambdify((x,y),f_symb.diff(x),np)
Fy = sympy.lambdify((x,y),f_symb.diff(y),np)
F(1,1),cost([1,1])
x0,y0 = -1,1
h = 0.01/(2*17) 
for i in range(500):
    plt.plot(x0,y0,'go')
    #print(i,x0,y0)
    x0 += -h * Fx(x0,y0)
    y0 += -h * Fy(x0,y0)
Explanation: Optymalizacja metodą iteracyjną,
Nie zakładamy, że mamy problem regresji liniowej.
Używamy: https://docs.scipy.org/doc/scipy-0.18.1/reference/tutorial/optimize.html
End of explanation
# tf Graph Input
X = tf.placeholder("float")
Y = tf.placeholder("float")
# Set model weights
W = tf.Variable(1.0, name="weight")
b = tf.Variable(1.0, name="bias")
# Construct a linear model
pred = tf.add(tf.multiply(X, W), b)
# Mean squared error
cost = tf.reduce_sum(tf.pow(pred-Y, 2))/(2*n_samples)
# Gradient descent
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
# Initializing the variables
init = tf.global_variables_initializer()
# TEST
with tf.Session() as sess:
    sess.run(init)
    sess.run(tf.assign(W,1.0))
    sess.run(tf.assign(b,2.0))
    print(sess.run(b),sess.run(cost, feed_dict={X: train_X, Y: train_Y}))
# Launch the graph
x_tf_lst = [] 
y_tf_lst = [] 
with tf.Session() as sess:
    sess.run(init)
    # Fit all training data
    for epoch in range(training_epochs):
        for (x, y) in zip(train_X, train_Y):
            sess.run(optimizer, feed_dict={X: x, Y: y})
        #Display logs per epoch step
        if (epoch+1) % display_step == 0:
            c = sess.run(cost, feed_dict={X: train_X, Y:train_Y})
            print ("Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(c), \
                "W=", sess.run(W), "b=", sess.run(b))
        x_tf_lst.append(sess.run(b))
        y_tf_lst.append(sess.run(W))
        
    training_cost = sess.run(cost, feed_dict={X: train_X, Y: train_Y})
    print ("Training cost=", training_cost, "W=", sess.run(W), "b=", sess.run(b), '\n')
    
    
plt.plot(x_tf_lst,y_tf_lst,'yo')
Explanation: Tensor flow - gradient descend
End of explanation
<END_TASK>