

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>hep_ml.nnet &mdash; hep_ml 0.7.0 documentation</title>
  

  
  <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />

  
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
        <script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/jquery.js"></script>
        <script src="../../_static/underscore.js"></script>
        <script src="../../_static/doctools.js"></script>
    
    <script type="text/javascript" src="../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../genindex.html" />
    <link rel="search" title="Search" href="../../search.html" /> 
</head>

<body class="wy-body-for-nav">

   
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search" >
          

          
            <a href="../../index.html" class="icon icon-home"> hep_ml
          

          
          </a>

          
            
            
              <div class="version">
                0.7.0
              </div>
            
          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul>
<li class="toctree-l1"><a class="reference internal" href="../../index.html">hep_ml documentation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../gb.html">Gradient boosting</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../losses.html">Losses for Gradient Boosting</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../uboost.html">uBoost</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../metrics.html">Metric functions</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../nnet.html">Neural networks</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../preprocessing.html">Preprocessing data</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../reweight.html">Reweighting algorithms</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../speedup.html">Fast predictions</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../splot.html">sPlot</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../notebooks.html">Code Examples</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../index.html">hep_ml</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          

















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../../index.html" class="icon icon-home"></a> &raquo;</li>
        
          <li><a href="../index.html">Module code</a> &raquo;</li>
        
      <li>hep_ml.nnet</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
      </li>
    
  </ul>

  
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
  <h1>Source code for hep_ml.nnet</h1><div class="highlight"><pre>
<span></span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd">**hep_ml.nnet** is minimalistic **theano**-powered version of feed-forward neural networks.</span>
<span class="sd">The neural networks from this library provide sklearn classifier&#39;s interface.</span>

<span class="sd">Definitions for loss functions, trainers of neural networks are defined in this file too.</span>
<span class="sd">Main point of this library: black-box stochastic optimization of any given loss function.</span>
<span class="sd">This gives ability to define any activation expression (at the cost of unavailability of pretraining).</span>

<span class="sd">In this file we have **examples** of neural networks,</span>
<span class="sd">user is encouraged to write his own specific architecture,</span>
<span class="sd">which can be much more complex than those used usually.</span>

<span class="sd">If you don&#39;t want to dive into details, use</span>
<span class="sd">:class:`hep_ml.nnet.MLPClassifier`, :class:`hep_ml.nnet.MLPRegressor`</span>

<span class="sd">This library should be preferred for different experiments with architectures.</span>
<span class="sd">Also **hep_ml.nnet** allows optimization of parameters in any differentiable decision function.</span>

<span class="sd">Being written in theano, these neural networks are able to make use of your GPU.</span>

<span class="sd">See also libraries: keras, mxnet, pytorch.</span>

<span class="sd">Examples</span>
<span class="sd">________</span>

<span class="sd">Training a neural network with two hidden layers using IRPROP- algorithm</span>

<span class="sd">&gt;&gt;&gt; network = MLPClassifier(layers=[7, 7], loss=&#39;log_loss&#39;, trainer=&#39;irprop-&#39;, epochs=1000)</span>
<span class="sd">&gt;&gt;&gt; network.fit(X, y)</span>
<span class="sd">&gt;&gt;&gt; probability = network.predict_proba(X)</span>

<span class="sd">Training an AdaBoost over neural network with adadelta trainer. Trainer specific parameter was used</span>
<span class="sd">(size of minibatch)</span>

<span class="sd">&gt;&gt;&gt; from sklearn.ensemble import AdaBoostClassifier</span>
<span class="sd">&gt;&gt;&gt; base_network = MLPClassifier(layers=[10], trainer=&#39;adadelta&#39;, trainer_parameters={&#39;batch&#39;: 600})</span>
<span class="sd">&gt;&gt;&gt; classifier = AdaBoostClassifier(base_estimator=base_network, n_estimators=20)</span>
<span class="sd">&gt;&gt;&gt; classifier.fit(X, y)</span>

<span class="sd">Using custom pretransformer and ExponentialLoss:</span>

<span class="sd">&gt;&gt;&gt; from sklearn.preprocessing import PolynomialFeatures</span>
<span class="sd">&gt;&gt;&gt; network = MLPClassifier(layers=[10], scaler=PolynomialFeatures(), loss=&#39;exp_loss&#39;)</span>

<span class="sd">To create custom neural network, see code of SimpleNeuralNetwork, which is good place to start.</span>

<span class="sd">Interface</span>
<span class="sd">_________</span>

<span class="sd">Below an interface for classifier and regressor is demonstrated.</span>

<span class="sd">.. autoclass :: AbstractNeuralNetworkClassifier</span>
<span class="sd">    :members: fit, predict, predict_proba</span>

<span class="sd">.. autoclass :: AbstractNeuralNetworkRegressor</span>
<span class="sd">    :members: fit, predict</span>


<span class="sd">Recommended networks</span>
<span class="sd">____________________</span>

<span class="sd">Multilayer Perceptron is well-known popular algorithm working well in most applications.</span>

<span class="sd">.. autoclass :: MLPClassifier</span>

<span class="sd">.. autoclass :: MLPRegressor</span>

<span class="sd">.. autoclass :: MLPMultiClassifier</span>

<span class="sd">Custom networks</span>
<span class="sd">_______________</span>

<span class="sd">Below some examples of custom networks are given.</span>
<span class="sd">Those are initial point for constructing own architectures.</span>

<span class="sd">.. autoclass :: SimpleNeuralNetwork</span>

<span class="sd">.. autoclass :: SoftmaxNeuralNetwork</span>

<span class="sd">.. autoclass :: RBFNeuralNetwork</span>

<span class="sd">.. autoclass :: PairwiseNeuralNetwork</span>

<span class="sd">.. autoclass :: PairwiseSoftplusNeuralNetwork</span>


<span class="sd">Loss functions</span>
<span class="sd">______________</span>

<span class="sd">The following loss functions are available for **classification**:</span>


<span class="sd">.. autofunction:: hep_ml.nnet.log_loss</span>

<span class="sd">.. autofunction:: hep_ml.nnet.exp_loss</span>

<span class="sd">.. autofunction:: hep_ml.nnet.exp_log_loss</span>

<span class="sd">.. autofunction:: hep_ml.nnet.squared_loss</span>

<span class="sd">The following loss functions are available for **regression**:</span>

<span class="sd">.. autofunction:: hep_ml.nnet.mse_loss</span>

<span class="sd">.. autofunction:: hep_ml.nnet.smooth_huber_loss</span>


<span class="sd">Trainers</span>
<span class="sd">________</span>

<span class="sd">The trainers are optimization algorithms used to minimize target loss function in neural networks.</span>
<span class="sd">The trainers are implemented as functions with some standard parameters and some optional.</span>


<span class="sd">.. autofunction:: hep_ml.nnet.sgd_trainer</span>

<span class="sd">.. autofunction:: hep_ml.nnet.irprop_minus_trainer</span>

<span class="sd">.. autofunction:: hep_ml.nnet.irprop_plus_trainer</span>

<span class="sd">.. autofunction:: hep_ml.nnet.adadelta_trainer</span>


<span class="sd">&quot;&quot;&quot;</span>
<span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">print_function</span><span class="p">,</span> <span class="n">division</span><span class="p">,</span> <span class="n">absolute_import</span>
<span class="kn">from</span> <span class="nn">copy</span> <span class="kn">import</span> <span class="n">deepcopy</span>

<span class="kn">import</span> <span class="nn">numpy</span>
<span class="kn">import</span> <span class="nn">theano</span>
<span class="kn">import</span> <span class="nn">theano.tensor</span> <span class="k">as</span> <span class="nn">T</span>
<span class="kn">from</span> <span class="nn">theano.ifelse</span> <span class="kn">import</span> <span class="n">ifelse</span>
<span class="kn">from</span> <span class="nn">theano.tensor.shared_randomstreams</span> <span class="kn">import</span> <span class="n">RandomStreams</span>

<span class="kn">from</span> <span class="nn">sklearn.utils.validation</span> <span class="kn">import</span> <span class="n">check_random_state</span>
<span class="kn">from</span> <span class="nn">sklearn.base</span> <span class="kn">import</span> <span class="n">BaseEstimator</span><span class="p">,</span> <span class="n">ClassifierMixin</span><span class="p">,</span> <span class="n">RegressorMixin</span><span class="p">,</span> <span class="n">TransformerMixin</span><span class="p">,</span> <span class="n">clone</span>
<span class="kn">from</span> <span class="nn">sklearn</span> <span class="kn">import</span> <span class="n">preprocessing</span>
<span class="kn">from</span> <span class="nn">.commonutils</span> <span class="kn">import</span> <span class="n">check_xyw</span><span class="p">,</span> <span class="n">check_sample_weight</span><span class="p">,</span> <span class="n">score_to_proba</span>
<span class="kn">from</span> <span class="nn">.preprocessing</span> <span class="kn">import</span> <span class="n">IronTransformer</span>

<span class="n">floatX</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">floatX</span>
<span class="n">DEFAULT_MINIBATCH</span> <span class="o">=</span> <span class="mi">30</span>

<span class="n">__author__</span> <span class="o">=</span> <span class="s1">&#39;Alex Rogozhnikov&#39;</span>
<span class="n">__all__</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;MLPClassifier&#39;</span><span class="p">,</span>
           <span class="s1">&#39;MLPRegressor&#39;</span><span class="p">,</span>
           <span class="s1">&#39;SimpleNeuralNetwork&#39;</span><span class="p">,</span>
           <span class="s1">&#39;SoftmaxNeuralNetwork&#39;</span><span class="p">,</span>
           <span class="s1">&#39;RBFNeuralNetwork&#39;</span><span class="p">,</span>
           <span class="s1">&#39;PairwiseNeuralNetwork&#39;</span><span class="p">,</span>
           <span class="s1">&#39;PairwiseSoftplusNeuralNetwork&#39;</span><span class="p">,</span>
           <span class="p">]</span>


<span class="c1"># region Loss functions</span>


<div class="viewcode-block" id="squared_loss"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.squared_loss">[docs]</a><span class="k">def</span> <span class="nf">squared_loss</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">pred</span><span class="p">,</span> <span class="n">w</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot; Squared loss for classification, not to be messed up with MSE&quot;&quot;&quot;</span>
    <span class="k">return</span> <span class="n">T</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">w</span> <span class="o">*</span> <span class="p">(</span><span class="n">y</span> <span class="o">-</span> <span class="n">T</span><span class="o">.</span><span class="n">nnet</span><span class="o">.</span><span class="n">sigmoid</span><span class="p">(</span><span class="n">pred</span><span class="p">))</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span></div>


<div class="viewcode-block" id="log_loss"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.log_loss">[docs]</a><span class="k">def</span> <span class="nf">log_loss</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">pred</span><span class="p">,</span> <span class="n">w</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot; Logistic loss for classification (aka cross-entropy, aka binomial deviance) &quot;&quot;&quot;</span>
    <span class="n">margin</span> <span class="o">=</span> <span class="n">pred</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="mi">2</span> <span class="o">*</span> <span class="n">y</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">T</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">w</span> <span class="o">*</span> <span class="n">T</span><span class="o">.</span><span class="n">nnet</span><span class="o">.</span><span class="n">softplus</span><span class="p">(</span><span class="n">margin</span><span class="p">))</span></div>


<div class="viewcode-block" id="exp_loss"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.exp_loss">[docs]</a><span class="k">def</span> <span class="nf">exp_loss</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">pred</span><span class="p">,</span> <span class="n">w</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot; Exponential loss for classification (aka AdaLoss function) &quot;&quot;&quot;</span>
    <span class="n">margin</span> <span class="o">=</span> <span class="n">pred</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="mi">2</span> <span class="o">*</span> <span class="n">y</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">T</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">w</span> <span class="o">*</span> <span class="n">T</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="n">margin</span><span class="p">))</span></div>


<div class="viewcode-block" id="exp_log_loss"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.exp_log_loss">[docs]</a><span class="k">def</span> <span class="nf">exp_log_loss</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">pred</span><span class="p">,</span> <span class="n">w</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot; Classification loss function,</span>
<span class="sd">    combines logistic loss for signal and exponential loss for background &quot;&quot;&quot;</span>
    <span class="k">return</span> <span class="mi">2</span> <span class="o">*</span> <span class="n">log_loss</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">pred</span><span class="p">,</span> <span class="n">w</span><span class="o">=</span><span class="n">w</span> <span class="o">*</span> <span class="n">y</span><span class="p">)</span> <span class="o">+</span> <span class="n">exp_loss</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">pred</span><span class="p">,</span> <span class="n">w</span><span class="o">=</span><span class="n">w</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">y</span><span class="p">))</span></div>


<div class="viewcode-block" id="mse_loss"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.mse_loss">[docs]</a><span class="k">def</span> <span class="nf">mse_loss</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">pred</span><span class="p">,</span> <span class="n">w</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot; Regression loss function, mean squared error. &quot;&quot;&quot;</span>
    <span class="k">return</span> <span class="n">T</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">w</span> <span class="o">*</span> <span class="p">(</span><span class="n">y</span> <span class="o">-</span> <span class="n">pred</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span></div>


<div class="viewcode-block" id="smooth_huber_loss"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.smooth_huber_loss">[docs]</a><span class="k">def</span> <span class="nf">smooth_huber_loss</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">pred</span><span class="p">,</span> <span class="n">w</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot; Regression loss function, smooth version of Huber loss function. &quot;&quot;&quot;</span>
    <span class="n">difference</span> <span class="o">=</span> <span class="nb">abs</span><span class="p">(</span><span class="n">y</span> <span class="o">-</span> <span class="n">pred</span><span class="p">)</span>
    <span class="c1"># error = logaddexp(-difference, difference), but this is not implemented in theano.</span>
    <span class="n">error</span> <span class="o">=</span> <span class="n">difference</span> <span class="o">+</span> <span class="n">T</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="mi">1</span> <span class="o">+</span> <span class="n">T</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="mi">2</span> <span class="o">*</span> <span class="n">difference</span><span class="p">))</span>
    <span class="k">return</span> <span class="n">T</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">w</span> <span class="o">*</span> <span class="n">error</span><span class="p">)</span></div>


<span class="n">classification_losses</span> <span class="o">=</span> <span class="p">{</span>
    <span class="s1">&#39;exp_loss&#39;</span><span class="p">:</span> <span class="n">exp_loss</span><span class="p">,</span>
    <span class="s1">&#39;log_loss&#39;</span><span class="p">:</span> <span class="n">log_loss</span><span class="p">,</span>
    <span class="s1">&#39;exp_log_loss&#39;</span><span class="p">:</span> <span class="n">exp_log_loss</span><span class="p">,</span>
    <span class="s1">&#39;squared_loss&#39;</span><span class="p">:</span> <span class="n">squared_loss</span><span class="p">,</span>
<span class="p">}</span>

<span class="n">regression_losses</span> <span class="o">=</span> <span class="p">{</span>
    <span class="s1">&#39;mse_loss&#39;</span><span class="p">:</span> <span class="n">mse_loss</span><span class="p">,</span>
    <span class="s1">&#39;smooth_huber_loss&#39;</span><span class="p">:</span> <span class="n">smooth_huber_loss</span><span class="p">,</span>
<span class="p">}</span>

<span class="n">losses</span> <span class="o">=</span> <span class="n">classification_losses</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
<span class="n">losses</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">regression_losses</span><span class="p">)</span>


<span class="c1"># endregion</span>


<span class="c1"># region Trainers</span>
<span class="k">def</span> <span class="nf">get_batch</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">random_stream</span><span class="p">,</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">DEFAULT_MINIBATCH</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot; Generates subset of training dataset, of size batch&quot;&quot;&quot;</span>
    <span class="n">indices</span> <span class="o">=</span> <span class="n">random_stream</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="n">a</span><span class="o">=</span><span class="n">T</span><span class="o">.</span><span class="n">shape</span><span class="p">(</span><span class="n">x</span><span class="p">)[</span><span class="mi">0</span><span class="p">],</span> <span class="n">size</span><span class="o">=</span><span class="p">(</span><span class="n">batch_size</span><span class="p">,))</span>
    <span class="k">return</span> <span class="n">x</span><span class="p">[</span><span class="n">indices</span><span class="p">],</span> <span class="n">y</span><span class="p">[</span><span class="n">indices</span><span class="p">],</span> <span class="n">w</span><span class="p">[</span><span class="n">indices</span><span class="p">]</span>


<div class="viewcode-block" id="sgd_trainer"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.sgd_trainer">[docs]</a><span class="k">def</span> <span class="nf">sgd_trainer</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">parameters</span><span class="p">,</span> <span class="n">loss</span><span class="p">,</span> <span class="n">random_stream</span><span class="p">,</span> <span class="n">batch</span><span class="o">=</span><span class="n">DEFAULT_MINIBATCH</span><span class="p">,</span>
                <span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.1</span><span class="p">,</span> <span class="n">l2_penalty</span><span class="o">=</span><span class="mf">0.001</span><span class="p">,</span> <span class="n">momentum</span><span class="o">=</span><span class="mf">0.9</span><span class="p">,</span> <span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;Stochastic gradient descent with momentum, trivial but very popular.</span>

<span class="sd">    :param int batch: size of minibatch, each time averaging gradient over minibatch.</span>
<span class="sd">    :param float learning_rate: size of step</span>
<span class="sd">    :param float l2_penalty: speed of weights&#39; decay, l2 regularization prevents overfitting</span>
<span class="sd">    :param float momentum: momentum to stabilize learning process.</span>
<span class="sd">    &quot;&quot;&quot;</span>
    <span class="n">updates</span> <span class="o">=</span> <span class="p">[]</span>
    <span class="n">shareds</span> <span class="o">=</span> <span class="p">[]</span>
    <span class="n">xp</span><span class="p">,</span> <span class="n">yp</span><span class="p">,</span> <span class="n">wp</span> <span class="o">=</span> <span class="n">get_batch</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">batch</span><span class="p">,</span> <span class="n">random_stream</span><span class="o">=</span><span class="n">random_stream</span><span class="p">)</span>
    <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">param</span> <span class="ow">in</span> <span class="n">parameters</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
        <span class="n">der</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">grad</span><span class="p">(</span><span class="n">loss</span><span class="p">(</span><span class="n">xp</span><span class="p">,</span> <span class="n">yp</span><span class="p">,</span> <span class="n">wp</span><span class="p">),</span> <span class="n">param</span><span class="p">)</span>
        <span class="n">momentum_</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span><span class="n">param</span><span class="o">.</span><span class="n">get_value</span><span class="p">()</span> <span class="o">*</span> <span class="mf">0.</span><span class="p">)</span>
        <span class="n">shareds</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">momentum_</span><span class="p">)</span>
        <span class="n">updates</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">momentum_</span><span class="p">,</span> <span class="n">momentum_</span> <span class="o">*</span> <span class="n">momentum</span> <span class="o">+</span> <span class="p">(</span><span class="mf">1.</span> <span class="o">-</span> <span class="n">momentum</span><span class="p">)</span> <span class="o">*</span> <span class="n">der</span><span class="p">])</span>
        <span class="n">updates</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">param</span><span class="p">,</span> <span class="n">param</span> <span class="o">*</span> <span class="p">(</span><span class="mf">1.</span> <span class="o">-</span> <span class="n">learning_rate</span> <span class="o">*</span> <span class="n">l2_penalty</span><span class="p">)</span> <span class="o">-</span> <span class="n">learning_rate</span> <span class="o">*</span> <span class="n">momentum_</span><span class="p">])</span>
    <span class="k">return</span> <span class="n">shareds</span><span class="p">,</span> <span class="n">updates</span></div>


<div class="viewcode-block" id="irprop_minus_trainer"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.irprop_minus_trainer">[docs]</a><span class="k">def</span> <span class="nf">irprop_minus_trainer</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">parameters</span><span class="p">,</span> <span class="n">loss</span><span class="p">,</span> <span class="n">random_stream</span><span class="p">,</span>
                         <span class="n">positive_step</span><span class="o">=</span><span class="mf">1.2</span><span class="p">,</span> <span class="n">negative_step</span><span class="o">=</span><span class="mf">0.5</span><span class="p">,</span> <span class="n">max_step</span><span class="o">=</span><span class="mf">1.</span><span class="p">,</span> <span class="n">min_step</span><span class="o">=</span><span class="mf">1e-6</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;IRPROP- is batch trainer, for details see http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.21.3428 .</span>
<span class="sd">    This is default trainer, very stable for classification.</span>

<span class="sd">    :param positive_step: factor, by which the step is increased when continuing going in the direction</span>
<span class="sd">    :param negative_step: factor, by which the step is increased when changing direction to opposite</span>
<span class="sd">    :param min_step: minimal change of weight during iteration</span>
<span class="sd">    :param max_step: maximal change of weight during iteration</span>
<span class="sd">    &quot;&quot;&quot;</span>
    <span class="n">shareds</span> <span class="o">=</span> <span class="p">[]</span>
    <span class="n">updates</span> <span class="o">=</span> <span class="p">[]</span>
    <span class="n">loss_value</span> <span class="o">=</span> <span class="n">loss</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">)</span>
    <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">param</span> <span class="ow">in</span> <span class="n">parameters</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
        <span class="n">old_derivative</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span><span class="n">param</span><span class="o">.</span><span class="n">get_value</span><span class="p">()</span> <span class="o">*</span> <span class="mf">0.</span><span class="p">)</span>
        <span class="n">delta</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span><span class="n">param</span><span class="o">.</span><span class="n">get_value</span><span class="p">()</span> <span class="o">*</span> <span class="mf">0.</span> <span class="o">+</span> <span class="mf">1e-3</span><span class="p">)</span>
        <span class="n">shareds</span><span class="o">.</span><span class="n">extend</span><span class="p">([</span><span class="n">old_derivative</span><span class="p">,</span> <span class="n">delta</span><span class="p">])</span>
        <span class="n">new_derivative</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">grad</span><span class="p">(</span><span class="n">loss_value</span><span class="p">,</span> <span class="n">param</span><span class="p">)</span>

        <span class="n">new_delta</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">where</span><span class="p">(</span><span class="n">new_derivative</span> <span class="o">*</span> <span class="n">old_derivative</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">,</span> <span class="n">delta</span> <span class="o">*</span> <span class="n">positive_step</span><span class="p">,</span> <span class="n">delta</span> <span class="o">*</span> <span class="n">negative_step</span><span class="p">)</span>
        <span class="n">new_delta</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">clip</span><span class="p">(</span><span class="n">new_delta</span><span class="p">,</span> <span class="n">min_step</span><span class="p">,</span> <span class="n">max_step</span><span class="p">)</span>

        <span class="n">updates</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">param</span><span class="p">,</span> <span class="n">param</span> <span class="o">-</span> <span class="n">new_delta</span> <span class="o">*</span> <span class="n">T</span><span class="o">.</span><span class="n">sgn</span><span class="p">(</span><span class="n">new_derivative</span><span class="p">)])</span>
        <span class="n">updates</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">delta</span><span class="p">,</span> <span class="n">new_delta</span><span class="p">])</span>

        <span class="n">new_old_derivative</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">where</span><span class="p">(</span><span class="n">new_derivative</span> <span class="o">*</span> <span class="n">old_derivative</span> <span class="o">&lt;</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">new_derivative</span><span class="p">)</span>
        <span class="n">updates</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">old_derivative</span><span class="p">,</span> <span class="n">new_old_derivative</span><span class="p">])</span>
    <span class="k">return</span> <span class="n">shareds</span><span class="p">,</span> <span class="n">updates</span></div>


<div class="viewcode-block" id="irprop_plus_trainer"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.irprop_plus_trainer">[docs]</a><span class="k">def</span> <span class="nf">irprop_plus_trainer</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">parameters</span><span class="p">,</span> <span class="n">loss</span><span class="p">,</span> <span class="n">random_stream</span><span class="p">,</span>
                        <span class="n">positive_step</span><span class="o">=</span><span class="mf">1.2</span><span class="p">,</span> <span class="n">negative_step</span><span class="o">=</span><span class="mf">0.5</span><span class="p">,</span> <span class="n">max_step</span><span class="o">=</span><span class="mf">1.</span><span class="p">,</span> <span class="n">min_step</span><span class="o">=</span><span class="mf">1e-6</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;IRPROP+ is batch trainer, for details see http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.21.3428</span>

<span class="sd">    :param positive_step: factor, by which the step is increased when continuing going in the direction</span>
<span class="sd">    :param negative_step: factor, by which the step is increased when changing direction to opposite</span>
<span class="sd">    :param min_step: minimal change of weight during iteration</span>
<span class="sd">    :param max_step: maximal change of weight during iteration</span>
<span class="sd">    &quot;&quot;&quot;</span>
    <span class="n">loss_value</span> <span class="o">=</span> <span class="n">loss</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">)</span>
    <span class="n">prev_loss_value</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span><span class="mf">1e10</span><span class="p">)</span>
    <span class="n">shareds</span> <span class="o">=</span> <span class="p">[</span><span class="n">prev_loss_value</span><span class="p">]</span>
    <span class="n">updates</span> <span class="o">=</span> <span class="p">[]</span>
    <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">param</span> <span class="ow">in</span> <span class="n">parameters</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
        <span class="n">old_derivative</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span><span class="n">param</span><span class="o">.</span><span class="n">get_value</span><span class="p">()</span> <span class="o">*</span> <span class="mf">0.</span><span class="p">)</span>
        <span class="n">delta</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span><span class="n">param</span><span class="o">.</span><span class="n">get_value</span><span class="p">()</span> <span class="o">*</span> <span class="mf">0.</span> <span class="o">+</span> <span class="mf">1e-3</span><span class="p">)</span>
        <span class="n">new_derivative</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">grad</span><span class="p">(</span><span class="n">loss_value</span><span class="p">,</span> <span class="n">param</span><span class="p">)</span>

        <span class="n">shift_if_bad_step</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">where</span><span class="p">(</span><span class="n">new_derivative</span> <span class="o">*</span> <span class="n">old_derivative</span> <span class="o">&lt;</span> <span class="mi">0</span><span class="p">,</span> <span class="n">delta</span> <span class="o">*</span> <span class="n">T</span><span class="o">.</span><span class="n">sgn</span><span class="p">(</span><span class="n">old_derivative</span><span class="p">),</span> <span class="mi">0</span><span class="p">)</span>
        <span class="n">shift</span> <span class="o">=</span> <span class="n">ifelse</span><span class="p">(</span><span class="n">loss_value</span> <span class="o">&gt;</span> <span class="n">prev_loss_value</span><span class="p">,</span> <span class="n">shift_if_bad_step</span><span class="p">,</span> <span class="mf">0.</span> <span class="o">*</span> <span class="n">param</span><span class="p">)</span>
        <span class="c1"># unfortunately we can&#39;t do it this way: param += shift</span>

        <span class="n">new_delta</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">where</span><span class="p">(</span><span class="n">new_derivative</span> <span class="o">*</span> <span class="n">old_derivative</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">,</span> <span class="n">delta</span> <span class="o">*</span> <span class="n">positive_step</span><span class="p">,</span> <span class="n">delta</span> <span class="o">*</span> <span class="n">negative_step</span><span class="p">)</span>
        <span class="n">new_delta</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">clip</span><span class="p">(</span><span class="n">new_delta</span><span class="p">,</span> <span class="n">min_step</span><span class="p">,</span> <span class="n">max_step</span><span class="p">)</span>

        <span class="n">updates</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">param</span><span class="p">,</span> <span class="n">param</span> <span class="o">+</span> <span class="n">shift</span> <span class="o">-</span> <span class="n">new_delta</span> <span class="o">*</span> <span class="n">T</span><span class="o">.</span><span class="n">sgn</span><span class="p">(</span><span class="n">new_derivative</span><span class="p">)])</span>
        <span class="n">updates</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">delta</span><span class="p">,</span> <span class="n">new_delta</span><span class="p">])</span>

        <span class="n">new_old_derivative</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">where</span><span class="p">(</span><span class="n">new_derivative</span> <span class="o">*</span> <span class="n">old_derivative</span> <span class="o">&lt;</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">new_derivative</span><span class="p">)</span>
        <span class="n">updates</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">old_derivative</span><span class="p">,</span> <span class="n">new_old_derivative</span><span class="p">])</span>
        <span class="n">shareds</span><span class="o">.</span><span class="n">extend</span><span class="p">([</span><span class="n">old_derivative</span><span class="p">,</span> <span class="n">delta</span><span class="p">])</span>

    <span class="n">updates</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">prev_loss_value</span><span class="p">,</span> <span class="n">loss_value</span><span class="p">])</span>
    <span class="k">return</span> <span class="n">shareds</span><span class="p">,</span> <span class="n">updates</span></div>


<div class="viewcode-block" id="adadelta_trainer"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.adadelta_trainer">[docs]</a><span class="k">def</span> <span class="nf">adadelta_trainer</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">parameters</span><span class="p">,</span> <span class="n">loss</span><span class="p">,</span> <span class="n">random_stream</span><span class="p">,</span> <span class="n">batch</span><span class="o">=</span><span class="n">DEFAULT_MINIBATCH</span><span class="p">,</span>
                     <span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.1</span><span class="p">,</span> <span class="n">half_life</span><span class="o">=</span><span class="mi">1000</span><span class="p">,</span> <span class="n">epsilon</span><span class="o">=</span><span class="mf">1e-4</span><span class="p">,</span> <span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;AdaDelta is trainer with adaptive learning rate.</span>

<span class="sd">    :param half_life: momentum-like parameter. The estimated parameters are decreased by 2 after so many events.</span>
<span class="sd">        It is recommended for small datasets to put halflife = number of samples in dataset.</span>
<span class="sd">    :param learning_rate: size of step</span>
<span class="sd">    :param batch: size of minibatch</span>
<span class="sd">    :param epsilon: regularization</span>
<span class="sd">    &quot;&quot;&quot;</span>
    <span class="n">shareds</span> <span class="o">=</span> <span class="p">[]</span>
    <span class="n">updates</span> <span class="o">=</span> <span class="p">[]</span>
    <span class="n">decay_rate</span> <span class="o">=</span> <span class="nb">pow</span><span class="p">(</span><span class="mf">0.5</span><span class="p">,</span> <span class="n">batch</span> <span class="o">/</span> <span class="nb">float</span><span class="p">(</span><span class="n">half_life</span><span class="p">))</span>

    <span class="n">xp</span><span class="p">,</span> <span class="n">yp</span><span class="p">,</span> <span class="n">wp</span> <span class="o">=</span> <span class="n">get_batch</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">batch</span><span class="p">,</span> <span class="n">random_stream</span><span class="o">=</span><span class="n">random_stream</span><span class="p">)</span>
    <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">param</span> <span class="ow">in</span> <span class="n">parameters</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
        <span class="n">derivative</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">grad</span><span class="p">(</span><span class="n">loss</span><span class="p">(</span><span class="n">xp</span><span class="p">,</span> <span class="n">yp</span><span class="p">,</span> <span class="n">wp</span><span class="p">),</span> <span class="n">param</span><span class="p">)</span>
        <span class="n">cumulative_derivative</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span><span class="n">param</span><span class="o">.</span><span class="n">get_value</span><span class="p">()</span> <span class="o">*</span> <span class="mf">0.</span><span class="p">)</span>
        <span class="n">cumulative_step</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span><span class="n">param</span><span class="o">.</span><span class="n">get_value</span><span class="p">()</span> <span class="o">*</span> <span class="mf">0.</span><span class="p">)</span>
        <span class="n">shareds</span><span class="o">.</span><span class="n">extend</span><span class="p">([</span><span class="n">cumulative_derivative</span><span class="p">,</span> <span class="n">cumulative_step</span><span class="p">])</span>

        <span class="n">new_cumulative_derivative</span> <span class="o">=</span> <span class="n">cumulative_derivative</span> <span class="o">*</span> <span class="n">decay_rate</span> <span class="o">+</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">decay_rate</span><span class="p">)</span> <span class="o">*</span> <span class="n">derivative</span> <span class="o">**</span> <span class="mi">2</span>
        <span class="n">step</span> <span class="o">=</span> <span class="o">-</span> <span class="n">derivative</span> <span class="o">*</span> <span class="n">T</span><span class="o">.</span><span class="n">sqrt</span><span class="p">((</span><span class="n">cumulative_step</span> <span class="o">+</span> <span class="n">epsilon</span><span class="p">)</span> <span class="o">/</span> <span class="p">(</span><span class="n">new_cumulative_derivative</span> <span class="o">+</span> <span class="n">epsilon</span><span class="p">))</span>

        <span class="n">updates</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">cumulative_derivative</span><span class="p">,</span> <span class="n">new_cumulative_derivative</span><span class="p">])</span>
        <span class="n">updates</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">cumulative_step</span><span class="p">,</span> <span class="n">cumulative_step</span> <span class="o">*</span> <span class="n">decay_rate</span> <span class="o">+</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">decay_rate</span><span class="p">)</span> <span class="o">*</span> <span class="n">step</span> <span class="o">**</span> <span class="mi">2</span><span class="p">])</span>
        <span class="n">updates</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">param</span><span class="p">,</span> <span class="n">param</span> <span class="o">+</span> <span class="n">learning_rate</span> <span class="o">*</span> <span class="n">step</span><span class="p">])</span>

    <span class="k">return</span> <span class="n">shareds</span><span class="p">,</span> <span class="n">updates</span></div>


<span class="n">trainers</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;sgd&#39;</span><span class="p">:</span> <span class="n">sgd_trainer</span><span class="p">,</span>
            <span class="s1">&#39;irprop-&#39;</span><span class="p">:</span> <span class="n">irprop_minus_trainer</span><span class="p">,</span>
            <span class="s1">&#39;irprop+&#39;</span><span class="p">:</span> <span class="n">irprop_plus_trainer</span><span class="p">,</span>
            <span class="s1">&#39;adadelta&#39;</span><span class="p">:</span> <span class="n">adadelta_trainer</span><span class="p">,</span>
            <span class="p">}</span>


<span class="c1"># endregion</span>


<span class="k">def</span> <span class="nf">_prepare_scaler</span><span class="p">(</span><span class="n">transform</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;Returns new transformer used in neural network</span>
<span class="sd">    :param transform: str ot transformer</span>
<span class="sd">    :return: transformer, cloned or created.</span>
<span class="sd">    &quot;&quot;&quot;</span>
    <span class="k">if</span> <span class="n">transform</span> <span class="o">==</span> <span class="s1">&#39;standard&#39;</span><span class="p">:</span>
        <span class="k">return</span> <span class="n">preprocessing</span><span class="o">.</span><span class="n">StandardScaler</span><span class="p">()</span>
    <span class="k">elif</span> <span class="n">transform</span> <span class="o">==</span> <span class="s1">&#39;minmax&#39;</span><span class="p">:</span>
        <span class="k">return</span> <span class="n">preprocessing</span><span class="o">.</span><span class="n">MinMaxScaler</span><span class="p">()</span>
    <span class="k">elif</span> <span class="n">transform</span> <span class="o">==</span> <span class="s1">&#39;iron&#39;</span><span class="p">:</span>
        <span class="k">return</span> <span class="n">IronTransformer</span><span class="p">(</span><span class="n">symmetrize</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
    <span class="k">else</span><span class="p">:</span>
        <span class="k">assert</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">transform</span><span class="p">,</span> <span class="n">TransformerMixin</span><span class="p">),</span> <span class="s1">&#39;provided transformer should be derived from TransformerMixin&#39;</span>
        <span class="k">return</span> <span class="n">clone</span><span class="p">(</span><span class="n">transform</span><span class="p">)</span>


<span class="k">class</span> <span class="nc">AbstractNeuralNetworkBase</span><span class="p">(</span><span class="n">BaseEstimator</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;Base class for neural networks&quot;&quot;&quot;</span>

    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">scaler</span><span class="p">,</span> <span class="n">loss</span><span class="p">,</span> <span class="n">trainer</span><span class="p">,</span> <span class="n">epochs</span><span class="p">,</span> <span class="n">trainer_parameters</span><span class="p">,</span> <span class="n">random_state</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">        :param layers: list of int, e.g [9, 7] - the number of units in each *hidden* layer</span>
<span class="sd">        :param scaler: &#39;standard&#39;, &#39;minmax&#39;, &#39;iron&#39; or some other Transformer used to transform features.</span>
<span class="sd">            Default is &#39;standard&#39;, which will apply StandardScaler.</span>
<span class="sd">            &#39;Iron&#39; is for heavy tails distributions.</span>
<span class="sd">        :param loss: loss function used.</span>
<span class="sd">            Options: {loss_options}</span>
<span class="sd">        :param trainer: string, name of optimization method used.</span>
<span class="sd">            Options: {trainer_options}</span>
<span class="sd">        :param epochs: number of times each sample takes part in training</span>
<span class="sd">        :param dict trainer_parameters: parameters passed to trainer function (learning_rate, etc., trainer-specific).</span>
<span class="sd">            See parameters in documentation.</span>
<span class="sd">        &quot;&quot;&quot;</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">scaler</span> <span class="o">=</span> <span class="n">scaler</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">layers</span> <span class="o">=</span> <span class="n">layers</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">loss</span> <span class="o">=</span> <span class="n">loss</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">epochs</span> <span class="o">=</span> <span class="n">epochs</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">parameters</span> <span class="o">=</span> <span class="p">{}</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">trainer</span> <span class="o">=</span> <span class="n">trainer</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">trainer_parameters</span> <span class="o">=</span> <span class="n">deepcopy</span><span class="p">(</span><span class="n">trainer_parameters</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">random_state</span> <span class="o">=</span> <span class="n">random_state</span>

    <span class="k">def</span> <span class="nf">_create_matrix_parameter</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">n1</span><span class="p">,</span> <span class="n">n2</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;Creates and registers matrix parameter of neural network&quot;&quot;&quot;</span>
        <span class="n">matrix</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span><span class="n">value</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">random_state_</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="n">size</span><span class="o">=</span><span class="p">[</span><span class="n">n1</span><span class="p">,</span> <span class="n">n2</span><span class="p">])</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="n">floatX</span><span class="p">)</span> <span class="o">*</span> <span class="mf">0.01</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="n">name</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">parameters</span><span class="p">[</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">matrix</span>
        <span class="k">return</span> <span class="n">matrix</span>

    <span class="k">def</span> <span class="nf">_create_scalar_parameter</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;Creates and registers scalar parameter of neural network&quot;&quot;&quot;</span>
        <span class="n">scalar_param</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span><span class="n">value</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">random_state_</span><span class="o">.</span><span class="n">normal</span><span class="p">()</span> <span class="o">*</span> <span class="mf">0.01</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="n">name</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">parameters</span><span class="p">[</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">scalar_param</span>
        <span class="k">return</span> <span class="n">scalar_param</span>

    <span class="k">def</span> <span class="nf">prepare</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;This method should provide activation function and set parameters.</span>
<span class="sd">        Each network overrides this function.</span>

<span class="sd">        :return: Activation function, f: X -&gt; df,</span>
<span class="sd">            X of shape [n_events, n_outputs], df (df = decision function) of shape [n_events].</span>
<span class="sd">            For classification, df is arbitrary real, the greater df, the more signal-like the event.</span>
<span class="sd">            Probabilities are computed by applying logistic function to output of activation.</span>
<span class="sd">        &quot;&quot;&quot;</span>
        <span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">()</span>

    <span class="k">def</span> <span class="nf">_prepare</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">n_input_features</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;This function is called once, it creates the activation function, it&#39;s gradient</span>
<span class="sd">        and initializes the weights</span>

<span class="sd">        :return: loss function as lambda (x, y, w) -&gt; loss</span>
<span class="sd">        &quot;&quot;&quot;</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">random_state_</span> <span class="o">=</span> <span class="n">check_random_state</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">random_state</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">layers_</span> <span class="o">=</span> <span class="p">[</span><span class="n">n_input_features</span><span class="p">]</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">)</span> <span class="o">+</span> <span class="p">[</span><span class="mi">1</span><span class="p">]</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">parameters</span> <span class="o">=</span> <span class="p">{}</span>
        <span class="n">loss_function</span> <span class="o">=</span> <span class="n">losses</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">loss</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">loss</span><span class="p">)</span>

        <span class="n">x</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">matrix</span><span class="p">(</span><span class="s1">&#39;X&#39;</span><span class="p">)</span>
        <span class="n">y</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">vector</span><span class="p">(</span><span class="s1">&#39;y&#39;</span><span class="p">)</span>
        <span class="n">w</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">vector</span><span class="p">(</span><span class="s1">&#39;w&#39;</span><span class="p">)</span>
        <span class="n">activation_raw</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">prepare</span><span class="p">()</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">Activation</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">function</span><span class="p">([</span><span class="n">x</span><span class="p">],</span> <span class="n">activation_raw</span><span class="p">(</span><span class="n">x</span><span class="p">)</span><span class="o">.</span><span class="n">flatten</span><span class="p">(),</span> <span class="n">allow_input_downcast</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
        <span class="n">loss_</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">:</span> <span class="n">loss_function</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">activation_raw</span><span class="p">(</span><span class="n">x</span><span class="p">)</span><span class="o">.</span><span class="n">flatten</span><span class="p">(),</span> <span class="n">w</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">Loss</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">function</span><span class="p">([</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">],</span> <span class="n">loss_</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">))</span>
        <span class="k">return</span> <span class="n">loss_</span>

    <span class="k">def</span> <span class="nf">_transform</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">fit</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;Apply selected scaler or transformer to dataset</span>
<span class="sd">        (also this method adds a column filled with ones).</span>

<span class="sd">        :param numpy.array X: of shape [n_samples, n_features], data</span>
<span class="sd">        :param numpy.array y: of shape [n_samples], labels</span>
<span class="sd">        :param bool fit: if True, fits transformer</span>
<span class="sd">        :return: transformed data, numpy.array of shape [n_samples, n_output_features]</span>
<span class="sd">        &quot;&quot;&quot;</span>
        <span class="c1"># Fighting copy-bug of sklearn&#39;s transformers</span>
        <span class="n">X</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="nb">float</span><span class="p">)</span>

        <span class="k">if</span> <span class="n">fit</span><span class="p">:</span>
            <span class="bp">self</span><span class="o">.</span><span class="n">scaler_</span> <span class="o">=</span> <span class="n">_prepare_scaler</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">scaler</span><span class="p">)</span>
            <span class="bp">self</span><span class="o">.</span><span class="n">scaler_</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>

        <span class="n">result</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">scaler_</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
        <span class="n">result</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">hstack</span><span class="p">([</span><span class="n">result</span><span class="p">,</span> <span class="n">numpy</span><span class="o">.</span><span class="n">ones</span><span class="p">([</span><span class="nb">len</span><span class="p">(</span><span class="n">X</span><span class="p">),</span> <span class="mi">1</span><span class="p">])])</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="s1">&#39;float32&#39;</span><span class="p">)</span>

        <span class="k">return</span> <span class="n">result</span>

    <span class="k">def</span> <span class="nf">_prepare_inputs</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span><span class="p">):</span>
        <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span> <span class="o">=</span> <span class="n">check_xyw</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span><span class="p">)</span>
        <span class="n">X</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_transform</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">fit</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span>

    <span class="k">def</span> <span class="nf">fit</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot; Prepare the model by optimizing selected loss function with some trainer.</span>

<span class="sd">        :param X: numpy.array of shape [n_samples, n_features]</span>
<span class="sd">        :param y: numpy.array of shape [n_samples]</span>
<span class="sd">        :param sample_weight: numpy.array of shape [n_samples], leave None for array of 1&#39;s</span>

<span class="sd">        :return: self</span>
<span class="sd">        &quot;&quot;&quot;</span>
        <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_prepare_inputs</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span><span class="o">=</span><span class="n">sample_weight</span><span class="p">)</span>

        <span class="n">loss_lambda</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_prepare</span><span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>

        <span class="n">trainer_function</span> <span class="o">=</span> <span class="n">trainers</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">trainer</span><span class="p">]</span>
        <span class="n">parameters_</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">trainer_parameters</span> <span class="ow">or</span> <span class="p">{}</span>

        <span class="n">x</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="n">floatX</span><span class="p">))</span>
        <span class="n">y</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span><span class="n">y</span><span class="p">)</span>
        <span class="n">w</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span><span class="n">sample_weight</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="n">floatX</span><span class="p">))</span>

        <span class="n">shareds</span><span class="p">,</span> <span class="n">updates</span> <span class="o">=</span> <span class="n">trainer_function</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">parameters</span><span class="p">,</span> <span class="n">loss_lambda</span><span class="p">,</span>
                                            <span class="n">RandomStreams</span><span class="p">(</span><span class="n">seed</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">random_state_</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1000</span><span class="p">)),</span> <span class="o">**</span><span class="n">parameters_</span><span class="p">)</span>

        <span class="n">make_one_step</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">function</span><span class="p">([],</span> <span class="p">[],</span> <span class="n">updates</span><span class="o">=</span><span class="n">updates</span><span class="p">,</span> <span class="n">allow_input_downcast</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>

        <span class="c1"># computing correct number of iterations in epoch is not simple:</span>
        <span class="k">if</span> <span class="s1">&#39;batch&#39;</span> <span class="ow">in</span> <span class="n">parameters_</span><span class="p">:</span>
            <span class="n">n_batches</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">X</span><span class="p">)</span> <span class="o">//</span> <span class="n">parameters_</span><span class="p">[</span><span class="s1">&#39;batch&#39;</span><span class="p">]</span> <span class="o">+</span> <span class="mi">1</span>
        <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">trainer</span> <span class="ow">in</span> <span class="p">[</span><span class="s1">&#39;adadelta&#39;</span><span class="p">,</span> <span class="s1">&#39;sgd&#39;</span><span class="p">]:</span>
            <span class="n">n_batches</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">X</span><span class="p">)</span> <span class="o">//</span> <span class="n">DEFAULT_MINIBATCH</span> <span class="o">+</span> <span class="mi">1</span>
        <span class="k">else</span><span class="p">:</span>
            <span class="n">n_batches</span> <span class="o">=</span> <span class="mi">1</span>

        <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">epochs</span><span class="p">):</span>
            <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_batches</span><span class="p">):</span>
                <span class="n">make_one_step</span><span class="p">()</span>

        <span class="k">return</span> <span class="bp">self</span>

    <span class="k">def</span> <span class="nf">decision_function</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">X</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">        Activates NN on particular dataset</span>

<span class="sd">        :param numpy.array X: of shape [n_samples, n_features]</span>
<span class="sd">        :return: numpy.array with results of shape [n_samples]</span>
<span class="sd">        &quot;&quot;&quot;</span>
        <span class="n">X</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_transform</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">fit</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">Activation</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">compute_loss</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;Computes loss (that was used in training) on labeled dataset</span>

<span class="sd">        :param X: numpy.array of shape [n_samples, n_features]</span>
<span class="sd">        :param y: numpy.array with integer labels of shape [n_samples],</span>
<span class="sd">            in two-class classification 0 and 1 labels should be used</span>
<span class="sd">        :param sample_weight: optional, numpy.array of shape [n_samples].</span>
<span class="sd">        :return float, the loss vales computed&quot;&quot;&quot;</span>
        <span class="n">sample_weight</span> <span class="o">=</span> <span class="n">check_sample_weight</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span><span class="p">)</span>
        <span class="n">X</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_transform</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">fit</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">Loss</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span><span class="p">)</span>


<div class="viewcode-block" id="AbstractNeuralNetworkClassifier"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.AbstractNeuralNetworkClassifier">[docs]</a><span class="k">class</span> <span class="nc">AbstractNeuralNetworkClassifier</span><span class="p">(</span><span class="n">AbstractNeuralNetworkBase</span><span class="p">,</span> <span class="n">ClassifierMixin</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">    Base class for classification neural networks.</span>
<span class="sd">    Supports only binary classification, supports weights, which makes it usable in boosting.</span>

<span class="sd">    Works as usual sklearn classifier, can be used in pipelines, ensembles, pickled, etc.</span>
<span class="sd">    &quot;&quot;&quot;</span>

    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">layers</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,),</span> <span class="n">scaler</span><span class="o">=</span><span class="s1">&#39;standard&#39;</span><span class="p">,</span> <span class="n">loss</span><span class="o">=</span><span class="s1">&#39;log_loss&#39;</span><span class="p">,</span> <span class="n">trainer</span><span class="o">=</span><span class="s1">&#39;irprop-&#39;</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span>
                 <span class="n">trainer_parameters</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
        <span class="n">AbstractNeuralNetworkBase</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">layers</span><span class="o">=</span><span class="n">layers</span><span class="p">,</span> <span class="n">scaler</span><span class="o">=</span><span class="n">scaler</span><span class="p">,</span> <span class="n">loss</span><span class="o">=</span><span class="n">loss</span><span class="p">,</span>
                                           <span class="n">trainer</span><span class="o">=</span><span class="n">trainer</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="n">epochs</span><span class="p">,</span> <span class="n">trainer_parameters</span><span class="o">=</span><span class="n">trainer_parameters</span><span class="p">,</span>
                                           <span class="n">random_state</span><span class="o">=</span><span class="n">random_state</span><span class="p">)</span>

    <span class="fm">__init__</span><span class="o">.</span><span class="vm">__doc__</span> <span class="o">=</span> <span class="n">AbstractNeuralNetworkBase</span><span class="o">.</span><span class="fm">__init__</span><span class="o">.</span><span class="vm">__doc__</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
        <span class="n">loss_options</span><span class="o">=</span><span class="n">classification_losses</span><span class="o">.</span><span class="n">keys</span><span class="p">(),</span> <span class="n">trainer_options</span><span class="o">=</span><span class="n">trainers</span><span class="o">.</span><span class="n">keys</span><span class="p">())</span>

    <span class="k">def</span> <span class="nf">_prepare_inputs</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span><span class="p">):</span>
        <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span> <span class="o">=</span> <span class="n">check_xyw</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span><span class="p">)</span>
        <span class="n">sample_weight</span> <span class="o">=</span> <span class="n">check_sample_weight</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span><span class="p">,</span> <span class="n">normalize</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
        <span class="n">X</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_transform</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">fit</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">classes_</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">])</span>
        <span class="k">assert</span> <span class="p">(</span><span class="n">numpy</span><span class="o">.</span><span class="n">unique</span><span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">classes_</span><span class="p">)</span><span class="o">.</span><span class="n">all</span><span class="p">(),</span> <span class="s1">&#39;only two-class classification supported, labels are 0 and 1&#39;</span>
        <span class="n">y</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="nb">int</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span>

<div class="viewcode-block" id="AbstractNeuralNetworkClassifier.predict_proba"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.AbstractNeuralNetworkClassifier.predict_proba">[docs]</a>    <span class="k">def</span> <span class="nf">predict_proba</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">X</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;Computes probability of each event to belong to each class</span>

<span class="sd">        :param numpy.array X: of shape [n_samples, n_features]</span>
<span class="sd">        :return: numpy.array of shape [n_samples, n_classes]</span>
<span class="sd">        &quot;&quot;&quot;</span>
        <span class="k">return</span> <span class="n">score_to_proba</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">decision_function</span><span class="p">(</span><span class="n">X</span><span class="p">))</span></div>

<div class="viewcode-block" id="AbstractNeuralNetworkClassifier.predict"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.AbstractNeuralNetworkClassifier.predict">[docs]</a>    <span class="k">def</span> <span class="nf">predict</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">X</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot; Predict the classes for new events (not recommended, use `predict_proba`).</span>

<span class="sd">        :param numpy.array X: of shape [n_samples, n_features]</span>
<span class="sd">        :return: numpy.array of shape [n_samples] with labels of predicted classes.</span>
<span class="sd">        &quot;&quot;&quot;</span>
        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">predict_proba</span><span class="p">(</span><span class="n">X</span><span class="p">)</span><span class="o">.</span><span class="n">argmax</span><span class="p">(</span><span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span></div></div>


<div class="viewcode-block" id="AbstractNeuralNetworkRegressor"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.AbstractNeuralNetworkRegressor">[docs]</a><span class="k">class</span> <span class="nc">AbstractNeuralNetworkRegressor</span><span class="p">(</span><span class="n">AbstractNeuralNetworkBase</span><span class="p">,</span> <span class="n">RegressorMixin</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">    Base class for regression neural networks. Supports weights.</span>

<span class="sd">    Works as usual sklearn classifier, can be used in pipelines, ensembles, pickled, etc.</span>
<span class="sd">    &quot;&quot;&quot;</span>

    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">layers</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,),</span> <span class="n">scaler</span><span class="o">=</span><span class="s1">&#39;standard&#39;</span><span class="p">,</span> <span class="n">loss</span><span class="o">=</span><span class="s1">&#39;mse_loss&#39;</span><span class="p">,</span> <span class="n">trainer</span><span class="o">=</span><span class="s1">&#39;irprop-&#39;</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span>
                 <span class="n">trainer_parameters</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
        <span class="n">AbstractNeuralNetworkBase</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">layers</span><span class="o">=</span><span class="n">layers</span><span class="p">,</span> <span class="n">scaler</span><span class="o">=</span><span class="n">scaler</span><span class="p">,</span> <span class="n">loss</span><span class="o">=</span><span class="n">loss</span><span class="p">,</span>
                                           <span class="n">trainer</span><span class="o">=</span><span class="n">trainer</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="n">epochs</span><span class="p">,</span> <span class="n">trainer_parameters</span><span class="o">=</span><span class="n">trainer_parameters</span><span class="p">,</span>
                                           <span class="n">random_state</span><span class="o">=</span><span class="n">random_state</span><span class="p">)</span>

    <span class="fm">__init__</span><span class="o">.</span><span class="vm">__doc__</span> <span class="o">=</span> <span class="n">AbstractNeuralNetworkBase</span><span class="o">.</span><span class="fm">__init__</span><span class="o">.</span><span class="vm">__doc__</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
        <span class="n">loss_options</span><span class="o">=</span><span class="n">regression_losses</span><span class="o">.</span><span class="n">keys</span><span class="p">(),</span> <span class="n">trainer_options</span><span class="o">=</span><span class="n">trainers</span><span class="o">.</span><span class="n">keys</span><span class="p">())</span>

<div class="viewcode-block" id="AbstractNeuralNetworkRegressor.predict"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.AbstractNeuralNetworkRegressor.predict">[docs]</a>    <span class="k">def</span> <span class="nf">predict</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">X</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot; Compute predictions for new events.</span>

<span class="sd">        :param numpy.array X: of shape [n_samples, n_features]</span>
<span class="sd">        :return: numpy.array of shape [n_samples] with labels of predicted classes &quot;&quot;&quot;</span>
        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">decision_function</span><span class="p">(</span><span class="n">X</span><span class="p">)</span></div></div>


<span class="c1"># region Neural networks</span>


<div class="viewcode-block" id="SimpleNeuralNetwork"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.SimpleNeuralNetwork">[docs]</a><span class="k">class</span> <span class="nc">SimpleNeuralNetwork</span><span class="p">(</span><span class="n">AbstractNeuralNetworkClassifier</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;The most simple NN with one hidden layer (sigmoid activation), for example purposes.</span>
<span class="sd">    Supports only one hidden layer.</span>

<span class="sd">    See source code as an example of custom NN.</span>
<span class="sd">    &quot;&quot;&quot;</span>

    <span class="k">def</span> <span class="nf">prepare</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="n">n1</span><span class="p">,</span> <span class="n">n2</span><span class="p">,</span> <span class="n">n3</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers_</span>
        <span class="n">W1</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_create_matrix_parameter</span><span class="p">(</span><span class="s1">&#39;W1&#39;</span><span class="p">,</span> <span class="n">n1</span><span class="p">,</span> <span class="n">n2</span><span class="p">)</span>
        <span class="n">W2</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_create_matrix_parameter</span><span class="p">(</span><span class="s1">&#39;W2&#39;</span><span class="p">,</span> <span class="n">n2</span><span class="p">,</span> <span class="n">n3</span><span class="p">)</span>

        <span class="k">def</span> <span class="nf">activation</span><span class="p">(</span><span class="nb">input</span><span class="p">):</span>
            <span class="n">first</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">nnet</span><span class="o">.</span><span class="n">sigmoid</span><span class="p">(</span><span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <span class="n">W1</span><span class="p">))</span>
            <span class="k">return</span> <span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">first</span><span class="p">,</span> <span class="n">W2</span><span class="p">)</span>

        <span class="k">return</span> <span class="n">activation</span></div>


<span class="k">class</span> <span class="nc">MLPBase</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">    MLPRegressor and MLPClassifier have identical implementation, but should be derived from different base classes.</span>
<span class="sd">    &quot;&quot;&quot;</span>

    <span class="k">def</span> <span class="nf">prepare</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="n">activation</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span>
        <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">layer</span> <span class="ow">in</span> <span class="nb">list</span><span class="p">(</span><span class="nb">enumerate</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">layers_</span><span class="p">))[</span><span class="mi">1</span><span class="p">:]:</span>
            <span class="n">W</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_create_matrix_parameter</span><span class="p">(</span><span class="s1">&#39;W&#39;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">i</span><span class="p">),</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers_</span><span class="p">[</span><span class="n">i</span> <span class="o">-</span> <span class="mi">1</span><span class="p">],</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers_</span><span class="p">[</span><span class="n">i</span><span class="p">])</span>
            <span class="c1"># act=activation and W_=W are tricks to avoid lambda-capturing</span>
            <span class="k">if</span> <span class="n">i</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
                <span class="n">activation</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">act</span><span class="o">=</span><span class="n">activation</span><span class="p">,</span> <span class="n">W_</span><span class="o">=</span><span class="n">W</span><span class="p">:</span> <span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">act</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="n">W_</span><span class="p">)</span>
            <span class="k">else</span><span class="p">:</span>
                <span class="n">activation</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">act</span><span class="o">=</span><span class="n">activation</span><span class="p">,</span> <span class="n">W_</span><span class="o">=</span><span class="n">W</span><span class="p">:</span> <span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">T</span><span class="o">.</span><span class="n">tanh</span><span class="p">(</span><span class="n">act</span><span class="p">(</span><span class="n">x</span><span class="p">)),</span> <span class="n">W_</span><span class="p">)</span>

        <span class="k">return</span> <span class="n">activation</span>


<div class="viewcode-block" id="MLPClassifier"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.MLPClassifier">[docs]</a><span class="k">class</span> <span class="nc">MLPClassifier</span><span class="p">(</span><span class="n">MLPBase</span><span class="p">,</span> <span class="n">AbstractNeuralNetworkClassifier</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">    MLP (MultiLayerPerceptron) for binary classification.</span>
<span class="sd">    Supports arbitrary number of layers (tanh is used as activation).</span>
<span class="sd">    &quot;&quot;&quot;</span>
    <span class="k">pass</span></div>


<div class="viewcode-block" id="MLPRegressor"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.MLPRegressor">[docs]</a><span class="k">class</span> <span class="nc">MLPRegressor</span><span class="p">(</span><span class="n">MLPBase</span><span class="p">,</span> <span class="n">AbstractNeuralNetworkRegressor</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">    MLP (MultiLayerPerceptron) regressor.</span>
<span class="sd">    Supports arbitrary number of layers (tanh is used as activation).</span>
<span class="sd">    &quot;&quot;&quot;</span>
    <span class="k">pass</span></div>


<div class="viewcode-block" id="MLPMultiClassifier"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.MLPMultiClassifier">[docs]</a><span class="k">class</span> <span class="nc">MLPMultiClassifier</span><span class="p">(</span><span class="n">MLPBase</span><span class="p">,</span> <span class="n">AbstractNeuralNetworkClassifier</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">    MLP (MultiLayerPerceptron) for multi-class classification.</span>
<span class="sd">    Supports arbitrary number of layers (tanh is used as activation).</span>
<span class="sd">    &quot;&quot;&quot;</span>

    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">layers</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,),</span> <span class="n">scaler</span><span class="o">=</span><span class="s1">&#39;standard&#39;</span><span class="p">,</span> <span class="n">trainer</span><span class="o">=</span><span class="s1">&#39;irprop-&#39;</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span>
                 <span class="n">trainer_parameters</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
        <span class="n">AbstractNeuralNetworkBase</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">layers</span><span class="o">=</span><span class="n">layers</span><span class="p">,</span> <span class="n">scaler</span><span class="o">=</span><span class="n">scaler</span><span class="p">,</span> <span class="n">loss</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
                                           <span class="n">trainer</span><span class="o">=</span><span class="n">trainer</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="n">epochs</span><span class="p">,</span> <span class="n">trainer_parameters</span><span class="o">=</span><span class="n">trainer_parameters</span><span class="p">,</span>
                                           <span class="n">random_state</span><span class="o">=</span><span class="n">random_state</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">_prepare</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">n_input_features</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;This function is called once, it creates the activation function, it&#39;s gradient</span>
<span class="sd">        and initializes the weights</span>

<span class="sd">        :return: loss function as lambda (x, y, w) -&gt; loss</span>
<span class="sd">        &quot;&quot;&quot;</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">random_state_</span> <span class="o">=</span> <span class="n">check_random_state</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">random_state</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">layers_</span> <span class="o">=</span> <span class="p">[</span><span class="n">n_input_features</span><span class="p">]</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">)</span> <span class="o">+</span> <span class="p">[</span><span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">classes_</span><span class="p">)]</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">parameters</span> <span class="o">=</span> <span class="p">{}</span>

        <span class="n">x</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">matrix</span><span class="p">(</span><span class="s1">&#39;X&#39;</span><span class="p">)</span>
        <span class="n">y</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">vector</span><span class="p">(</span><span class="s1">&#39;y&#39;</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="s1">&#39;int64&#39;</span><span class="p">)</span>
        <span class="n">w</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">vector</span><span class="p">(</span><span class="s1">&#39;w&#39;</span><span class="p">)</span>
        <span class="n">activation_raw</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">prepare</span><span class="p">()</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">Activation</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">function</span><span class="p">([</span><span class="n">x</span><span class="p">],</span> <span class="n">activation_raw</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="n">allow_input_downcast</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
        <span class="n">loss_</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">:</span> <span class="o">-</span><span class="n">T</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">w</span> <span class="o">*</span> <span class="n">T</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="n">T</span><span class="o">.</span><span class="n">nnet</span><span class="o">.</span><span class="n">softmax</span><span class="p">(</span><span class="n">activation_raw</span><span class="p">(</span><span class="n">x</span><span class="p">))[</span><span class="n">T</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="n">y</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]),</span> <span class="n">y</span><span class="p">]))</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">Loss</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">function</span><span class="p">([</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">],</span> <span class="n">loss_</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">w</span><span class="p">))</span>
        <span class="k">return</span> <span class="n">loss_</span>

    <span class="k">def</span> <span class="nf">_prepare_inputs</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span><span class="p">):</span>
        <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span> <span class="o">=</span> <span class="n">check_xyw</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span><span class="p">)</span>
        <span class="n">sample_weight</span> <span class="o">=</span> <span class="n">check_sample_weight</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span><span class="p">,</span> <span class="n">normalize</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
        <span class="n">X</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_transform</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">fit</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">classes_</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">unique</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">return_inverse</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
        <span class="n">y</span> <span class="o">=</span> <span class="n">y</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="s1">&#39;int32&#39;</span><span class="p">)</span>
        <span class="k">assert</span> <span class="n">numpy</span><span class="o">.</span><span class="n">allclose</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">classes_</span><span class="p">,</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">classes_</span><span class="p">))),</span> <span class="s1">&#39;Classes should be 0, 1... n_classes - 1&#39;</span>
        <span class="k">return</span> <span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">sample_weight</span>

    <span class="k">def</span> <span class="nf">predict_proba</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">X</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;Computes probability of each event to belong to each class</span>

<span class="sd">        :param numpy.array X: of shape [n_samples, n_features]</span>
<span class="sd">        :return: numpy.array of shape [n_samples, n_classes]</span>
<span class="sd">        &quot;&quot;&quot;</span>
        <span class="n">activations</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">decision_function</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
        <span class="c1"># taking softmax</span>
        <span class="n">activations</span> <span class="o">-=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">activations</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">keepdims</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
        <span class="n">probabilities</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="n">activations</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">probabilities</span> <span class="o">/</span> <span class="n">numpy</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">probabilities</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">keepdims</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span></div>


<div class="viewcode-block" id="RBFNeuralNetwork"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.RBFNeuralNetwork">[docs]</a><span class="k">class</span> <span class="nc">RBFNeuralNetwork</span><span class="p">(</span><span class="n">AbstractNeuralNetworkClassifier</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">    Neural network with one hidden layer with normalized RBF activation (Radial Basis Function).</span>
<span class="sd">    &quot;&quot;&quot;</span>

    <span class="k">def</span> <span class="nf">prepare</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="n">n1</span><span class="p">,</span> <span class="n">n2</span><span class="p">,</span> <span class="n">n3</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers_</span>
        <span class="n">W1</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_create_matrix_parameter</span><span class="p">(</span><span class="s1">&#39;W1&#39;</span><span class="p">,</span> <span class="n">n2</span><span class="p">,</span> <span class="n">n1</span><span class="p">)</span>
        <span class="n">W2</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_create_matrix_parameter</span><span class="p">(</span><span class="s1">&#39;W2&#39;</span><span class="p">,</span> <span class="n">n2</span><span class="p">,</span> <span class="n">n3</span><span class="p">)</span>
        <span class="c1"># this parameter is responsible for scaling, it is optimised too</span>
        <span class="n">G</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span><span class="n">value</span><span class="o">=</span><span class="mf">0.1</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s1">&#39;G&#39;</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">parameters</span><span class="p">[</span><span class="s1">&#39;G&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">G</span>

        <span class="k">def</span> <span class="nf">activation</span><span class="p">(</span><span class="nb">input</span><span class="p">):</span>
            <span class="n">translation_vectors</span> <span class="o">=</span> <span class="n">W1</span><span class="o">.</span><span class="n">reshape</span><span class="p">((</span><span class="mi">1</span><span class="p">,</span> <span class="n">W1</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="o">-</span><span class="mi">1</span><span class="p">))</span> <span class="o">-</span> <span class="nb">input</span><span class="o">.</span><span class="n">reshape</span><span class="p">((</span><span class="nb">input</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">))</span>
            <span class="n">minkowski_distances</span> <span class="o">=</span> <span class="p">(</span><span class="nb">abs</span><span class="p">(</span><span class="n">translation_vectors</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
            <span class="n">first</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">nnet</span><span class="o">.</span><span class="n">softmax</span><span class="p">(</span><span class="o">-</span> <span class="p">(</span><span class="mf">0.001</span> <span class="o">+</span> <span class="n">G</span> <span class="o">*</span> <span class="n">G</span><span class="p">)</span> <span class="o">*</span> <span class="n">minkowski_distances</span><span class="p">)</span>
            <span class="k">return</span> <span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">first</span><span class="p">,</span> <span class="n">W2</span><span class="p">)</span>

        <span class="k">return</span> <span class="n">activation</span></div>


<div class="viewcode-block" id="SoftmaxNeuralNetwork"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.SoftmaxNeuralNetwork">[docs]</a><span class="k">class</span> <span class="nc">SoftmaxNeuralNetwork</span><span class="p">(</span><span class="n">AbstractNeuralNetworkClassifier</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;Neural network with one hidden layer, softmax activation function &quot;&quot;&quot;</span>

    <span class="k">def</span> <span class="nf">prepare</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="n">n1</span><span class="p">,</span> <span class="n">n2</span><span class="p">,</span> <span class="n">n3</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers_</span>
        <span class="n">W1</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_create_matrix_parameter</span><span class="p">(</span><span class="s1">&#39;W1&#39;</span><span class="p">,</span> <span class="n">n1</span><span class="p">,</span> <span class="n">n2</span><span class="p">)</span>
        <span class="n">W2</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_create_matrix_parameter</span><span class="p">(</span><span class="s1">&#39;W2&#39;</span><span class="p">,</span> <span class="n">n2</span><span class="p">,</span> <span class="n">n3</span><span class="p">)</span>

        <span class="k">def</span> <span class="nf">activation</span><span class="p">(</span><span class="nb">input</span><span class="p">):</span>
            <span class="n">first</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">nnet</span><span class="o">.</span><span class="n">softmax</span><span class="p">(</span><span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <span class="n">W1</span><span class="p">))</span>
            <span class="k">return</span> <span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">first</span><span class="p">,</span> <span class="n">W2</span><span class="p">)</span>

        <span class="k">return</span> <span class="n">activation</span></div>


<div class="viewcode-block" id="PairwiseNeuralNetwork"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.PairwiseNeuralNetwork">[docs]</a><span class="k">class</span> <span class="nc">PairwiseNeuralNetwork</span><span class="p">(</span><span class="n">AbstractNeuralNetworkClassifier</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;The result is computed as :math:`h = sigmoid(Ax)`, :math:`output = \sum_{ij} B_{ij} h_i (1 - h_j)`,</span>
<span class="sd">    this is a brilliant example when easier to define activation</span>
<span class="sd">    function rather than trying to implement this inside some framework.</span>
<span class="sd">    &quot;&quot;&quot;</span>

    <span class="k">def</span> <span class="nf">prepare</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="n">n1</span><span class="p">,</span> <span class="n">n2</span><span class="p">,</span> <span class="n">n3</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers_</span>
        <span class="n">W1</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_create_matrix_parameter</span><span class="p">(</span><span class="s1">&#39;W1&#39;</span><span class="p">,</span> <span class="n">n1</span><span class="p">,</span> <span class="n">n2</span><span class="p">)</span>
        <span class="n">W2</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_create_matrix_parameter</span><span class="p">(</span><span class="s1">&#39;W2&#39;</span><span class="p">,</span> <span class="n">n2</span><span class="p">,</span> <span class="n">n2</span><span class="p">)</span>

        <span class="k">def</span> <span class="nf">activation</span><span class="p">(</span><span class="nb">input</span><span class="p">):</span>
            <span class="n">first</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">nnet</span><span class="o">.</span><span class="n">sigmoid</span><span class="p">(</span><span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <span class="n">W1</span><span class="p">))</span>
            <span class="k">return</span> <span class="n">T</span><span class="o">.</span><span class="n">batched_dot</span><span class="p">(</span><span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">first</span><span class="p">,</span> <span class="n">W2</span><span class="p">),</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">first</span><span class="p">)</span>

        <span class="k">return</span> <span class="n">activation</span></div>


<div class="viewcode-block" id="PairwiseSoftplusNeuralNetwork"><a class="viewcode-back" href="../../nnet.html#hep_ml.nnet.PairwiseSoftplusNeuralNetwork">[docs]</a><span class="k">class</span> <span class="nc">PairwiseSoftplusNeuralNetwork</span><span class="p">(</span><span class="n">AbstractNeuralNetworkClassifier</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;The result is computed as :math:`h1 = softplus(A_1 x)`, :math:`h2 = sigmoid(A_2 x)`,</span>
<span class="sd">        :math:`output = \sum_{ij} B_{ij} h1_i h2_j)`</span>
<span class="sd">    &quot;&quot;&quot;</span>

    <span class="k">def</span> <span class="nf">prepare</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="n">n1</span><span class="p">,</span> <span class="n">n2</span><span class="p">,</span> <span class="n">n3</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers_</span>
        <span class="n">A1</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_create_matrix_parameter</span><span class="p">(</span><span class="s1">&#39;A1&#39;</span><span class="p">,</span> <span class="n">n1</span><span class="p">,</span> <span class="n">n2</span><span class="p">)</span>
        <span class="n">A2</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_create_matrix_parameter</span><span class="p">(</span><span class="s1">&#39;A2&#39;</span><span class="p">,</span> <span class="n">n1</span><span class="p">,</span> <span class="n">n2</span><span class="p">)</span>
        <span class="n">B</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_create_matrix_parameter</span><span class="p">(</span><span class="s1">&#39;B&#39;</span><span class="p">,</span> <span class="n">n2</span><span class="p">,</span> <span class="n">n2</span><span class="p">)</span>

        <span class="k">def</span> <span class="nf">activation</span><span class="p">(</span><span class="nb">input</span><span class="p">):</span>
            <span class="n">first1</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">nnet</span><span class="o">.</span><span class="n">softplus</span><span class="p">(</span><span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <span class="n">A1</span><span class="p">))</span>
            <span class="n">first2</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">nnet</span><span class="o">.</span><span class="n">sigmoid</span><span class="p">(</span><span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <span class="n">A2</span><span class="p">))</span>
            <span class="k">return</span> <span class="n">T</span><span class="o">.</span><span class="n">batched_dot</span><span class="p">(</span><span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">first1</span><span class="p">,</span> <span class="n">B</span><span class="p">),</span> <span class="n">first2</span><span class="p">)</span>

        <span class="k">return</span> <span class="n">activation</span></div>

<span class="c1"># endregion</span>
</pre></div>

           </div>
           
          </div>
          <footer>

  <hr/>

  <div role="contentinfo">
    <p>
        &#169; Copyright 2015-2017, Yandex; Alex Rogozhnikov and contributors.

    </p>
  </div>
    
    
    
    Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
    
    <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
    
    provided by <a href="https://readthedocs.org">Read the Docs</a>. 

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>