

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>Neural networks &mdash; hep_ml 0.7.0 documentation</title>
  

  
  <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
  <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />

  
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
        <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
        <script src="_static/jquery.js"></script>
        <script src="_static/underscore.js"></script>
        <script src="_static/doctools.js"></script>
        <script async="async" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
    
    <script type="text/javascript" src="_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" />
    <link rel="next" title="Preprocessing data" href="preprocessing.html" />
    <link rel="prev" title="Metric functions" href="metrics.html" /> 
</head>

<body class="wy-body-for-nav">

   
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search" >
          

          
            <a href="index.html" class="icon icon-home"> hep_ml
          

          
          </a>

          
            
            
              <div class="version">
                0.7.0
              </div>
            
          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="index.html">hep_ml documentation</a></li>
<li class="toctree-l1"><a class="reference internal" href="gb.html">Gradient boosting</a></li>
<li class="toctree-l1"><a class="reference internal" href="losses.html">Losses for Gradient Boosting</a></li>
<li class="toctree-l1"><a class="reference internal" href="uboost.html">uBoost</a></li>
<li class="toctree-l1"><a class="reference internal" href="metrics.html">Metric functions</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Neural networks</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#examples">Examples</a></li>
<li class="toctree-l2"><a class="reference internal" href="#interface">Interface</a></li>
<li class="toctree-l2"><a class="reference internal" href="#recommended-networks">Recommended networks</a></li>
<li class="toctree-l2"><a class="reference internal" href="#custom-networks">Custom networks</a></li>
<li class="toctree-l2"><a class="reference internal" href="#loss-functions">Loss functions</a></li>
<li class="toctree-l2"><a class="reference internal" href="#trainers">Trainers</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="preprocessing.html">Preprocessing data</a></li>
<li class="toctree-l1"><a class="reference internal" href="reweight.html">Reweighting algorithms</a></li>
<li class="toctree-l1"><a class="reference internal" href="speedup.html">Fast predictions</a></li>
<li class="toctree-l1"><a class="reference internal" href="splot.html">sPlot</a></li>
<li class="toctree-l1"><a class="reference internal" href="notebooks.html">Code Examples</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="index.html">hep_ml</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          

















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="index.html" class="icon icon-home"></a> &raquo;</li>
        
      <li>Neural networks</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
          
            <a href="_sources/nnet.rst.txt" rel="nofollow"> View page source</a>
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
  <div class="section" id="module-hep_ml.nnet">
<span id="neural-networks"></span><h1>Neural networks<a class="headerlink" href="#module-hep_ml.nnet" title="Permalink to this headline">¶</a></h1>
<p><strong>hep_ml.nnet</strong> is minimalistic <strong>theano</strong>-powered version of feed-forward neural networks.
The neural networks from this library provide sklearn classifier’s interface.</p>
<p>Definitions for loss functions, trainers of neural networks are defined in this file too.
Main point of this library: black-box stochastic optimization of any given loss function.
This gives ability to define any activation expression (at the cost of unavailability of pretraining).</p>
<p>In this file we have <strong>examples</strong> of neural networks,
user is encouraged to write his own specific architecture,
which can be much more complex than those used usually.</p>
<p>If you don’t want to dive into details, use
<a class="reference internal" href="#hep_ml.nnet.MLPClassifier" title="hep_ml.nnet.MLPClassifier"><code class="xref py py-class docutils literal notranslate"><span class="pre">hep_ml.nnet.MLPClassifier</span></code></a>, <a class="reference internal" href="#hep_ml.nnet.MLPRegressor" title="hep_ml.nnet.MLPRegressor"><code class="xref py py-class docutils literal notranslate"><span class="pre">hep_ml.nnet.MLPRegressor</span></code></a></p>
<p>This library should be preferred for different experiments with architectures.
Also <strong>hep_ml.nnet</strong> allows optimization of parameters in any differentiable decision function.</p>
<p>Being written in theano, these neural networks are able to make use of your GPU.</p>
<p>See also libraries: keras, mxnet, pytorch.</p>
<div class="section" id="examples">
<h2>Examples<a class="headerlink" href="#examples" title="Permalink to this headline">¶</a></h2>
<p>Training a neural network with two hidden layers using IRPROP- algorithm</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">network</span> <span class="o">=</span> <span class="n">MLPClassifier</span><span class="p">(</span><span class="n">layers</span><span class="o">=</span><span class="p">[</span><span class="mi">7</span><span class="p">,</span> <span class="mi">7</span><span class="p">],</span> <span class="n">loss</span><span class="o">=</span><span class="s1">&#39;log_loss&#39;</span><span class="p">,</span> <span class="n">trainer</span><span class="o">=</span><span class="s1">&#39;irprop-&#39;</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="mi">1000</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">network</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">probability</span> <span class="o">=</span> <span class="n">network</span><span class="o">.</span><span class="n">predict_proba</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
</pre></div>
</div>
<p>Training an AdaBoost over neural network with adadelta trainer. Trainer specific parameter was used
(size of minibatch)</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">sklearn.ensemble</span> <span class="kn">import</span> <span class="n">AdaBoostClassifier</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">base_network</span> <span class="o">=</span> <span class="n">MLPClassifier</span><span class="p">(</span><span class="n">layers</span><span class="o">=</span><span class="p">[</span><span class="mi">10</span><span class="p">],</span> <span class="n">trainer</span><span class="o">=</span><span class="s1">&#39;adadelta&#39;</span><span class="p">,</span> <span class="n">trainer_parameters</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;batch&#39;</span><span class="p">:</span> <span class="mi">600</span><span class="p">})</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">classifier</span> <span class="o">=</span> <span class="n">AdaBoostClassifier</span><span class="p">(</span><span class="n">base_estimator</span><span class="o">=</span><span class="n">base_network</span><span class="p">,</span> <span class="n">n_estimators</span><span class="o">=</span><span class="mi">20</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">classifier</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
</pre></div>
</div>
<p>Using custom pretransformer and ExponentialLoss:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">PolynomialFeatures</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">network</span> <span class="o">=</span> <span class="n">MLPClassifier</span><span class="p">(</span><span class="n">layers</span><span class="o">=</span><span class="p">[</span><span class="mi">10</span><span class="p">],</span> <span class="n">scaler</span><span class="o">=</span><span class="n">PolynomialFeatures</span><span class="p">(),</span> <span class="n">loss</span><span class="o">=</span><span class="s1">&#39;exp_loss&#39;</span><span class="p">)</span>
</pre></div>
</div>
<p>To create custom neural network, see code of SimpleNeuralNetwork, which is good place to start.</p>
</div>
<div class="section" id="interface">
<h2>Interface<a class="headerlink" href="#interface" title="Permalink to this headline">¶</a></h2>
<p>Below an interface for classifier and regressor is demonstrated.</p>
<dl class="py class">
<dt class="sig sig-object py" id="hep_ml.nnet.AbstractNeuralNetworkClassifier">
<em class="property"><span class="pre">class</span> </em><span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">AbstractNeuralNetworkClassifier</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">layers</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">(10)</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">scaler</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'standard'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">loss</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'log_loss'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'irprop-'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">epochs</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">100</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer_parameters</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">random_state</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#AbstractNeuralNetworkClassifier"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.AbstractNeuralNetworkClassifier" title="Permalink to this definition">¶</a></dt>
<dd><p>Base class for classification neural networks.
Supports only binary classification, supports weights, which makes it usable in boosting.</p>
<p>Works as usual sklearn classifier, can be used in pipelines, ensembles, pickled, etc.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>layers</strong> – list of int, e.g [9, 7] - the number of units in each <em>hidden</em> layer</p></li>
<li><p><strong>scaler</strong> – ‘standard’, ‘minmax’, ‘iron’ or some other Transformer used to transform features.
Default is ‘standard’, which will apply StandardScaler.
‘Iron’ is for heavy tails distributions.</p></li>
<li><p><strong>loss</strong> – loss function used.
Options: dict_keys([‘exp_loss’, ‘log_loss’, ‘exp_log_loss’, ‘squared_loss’])</p></li>
<li><p><strong>trainer</strong> – string, name of optimization method used.
Options: dict_keys([‘sgd’, ‘irprop-‘, ‘irprop+’, ‘adadelta’])</p></li>
<li><p><strong>epochs</strong> – number of times each sample takes part in training</p></li>
<li><p><strong>trainer_parameters</strong> (<em>dict</em>) – parameters passed to trainer function (learning_rate, etc., trainer-specific).
See parameters in documentation.</p></li>
</ul>
</dd>
</dl>
<dl class="py method">
<dt class="sig sig-object py" id="hep_ml.nnet.AbstractNeuralNetworkClassifier.fit">
<span class="sig-name descname"><span class="pre">fit</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">X</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">y</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sample_weight</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#hep_ml.nnet.AbstractNeuralNetworkClassifier.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Prepare the model by optimizing selected loss function with some trainer.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>X</strong> – numpy.array of shape [n_samples, n_features]</p></li>
<li><p><strong>y</strong> – numpy.array of shape [n_samples]</p></li>
<li><p><strong>sample_weight</strong> – numpy.array of shape [n_samples], leave None for array of 1’s</p></li>
</ul>
</dd>
<dt class="field-even">Returns</dt>
<dd class="field-even"><p>self</p>
</dd>
</dl>
</dd></dl>

<dl class="py method">
<dt class="sig sig-object py" id="hep_ml.nnet.AbstractNeuralNetworkClassifier.predict">
<span class="sig-name descname"><span class="pre">predict</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">X</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#AbstractNeuralNetworkClassifier.predict"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.AbstractNeuralNetworkClassifier.predict" title="Permalink to this definition">¶</a></dt>
<dd><p>Predict the classes for new events (not recommended, use <cite>predict_proba</cite>).</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><p><strong>X</strong> (<em>numpy.array</em>) – of shape [n_samples, n_features]</p>
</dd>
<dt class="field-even">Returns</dt>
<dd class="field-even"><p>numpy.array of shape [n_samples] with labels of predicted classes.</p>
</dd>
</dl>
</dd></dl>

<dl class="py method">
<dt class="sig sig-object py" id="hep_ml.nnet.AbstractNeuralNetworkClassifier.predict_proba">
<span class="sig-name descname"><span class="pre">predict_proba</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">X</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#AbstractNeuralNetworkClassifier.predict_proba"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.AbstractNeuralNetworkClassifier.predict_proba" title="Permalink to this definition">¶</a></dt>
<dd><p>Computes probability of each event to belong to each class</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><p><strong>X</strong> (<em>numpy.array</em>) – of shape [n_samples, n_features]</p>
</dd>
<dt class="field-even">Returns</dt>
<dd class="field-even"><p>numpy.array of shape [n_samples, n_classes]</p>
</dd>
</dl>
</dd></dl>

</dd></dl>

<dl class="py class">
<dt class="sig sig-object py" id="hep_ml.nnet.AbstractNeuralNetworkRegressor">
<em class="property"><span class="pre">class</span> </em><span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">AbstractNeuralNetworkRegressor</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">layers</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">(10)</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">scaler</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'standard'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">loss</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'mse_loss'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'irprop-'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">epochs</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">100</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer_parameters</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">random_state</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#AbstractNeuralNetworkRegressor"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.AbstractNeuralNetworkRegressor" title="Permalink to this definition">¶</a></dt>
<dd><p>Base class for regression neural networks. Supports weights.</p>
<p>Works as usual sklearn classifier, can be used in pipelines, ensembles, pickled, etc.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>layers</strong> – list of int, e.g [9, 7] - the number of units in each <em>hidden</em> layer</p></li>
<li><p><strong>scaler</strong> – ‘standard’, ‘minmax’, ‘iron’ or some other Transformer used to transform features.
Default is ‘standard’, which will apply StandardScaler.
‘Iron’ is for heavy tails distributions.</p></li>
<li><p><strong>loss</strong> – loss function used.
Options: dict_keys([‘mse_loss’, ‘smooth_huber_loss’])</p></li>
<li><p><strong>trainer</strong> – string, name of optimization method used.
Options: dict_keys([‘sgd’, ‘irprop-‘, ‘irprop+’, ‘adadelta’])</p></li>
<li><p><strong>epochs</strong> – number of times each sample takes part in training</p></li>
<li><p><strong>trainer_parameters</strong> (<em>dict</em>) – parameters passed to trainer function (learning_rate, etc., trainer-specific).
See parameters in documentation.</p></li>
</ul>
</dd>
</dl>
<dl class="py method">
<dt class="sig sig-object py" id="hep_ml.nnet.AbstractNeuralNetworkRegressor.fit">
<span class="sig-name descname"><span class="pre">fit</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">X</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">y</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">sample_weight</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#hep_ml.nnet.AbstractNeuralNetworkRegressor.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Prepare the model by optimizing selected loss function with some trainer.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>X</strong> – numpy.array of shape [n_samples, n_features]</p></li>
<li><p><strong>y</strong> – numpy.array of shape [n_samples]</p></li>
<li><p><strong>sample_weight</strong> – numpy.array of shape [n_samples], leave None for array of 1’s</p></li>
</ul>
</dd>
<dt class="field-even">Returns</dt>
<dd class="field-even"><p>self</p>
</dd>
</dl>
</dd></dl>

<dl class="py method">
<dt class="sig sig-object py" id="hep_ml.nnet.AbstractNeuralNetworkRegressor.predict">
<span class="sig-name descname"><span class="pre">predict</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">X</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#AbstractNeuralNetworkRegressor.predict"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.AbstractNeuralNetworkRegressor.predict" title="Permalink to this definition">¶</a></dt>
<dd><p>Compute predictions for new events.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><p><strong>X</strong> (<em>numpy.array</em>) – of shape [n_samples, n_features]</p>
</dd>
<dt class="field-even">Returns</dt>
<dd class="field-even"><p>numpy.array of shape [n_samples] with labels of predicted classes</p>
</dd>
</dl>
</dd></dl>

</dd></dl>

</div>
<div class="section" id="recommended-networks">
<h2>Recommended networks<a class="headerlink" href="#recommended-networks" title="Permalink to this headline">¶</a></h2>
<p>Multilayer Perceptron is well-known popular algorithm working well in most applications.</p>
<dl class="py class">
<dt class="sig sig-object py" id="hep_ml.nnet.MLPClassifier">
<em class="property"><span class="pre">class</span> </em><span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">MLPClassifier</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">layers</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">(10)</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">scaler</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'standard'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">loss</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'log_loss'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'irprop-'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">epochs</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">100</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer_parameters</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">random_state</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#MLPClassifier"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.MLPClassifier" title="Permalink to this definition">¶</a></dt>
<dd><p>MLP (MultiLayerPerceptron) for binary classification.
Supports arbitrary number of layers (tanh is used as activation).</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>layers</strong> – list of int, e.g [9, 7] - the number of units in each <em>hidden</em> layer</p></li>
<li><p><strong>scaler</strong> – ‘standard’, ‘minmax’, ‘iron’ or some other Transformer used to transform features.
Default is ‘standard’, which will apply StandardScaler.
‘Iron’ is for heavy tails distributions.</p></li>
<li><p><strong>loss</strong> – loss function used.
Options: dict_keys([‘exp_loss’, ‘log_loss’, ‘exp_log_loss’, ‘squared_loss’])</p></li>
<li><p><strong>trainer</strong> – string, name of optimization method used.
Options: dict_keys([‘sgd’, ‘irprop-‘, ‘irprop+’, ‘adadelta’])</p></li>
<li><p><strong>epochs</strong> – number of times each sample takes part in training</p></li>
<li><p><strong>trainer_parameters</strong> (<em>dict</em>) – parameters passed to trainer function (learning_rate, etc., trainer-specific).
See parameters in documentation.</p></li>
</ul>
</dd>
</dl>
</dd></dl>

<dl class="py class">
<dt class="sig sig-object py" id="hep_ml.nnet.MLPRegressor">
<em class="property"><span class="pre">class</span> </em><span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">MLPRegressor</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">layers</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">(10)</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">scaler</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'standard'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">loss</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'mse_loss'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'irprop-'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">epochs</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">100</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer_parameters</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">random_state</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#MLPRegressor"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.MLPRegressor" title="Permalink to this definition">¶</a></dt>
<dd><p>MLP (MultiLayerPerceptron) regressor.
Supports arbitrary number of layers (tanh is used as activation).</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>layers</strong> – list of int, e.g [9, 7] - the number of units in each <em>hidden</em> layer</p></li>
<li><p><strong>scaler</strong> – ‘standard’, ‘minmax’, ‘iron’ or some other Transformer used to transform features.
Default is ‘standard’, which will apply StandardScaler.
‘Iron’ is for heavy tails distributions.</p></li>
<li><p><strong>loss</strong> – loss function used.
Options: dict_keys([‘mse_loss’, ‘smooth_huber_loss’])</p></li>
<li><p><strong>trainer</strong> – string, name of optimization method used.
Options: dict_keys([‘sgd’, ‘irprop-‘, ‘irprop+’, ‘adadelta’])</p></li>
<li><p><strong>epochs</strong> – number of times each sample takes part in training</p></li>
<li><p><strong>trainer_parameters</strong> (<em>dict</em>) – parameters passed to trainer function (learning_rate, etc., trainer-specific).
See parameters in documentation.</p></li>
</ul>
</dd>
</dl>
</dd></dl>

<dl class="py class">
<dt class="sig sig-object py" id="hep_ml.nnet.MLPMultiClassifier">
<em class="property"><span class="pre">class</span> </em><span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">MLPMultiClassifier</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">layers</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">(10)</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">scaler</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'standard'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'irprop-'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">epochs</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">100</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer_parameters</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">random_state</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#MLPMultiClassifier"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.MLPMultiClassifier" title="Permalink to this definition">¶</a></dt>
<dd><p>MLP (MultiLayerPerceptron) for multi-class classification.
Supports arbitrary number of layers (tanh is used as activation).</p>
</dd></dl>

</div>
<div class="section" id="custom-networks">
<h2>Custom networks<a class="headerlink" href="#custom-networks" title="Permalink to this headline">¶</a></h2>
<p>Below some examples of custom networks are given.
Those are initial point for constructing own architectures.</p>
<dl class="py class">
<dt class="sig sig-object py" id="hep_ml.nnet.SimpleNeuralNetwork">
<em class="property"><span class="pre">class</span> </em><span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">SimpleNeuralNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">layers</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">(10)</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">scaler</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'standard'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">loss</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'log_loss'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'irprop-'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">epochs</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">100</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer_parameters</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">random_state</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#SimpleNeuralNetwork"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.SimpleNeuralNetwork" title="Permalink to this definition">¶</a></dt>
<dd><p>The most simple NN with one hidden layer (sigmoid activation), for example purposes.
Supports only one hidden layer.</p>
<p>See source code as an example of custom NN.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>layers</strong> – list of int, e.g [9, 7] - the number of units in each <em>hidden</em> layer</p></li>
<li><p><strong>scaler</strong> – ‘standard’, ‘minmax’, ‘iron’ or some other Transformer used to transform features.
Default is ‘standard’, which will apply StandardScaler.
‘Iron’ is for heavy tails distributions.</p></li>
<li><p><strong>loss</strong> – loss function used.
Options: dict_keys([‘exp_loss’, ‘log_loss’, ‘exp_log_loss’, ‘squared_loss’])</p></li>
<li><p><strong>trainer</strong> – string, name of optimization method used.
Options: dict_keys([‘sgd’, ‘irprop-‘, ‘irprop+’, ‘adadelta’])</p></li>
<li><p><strong>epochs</strong> – number of times each sample takes part in training</p></li>
<li><p><strong>trainer_parameters</strong> (<em>dict</em>) – parameters passed to trainer function (learning_rate, etc., trainer-specific).
See parameters in documentation.</p></li>
</ul>
</dd>
</dl>
</dd></dl>

<dl class="py class">
<dt class="sig sig-object py" id="hep_ml.nnet.SoftmaxNeuralNetwork">
<em class="property"><span class="pre">class</span> </em><span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">SoftmaxNeuralNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">layers</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">(10)</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">scaler</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'standard'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">loss</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'log_loss'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'irprop-'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">epochs</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">100</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer_parameters</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">random_state</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#SoftmaxNeuralNetwork"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.SoftmaxNeuralNetwork" title="Permalink to this definition">¶</a></dt>
<dd><p>Neural network with one hidden layer, softmax activation function</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>layers</strong> – list of int, e.g [9, 7] - the number of units in each <em>hidden</em> layer</p></li>
<li><p><strong>scaler</strong> – ‘standard’, ‘minmax’, ‘iron’ or some other Transformer used to transform features.
Default is ‘standard’, which will apply StandardScaler.
‘Iron’ is for heavy tails distributions.</p></li>
<li><p><strong>loss</strong> – loss function used.
Options: dict_keys([‘exp_loss’, ‘log_loss’, ‘exp_log_loss’, ‘squared_loss’])</p></li>
<li><p><strong>trainer</strong> – string, name of optimization method used.
Options: dict_keys([‘sgd’, ‘irprop-‘, ‘irprop+’, ‘adadelta’])</p></li>
<li><p><strong>epochs</strong> – number of times each sample takes part in training</p></li>
<li><p><strong>trainer_parameters</strong> (<em>dict</em>) – parameters passed to trainer function (learning_rate, etc., trainer-specific).
See parameters in documentation.</p></li>
</ul>
</dd>
</dl>
</dd></dl>

<dl class="py class">
<dt class="sig sig-object py" id="hep_ml.nnet.RBFNeuralNetwork">
<em class="property"><span class="pre">class</span> </em><span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">RBFNeuralNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">layers</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">(10)</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">scaler</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'standard'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">loss</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'log_loss'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'irprop-'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">epochs</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">100</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer_parameters</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">random_state</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#RBFNeuralNetwork"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.RBFNeuralNetwork" title="Permalink to this definition">¶</a></dt>
<dd><p>Neural network with one hidden layer with normalized RBF activation (Radial Basis Function).</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>layers</strong> – list of int, e.g [9, 7] - the number of units in each <em>hidden</em> layer</p></li>
<li><p><strong>scaler</strong> – ‘standard’, ‘minmax’, ‘iron’ or some other Transformer used to transform features.
Default is ‘standard’, which will apply StandardScaler.
‘Iron’ is for heavy tails distributions.</p></li>
<li><p><strong>loss</strong> – loss function used.
Options: dict_keys([‘exp_loss’, ‘log_loss’, ‘exp_log_loss’, ‘squared_loss’])</p></li>
<li><p><strong>trainer</strong> – string, name of optimization method used.
Options: dict_keys([‘sgd’, ‘irprop-‘, ‘irprop+’, ‘adadelta’])</p></li>
<li><p><strong>epochs</strong> – number of times each sample takes part in training</p></li>
<li><p><strong>trainer_parameters</strong> (<em>dict</em>) – parameters passed to trainer function (learning_rate, etc., trainer-specific).
See parameters in documentation.</p></li>
</ul>
</dd>
</dl>
</dd></dl>

<dl class="py class">
<dt class="sig sig-object py" id="hep_ml.nnet.PairwiseNeuralNetwork">
<em class="property"><span class="pre">class</span> </em><span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">PairwiseNeuralNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">layers</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">(10)</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">scaler</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'standard'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">loss</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'log_loss'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'irprop-'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">epochs</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">100</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer_parameters</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">random_state</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#PairwiseNeuralNetwork"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.PairwiseNeuralNetwork" title="Permalink to this definition">¶</a></dt>
<dd><p>The result is computed as <span class="math notranslate nohighlight">\(h = sigmoid(Ax)\)</span>, <span class="math notranslate nohighlight">\(output = \sum_{ij} B_{ij} h_i (1 - h_j)\)</span>,
this is a brilliant example when easier to define activation
function rather than trying to implement this inside some framework.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>layers</strong> – list of int, e.g [9, 7] - the number of units in each <em>hidden</em> layer</p></li>
<li><p><strong>scaler</strong> – ‘standard’, ‘minmax’, ‘iron’ or some other Transformer used to transform features.
Default is ‘standard’, which will apply StandardScaler.
‘Iron’ is for heavy tails distributions.</p></li>
<li><p><strong>loss</strong> – loss function used.
Options: dict_keys([‘exp_loss’, ‘log_loss’, ‘exp_log_loss’, ‘squared_loss’])</p></li>
<li><p><strong>trainer</strong> – string, name of optimization method used.
Options: dict_keys([‘sgd’, ‘irprop-‘, ‘irprop+’, ‘adadelta’])</p></li>
<li><p><strong>epochs</strong> – number of times each sample takes part in training</p></li>
<li><p><strong>trainer_parameters</strong> (<em>dict</em>) – parameters passed to trainer function (learning_rate, etc., trainer-specific).
See parameters in documentation.</p></li>
</ul>
</dd>
</dl>
</dd></dl>

<dl class="py class">
<dt class="sig sig-object py" id="hep_ml.nnet.PairwiseSoftplusNeuralNetwork">
<em class="property"><span class="pre">class</span> </em><span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">PairwiseSoftplusNeuralNetwork</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">layers</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">(10)</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">scaler</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'standard'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">loss</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'log_loss'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'irprop-'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">epochs</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">100</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trainer_parameters</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">random_state</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#PairwiseSoftplusNeuralNetwork"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.PairwiseSoftplusNeuralNetwork" title="Permalink to this definition">¶</a></dt>
<dd><p>The result is computed as <span class="math notranslate nohighlight">\(h1 = softplus(A_1 x)\)</span>, <span class="math notranslate nohighlight">\(h2 = sigmoid(A_2 x)\)</span>,
<span class="math notranslate nohighlight">\(output = \sum_{ij} B_{ij} h1_i h2_j)\)</span></p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>layers</strong> – list of int, e.g [9, 7] - the number of units in each <em>hidden</em> layer</p></li>
<li><p><strong>scaler</strong> – ‘standard’, ‘minmax’, ‘iron’ or some other Transformer used to transform features.
Default is ‘standard’, which will apply StandardScaler.
‘Iron’ is for heavy tails distributions.</p></li>
<li><p><strong>loss</strong> – loss function used.
Options: dict_keys([‘exp_loss’, ‘log_loss’, ‘exp_log_loss’, ‘squared_loss’])</p></li>
<li><p><strong>trainer</strong> – string, name of optimization method used.
Options: dict_keys([‘sgd’, ‘irprop-‘, ‘irprop+’, ‘adadelta’])</p></li>
<li><p><strong>epochs</strong> – number of times each sample takes part in training</p></li>
<li><p><strong>trainer_parameters</strong> (<em>dict</em>) – parameters passed to trainer function (learning_rate, etc., trainer-specific).
See parameters in documentation.</p></li>
</ul>
</dd>
</dl>
</dd></dl>

</div>
<div class="section" id="loss-functions">
<h2>Loss functions<a class="headerlink" href="#loss-functions" title="Permalink to this headline">¶</a></h2>
<p>The following loss functions are available for <strong>classification</strong>:</p>
<dl class="py function">
<dt class="sig sig-object py" id="hep_ml.nnet.log_loss">
<span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">log_loss</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">y</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">pred</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">w</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#log_loss"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.log_loss" title="Permalink to this definition">¶</a></dt>
<dd><p>Logistic loss for classification (aka cross-entropy, aka binomial deviance)</p>
</dd></dl>

<dl class="py function">
<dt class="sig sig-object py" id="hep_ml.nnet.exp_loss">
<span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">exp_loss</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">y</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">pred</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">w</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#exp_loss"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.exp_loss" title="Permalink to this definition">¶</a></dt>
<dd><p>Exponential loss for classification (aka AdaLoss function)</p>
</dd></dl>

<dl class="py function">
<dt class="sig sig-object py" id="hep_ml.nnet.exp_log_loss">
<span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">exp_log_loss</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">y</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">pred</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">w</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#exp_log_loss"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.exp_log_loss" title="Permalink to this definition">¶</a></dt>
<dd><p>Classification loss function,
combines logistic loss for signal and exponential loss for background</p>
</dd></dl>

<dl class="py function">
<dt class="sig sig-object py" id="hep_ml.nnet.squared_loss">
<span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">squared_loss</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">y</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">pred</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">w</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#squared_loss"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.squared_loss" title="Permalink to this definition">¶</a></dt>
<dd><p>Squared loss for classification, not to be messed up with MSE</p>
</dd></dl>

<p>The following loss functions are available for <strong>regression</strong>:</p>
<dl class="py function">
<dt class="sig sig-object py" id="hep_ml.nnet.mse_loss">
<span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">mse_loss</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">y</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">pred</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">w</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#mse_loss"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.mse_loss" title="Permalink to this definition">¶</a></dt>
<dd><p>Regression loss function, mean squared error.</p>
</dd></dl>

<dl class="py function">
<dt class="sig sig-object py" id="hep_ml.nnet.smooth_huber_loss">
<span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">smooth_huber_loss</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">y</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">pred</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">w</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#smooth_huber_loss"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.smooth_huber_loss" title="Permalink to this definition">¶</a></dt>
<dd><p>Regression loss function, smooth version of Huber loss function.</p>
</dd></dl>

</div>
<div class="section" id="trainers">
<h2>Trainers<a class="headerlink" href="#trainers" title="Permalink to this headline">¶</a></h2>
<p>The trainers are optimization algorithms used to minimize target loss function in neural networks.
The trainers are implemented as functions with some standard parameters and some optional.</p>
<dl class="py function">
<dt class="sig sig-object py" id="hep_ml.nnet.sgd_trainer">
<span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">sgd_trainer</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">x</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">y</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">w</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">parameters</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">loss</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">random_stream</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">batch</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">30</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">learning_rate</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">0.1</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">l2_penalty</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">0.001</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">momentum</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">0.9</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#sgd_trainer"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.sgd_trainer" title="Permalink to this definition">¶</a></dt>
<dd><p>Stochastic gradient descent with momentum, trivial but very popular.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>batch</strong> (<em>int</em>) – size of minibatch, each time averaging gradient over minibatch.</p></li>
<li><p><strong>learning_rate</strong> (<em>float</em>) – size of step</p></li>
<li><p><strong>l2_penalty</strong> (<em>float</em>) – speed of weights’ decay, l2 regularization prevents overfitting</p></li>
<li><p><strong>momentum</strong> (<em>float</em>) – momentum to stabilize learning process.</p></li>
</ul>
</dd>
</dl>
</dd></dl>

<dl class="py function">
<dt class="sig sig-object py" id="hep_ml.nnet.irprop_minus_trainer">
<span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">irprop_minus_trainer</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">x</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">y</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">w</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">parameters</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">loss</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">random_stream</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">positive_step</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">1.2</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">negative_step</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">0.5</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_step</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">1.0</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">min_step</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">1e-06</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#irprop_minus_trainer"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.irprop_minus_trainer" title="Permalink to this definition">¶</a></dt>
<dd><p>IRPROP- is batch trainer, for details see <a class="reference external" href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.21.3428">http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.21.3428</a> .
This is default trainer, very stable for classification.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>positive_step</strong> – factor, by which the step is increased when continuing going in the direction</p></li>
<li><p><strong>negative_step</strong> – factor, by which the step is increased when changing direction to opposite</p></li>
<li><p><strong>min_step</strong> – minimal change of weight during iteration</p></li>
<li><p><strong>max_step</strong> – maximal change of weight during iteration</p></li>
</ul>
</dd>
</dl>
</dd></dl>

<dl class="py function">
<dt class="sig sig-object py" id="hep_ml.nnet.irprop_plus_trainer">
<span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">irprop_plus_trainer</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">x</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">y</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">w</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">parameters</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">loss</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">random_stream</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">positive_step</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">1.2</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">negative_step</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">0.5</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">max_step</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">1.0</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">min_step</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">1e-06</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#irprop_plus_trainer"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.irprop_plus_trainer" title="Permalink to this definition">¶</a></dt>
<dd><p>IRPROP+ is batch trainer, for details see <a class="reference external" href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.21.3428">http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.21.3428</a></p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>positive_step</strong> – factor, by which the step is increased when continuing going in the direction</p></li>
<li><p><strong>negative_step</strong> – factor, by which the step is increased when changing direction to opposite</p></li>
<li><p><strong>min_step</strong> – minimal change of weight during iteration</p></li>
<li><p><strong>max_step</strong> – maximal change of weight during iteration</p></li>
</ul>
</dd>
</dl>
</dd></dl>

<dl class="py function">
<dt class="sig sig-object py" id="hep_ml.nnet.adadelta_trainer">
<span class="sig-prename descclassname"><span class="pre">hep_ml.nnet.</span></span><span class="sig-name descname"><span class="pre">adadelta_trainer</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">x</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">y</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">w</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">parameters</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">loss</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">random_stream</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">batch</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">30</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">learning_rate</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">0.1</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">half_life</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">1000</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">epsilon</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">0.0001</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hep_ml/nnet.html#adadelta_trainer"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#hep_ml.nnet.adadelta_trainer" title="Permalink to this definition">¶</a></dt>
<dd><p>AdaDelta is trainer with adaptive learning rate.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>half_life</strong> – momentum-like parameter. The estimated parameters are decreased by 2 after so many events.
It is recommended for small datasets to put halflife = number of samples in dataset.</p></li>
<li><p><strong>learning_rate</strong> – size of step</p></li>
<li><p><strong>batch</strong> – size of minibatch</p></li>
<li><p><strong>epsilon</strong> – regularization</p></li>
</ul>
</dd>
</dl>
</dd></dl>

</div>
</div>


           </div>
           
          </div>
          <footer>
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
        <a href="preprocessing.html" class="btn btn-neutral float-right" title="Preprocessing data" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
        <a href="metrics.html" class="btn btn-neutral float-left" title="Metric functions" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>
        &#169; Copyright 2015-2017, Yandex; Alex Rogozhnikov and contributors.

    </p>
  </div>
    
    
    
    Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
    
    <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
    
    provided by <a href="https://readthedocs.org">Read the Docs</a>. 

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>