<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>3.1. Generalized Linear Models — scikits.learn v0.6.0 documentation</title>
<link rel="stylesheet" href="../_static/nature.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../',
VERSION: '0.6.0',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<link rel="shortcut icon" href="../_static/favicon.ico"/>
<link rel="author" title="About these documents" href="../about.html" />
<link rel="top" title="scikits.learn v0.6.0 documentation" href="../index.html" />
<link rel="up" title="3. Supervised learning" href="../supervised_learning.html" />
<link rel="next" title="3.2. Support Vector Machines" href="svm.html" />
<link rel="prev" title="3. Supervised learning" href="../supervised_learning.html" />
</head>
<body>
<div class="header-wrapper">
<div class="header">
<p class="logo"><a href="../index.html">
<img src="../_static/scikit-learn-logo-small.png" alt="Logo"/>
</a>
</p><div class="navbar">
<ul>
<li><a href="../install.html">Download</a></li>
<li><a href="../support.html">Support</a></li>
<li><a href="../user_guide.html">User Guide</a></li>
<li><a href="../auto_examples/index.html">Examples</a></li>
<li><a href="../developers/index.html">Development</a></li>
</ul>
<div class="search_form">
<div id="cse" style="width: 100%;"></div>
<script src="http://www.google.com/jsapi" type="text/javascript"></script>
<script type="text/javascript">
google.load('search', '1', {language : 'en'});
google.setOnLoadCallback(function() {
var customSearchControl = new google.search.CustomSearchControl('016639176250731907682:tjtqbvtvij0');
customSearchControl.setResultSetSize(google.search.Search.FILTERED_CSE_RESULTSET);
var options = new google.search.DrawOptions();
options.setAutoComplete(true);
customSearchControl.draw('cse', options);
}, true);
</script>
</div>
</div> <!-- end navbar --></div>
</div>
<div class="content-wrapper">
<!-- <div id="blue_tile"></div> -->
<div class="sphinxsidebar">
<div class="rel">
<a href="../supervised_learning.html" title="3. Supervised learning"
accesskey="P">previous</a> |
<a href="svm.html" title="3.2. Support Vector Machines"
accesskey="N">next</a> |
<a href="../genindex.html" title="General Index"
accesskey="I">index</a>
</div>
<h3>Contents</h3>
<ul>
<li><a class="reference internal" href="#">3.1. Generalized Linear Models</a><ul>
<li><a class="reference internal" href="#ordinary-least-squares-ols">3.1.1. Ordinary Least Squares (OLS)</a><ul>
<li><a class="reference internal" href="#ols-complexity">3.1.1.1. OLS Complexity</a></li>
</ul>
</li>
<li><a class="reference internal" href="#ridge-regression">3.1.2. Ridge Regression</a><ul>
<li><a class="reference internal" href="#ridge-complexity">3.1.2.1. Ridge Complexity</a></li>
</ul>
</li>
<li><a class="reference internal" href="#lasso">3.1.3. Lasso</a></li>
<li><a class="reference internal" href="#elastic-net">3.1.4. Elastic Net</a></li>
<li><a class="reference internal" href="#least-angle-regression">3.1.5. Least Angle Regression</a></li>
<li><a class="reference internal" href="#lars-lasso">3.1.6. LARS Lasso</a><ul>
<li><a class="reference internal" href="#mathematical-formulation">3.1.6.1. Mathematical formulation</a></li>
</ul>
</li>
<li><a class="reference internal" href="#bayesian-regression">3.1.7. Bayesian Regression</a><ul>
<li><a class="reference internal" href="#bayesian-ridge-regression">3.1.7.1. Bayesian Ridge Regression</a></li>
</ul>
</li>
<li><a class="reference internal" href="#automatic-relevance-determination-ard">3.1.8. Automatic Relevance Determination - ARD</a><ul>
<li><a class="reference internal" href="#id3">3.1.8.1. Mathematical formulation</a></li>
</ul>
</li>
<li><a class="reference internal" href="#stochastic-gradient-descent-sgd">3.1.9. Stochastic Gradient Descent - SGD</a></li>
</ul>
</li>
</ul>
</div>
<div class="content">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="generalized-linear-models">
<span id="linear-model"></span><h1>3.1. Generalized Linear Models<a class="headerlink" href="#generalized-linear-models" title="Permalink to this headline">¶</a></h1>
<p>The following are a set of methods intended for regression in which
the target value is expected to be a linear combination of the input
variables. In mathematical notion, if <span class="math">\hat{y}</span> is the predicted
value.</p>
<div class="math">
<p><span class="math">\hat{y}(\beta, x) = \beta_0 + \beta_1 x_1 + ... + \beta_D x_D</span></p>
</div><p>Across the module, we designate the vector <span class="math">\beta = (\beta_1,
..., \beta_D)</span> as <tt class="docutils literal"><span class="pre">coef_</span></tt> and <span class="math">\beta_0</span> as <tt class="docutils literal"><span class="pre">intercept_</span></tt>.</p>
<div class="section" id="ordinary-least-squares-ols">
<span id="ordinary-least-squares"></span><h2>3.1.1. Ordinary Least Squares (OLS)<a class="headerlink" href="#ordinary-least-squares-ols" title="Permalink to this headline">¶</a></h2>
<p><tt class="xref py py-class docutils literal"><span class="pre">LinearRegression</span></tt> fits a linear model with coefficients
<span class="math">\beta = (\beta_1, ..., \beta_D)</span> to minimize the residual sum
of squares between the observed responses in the dataset, and the
responses predicted by the linear approximation.</p>
<div class="figure align-center">
<a class="reference external image-reference" href="../auto_examples/linear_model/plot_ols.html"><img alt="auto_examples/linear_model/images/plot_ols.png" src="auto_examples/linear_model/images/plot_ols.png" /></a>
</div>
<p><tt class="xref py py-class docutils literal"><span class="pre">LinearRegression</span></tt> will take in its <cite>fit</cite> method arrays X, y
and will store the coefficients <span class="math">w</span> of the linear model in its
<cite>coef_</cite> member.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">scikits.learn</span> <span class="kn">import</span> <span class="n">linear_model</span>
<span class="gp">>>> </span><span class="n">clf</span> <span class="o">=</span> <span class="n">linear_model</span><span class="o">.</span><span class="n">LinearRegression</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">clf</span><span class="o">.</span><span class="n">fit</span> <span class="p">([[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">],</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">]],</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">])</span>
<span class="go">LinearRegression(fit_intercept=True)</span>
<span class="gp">>>> </span><span class="n">clf</span><span class="o">.</span><span class="n">coef_</span>
<span class="go">array([ 0.5, 0.5])</span>
</pre></div>
</div>
<p>However, coefficient estimates for Ordinary Least Squares rely on the
independence of the model terms. When terms are correlated and the
columns of the design matrix <span class="math">X</span> have an approximate linear
dependence, the matrix <span class="math">X(X^T X)^{-1}</span> becomes close to singular
and as a result, the least-squares estimate becomes highly sensitive
to random errors in the observed response, producing a large
variance. This situation of <em>multicollinearity</em> can arise, for
example, when data are collected without an experimental design.</p>
<div class="topic">
<p class="topic-title first">Examples:</p>
<ul class="simple">
<li><a class="reference internal" href="../auto_examples/linear_model/plot_ols.html#example-linear-model-plot-ols-py"><em>Ordinary Least Squares</em></a></li>
</ul>
</div>
<div class="section" id="ols-complexity">
<h3>3.1.1.1. OLS Complexity<a class="headerlink" href="#ols-complexity" title="Permalink to this headline">¶</a></h3>
<p>This method computes the least squares solution using a singular value
decomposition of X. If X is a matrix of size (n, p ) this method has a
cost of <span class="math">O(n p^2)</span>, assuming that <span class="math">n \geq p</span>.</p>
</div>
</div>
<div class="section" id="ridge-regression">
<h2>3.1.2. Ridge Regression<a class="headerlink" href="#ridge-regression" title="Permalink to this headline">¶</a></h2>
<p><tt class="xref py py-class docutils literal"><span class="pre">Ridge</span></tt> regression addresses some of the problems of
<a class="reference internal" href="#ordinary-least-squares"><em>Ordinary Least Squares (OLS)</em></a> by imposing a penalty on the size of
coefficients. The ridge coefficients minimize a penalized residual sum
of squares,</p>
<div class="math">
<p><span class="math">\beta^{ridge} = \underset{\beta}{argmin} { \sum_{i=1}{N} (y_i -
\beta_0 - \sum_{j=1}{p} x_ij \beta_j)^2 + \alpha
\sum_{j=1}{p} \beta_{j}^2}</span></p>
</div><p>Here, <span class="math">\alpha \geq 0</span> is a complexity parameter that controls
the amount of shrinkage: the larger the value of <span class="math">\alpha</span>, the
greater the amount of shrinkage.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">scikits.learn</span> <span class="kn">import</span> <span class="n">linear_model</span>
<span class="gp">>>> </span><span class="n">clf</span> <span class="o">=</span> <span class="n">linear_model</span><span class="o">.</span><span class="n">Ridge</span> <span class="p">(</span><span class="n">alpha</span> <span class="o">=</span> <span class="o">.</span><span class="mi">5</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">clf</span><span class="o">.</span><span class="n">fit</span> <span class="p">([[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">]],</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">.</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">])</span>
<span class="go">Ridge(alpha=0.5, fit_intercept=True)</span>
<span class="gp">>>> </span><span class="n">clf</span><span class="o">.</span><span class="n">coef_</span>
<span class="go">array([ 0.34545455, 0.34545455])</span>
<span class="gp">>>> </span><span class="n">clf</span><span class="o">.</span><span class="n">intercept_</span>
<span class="go">0.13636...</span>
</pre></div>
</div>
<div class="section" id="ridge-complexity">
<h3>3.1.2.1. Ridge Complexity<a class="headerlink" href="#ridge-complexity" title="Permalink to this headline">¶</a></h3>
<p>This method has the same order of complexity than an
<a class="reference internal" href="#ordinary-least-squares"><em>Ordinary Least Squares (OLS)</em></a>.</p>
</div>
</div>
<div class="section" id="lasso">
<h2>3.1.3. Lasso<a class="headerlink" href="#lasso" title="Permalink to this headline">¶</a></h2>
<p>The <tt class="xref py py-class docutils literal"><span class="pre">Lasso</span></tt> is a linear model trained with L1 prior as
regularizer. The objective function to minimize is:</p>
<div class="math">
<p><span class="math">0.5 * ||y - X w||_2 ^ 2 + \alpha * ||w||_1</span></p>
</div><p>The lasso estimate thus solves the minimization of the
least-squares penalty with <span class="math">\alpha * ||w||_1</span> added, where
<span class="math">\alpha</span> is a constant and <span class="math">||w||_1</span> is the L1-norm of the
parameter vector.</p>
<p>This formulation is useful in some contexts due to its tendency to
prefer solutions with fewer parameter values, effectively reducing the
number of variables upon which the given solution is dependent. For
this reason, the Lasso and its variants are fundamental to the field
of compressed sensing.</p>
<p>This implementation uses coordinate descent as the algorithm to fit
the coefficients. See <a class="reference internal" href="#least-angle-regression"><em>Least Angle Regression</em></a> for another implementation.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">clf</span> <span class="o">=</span> <span class="n">linear_model</span><span class="o">.</span><span class="n">Lasso</span><span class="p">(</span><span class="n">alpha</span> <span class="o">=</span> <span class="mf">0.1</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">clf</span><span class="o">.</span><span class="n">fit</span> <span class="p">([[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">]],</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">])</span>
<span class="go">Lasso(alpha=0.1, fit_intercept=True)</span>
<span class="gp">>>> </span><span class="n">clf</span><span class="o">.</span><span class="n">predict</span> <span class="p">([[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">]])</span>
<span class="go">array([ 0.8])</span>
</pre></div>
</div>
<p>The function <tt class="xref py py-func docutils literal"><span class="pre">lasso_path()</span></tt> computes the coefficients along the full path
of possible values.</p>
<div class="topic">
<p class="topic-title first">Examples:</p>
<ul class="simple">
<li><a class="reference internal" href="../auto_examples/linear_model/lasso_and_elasticnet.html#example-linear-model-lasso-and-elasticnet-py"><em>Lasso regression example</em></a>,</li>
<li><a class="reference internal" href="../auto_examples/linear_model/lasso_path_with_crossvalidation.html#example-linear-model-lasso-path-with-crossvalidation-py"><em>Lasso parameter estimation with path and cross-validation</em></a></li>
</ul>
</div>
</div>
<div class="section" id="elastic-net">
<h2>3.1.4. Elastic Net<a class="headerlink" href="#elastic-net" title="Permalink to this headline">¶</a></h2>
<p><tt class="xref py py-class docutils literal"><span class="pre">ElasticNet</span></tt> is a linear model trained with L1 and L2 prior as
regularizer.</p>
<p>The objective function to minimize is in this case</p>
<div class="math">
<p><span class="math">0.5 * ||y - X w||_2 ^ 2 + \alpha * \rho * ||w||_1 + \alpha * (1-\rho) * 0.5 * ||w||_2 ^ 2</span></p>
</div><div class="topic">
<p class="topic-title first">Examples:</p>
<ul class="simple">
<li><a class="reference internal" href="../auto_examples/linear_model/lasso_and_elasticnet.html#example-linear-model-lasso-and-elasticnet-py"><em>Lasso regression example</em></a></li>
<li><a class="reference internal" href="../auto_examples/linear_model/plot_lasso_coordinate_descent_path.html#example-linear-model-plot-lasso-coordinate-descent-path-py"><em>Lasso and Elastic Net</em></a></li>
</ul>
</div>
</div>
<div class="section" id="least-angle-regression">
<span id="id1"></span><h2>3.1.5. Least Angle Regression<a class="headerlink" href="#least-angle-regression" title="Permalink to this headline">¶</a></h2>
<p>Least-angle regression (LARS) is a regression algorithm for
high-dimensional data, developed by Bradley Efron, Trevor Hastie, Iain
Johnstone and Robert Tibshirani.</p>
<p>The advantages of LARS are:</p>
<blockquote>
<ul class="simple">
<li>It is computationally just as fast as forward selection and has
the same order of complexity as an ordinary least squares.</li>
<li>It produces a full piecewise linear solution path, which is
useful in cross-validation or similar attempts to tune the model.</li>
<li>If two variables are almost equally correlated with the response,
then their coefficients should increase at approximately the same
rate. The algorithm thus behaves as intuition would expect, and
also is more stable.</li>
<li>It is easily modified to produce solutions for other estimators,
like the Lasso.</li>
<li>It is effective in contexts where p >> n (i.e., when the number of
dimensions is significantly greater than the number of points)</li>
</ul>
</blockquote>
<p>The disadvantages of the LARS method include:</p>
<blockquote>
<ul class="simple">
<li>Because LARS is based upon an iterative refitting of the
residuals, it would appear to be especially sensitive to the
effects of noise. This problem is discussed in detail by Weisberg
in the discussion section of the Efron et al. (2004) Annals of
Statistics article.</li>
</ul>
</blockquote>
<p>The LARS model can be used using estimator <tt class="xref py py-class docutils literal"><span class="pre">LARS</span></tt>, or its
low-level implementation <tt class="xref py py-func docutils literal"><span class="pre">lars_path()</span></tt>.</p>
</div>
<div class="section" id="lars-lasso">
<h2>3.1.6. LARS Lasso<a class="headerlink" href="#lars-lasso" title="Permalink to this headline">¶</a></h2>
<p><tt class="xref py py-class docutils literal"><span class="pre">LassoLARS</span></tt> is a lasso model implemented using the LARS
algorithm, and unlike the implementation based on coordinate_descent,
this yields the exact solution, which is piecewise linear as a
function of the norm of its coefficients.</p>
<div class="figure align-center">
<a class="reference external image-reference" href="../auto_examples/linear_model/plot_lasso_lars.html"><img alt="auto_examples/linear_model/images/plot_lasso_lars.png" src="auto_examples/linear_model/images/plot_lasso_lars.png" /></a>
</div>
<div class="topic">
<p class="topic-title first">Examples:</p>
<ul class="simple">
<li><a class="reference internal" href="../auto_examples/linear_model/plot_lasso_lars.html#example-linear-model-plot-lasso-lars-py"><em>Lasso path using LARS</em></a></li>
</ul>
</div>
<p>The LARS algorithm provides the full path of the coefficients along
the regularization parameter almost for free, thus a common operation
consist of retrieving the path with function <tt class="xref py py-func docutils literal"><span class="pre">lars_path()</span></tt></p>
<div class="section" id="mathematical-formulation">
<h3>3.1.6.1. Mathematical formulation<a class="headerlink" href="#mathematical-formulation" title="Permalink to this headline">¶</a></h3>
<p>The algorithm is similar to forward stepwise regression, but instead
of including variables at each step, the estimated parameters are
increased in a direction equiangular to each one’s correlations with
the residual.</p>
<p>Instead of giving a vector result, the LARS solution consists of a
curve denoting the solution for each value of the L1 norm of the
parameter vector. The full coeffients path is stored in the array
<tt class="docutils literal"><span class="pre">coef_path_</span></tt>, which has size (n_features, max_features+1). The first
column is always zero.</p>
<div class="topic">
<p class="topic-title first">References:</p>
<ul class="simple">
<li>Original Algorithm is detailed in the paper <a class="reference external" href="http://www-stat.stanford.edu/~hastie/Papers/LARS/LeastAngle_2002.pdf">Least Angle Regression</a>
by Hastie et al.</li>
</ul>
</div>
</div>
</div>
<div class="section" id="bayesian-regression">
<h2>3.1.7. Bayesian Regression<a class="headerlink" href="#bayesian-regression" title="Permalink to this headline">¶</a></h2>
<p>Bayesian regression techniques can be used to include regularization
parameters in the estimation procedure. This can be done by
introducing some prior knowledge over the parameters. For example,
penalization by weighted <span class="math">\ell_{2}</span> norm is equivalent to
setting Gaussian priors on the weights.</p>
<p>The advantages of <em>Bayesian Regression</em> are:</p>
<blockquote>
<ul class="simple">
<li>It adapts to the data at hand.</li>
<li>It can be used to include regularization parameters in the
estimation procedure.</li>
</ul>
</blockquote>
<p>The disadvantages of <em>Bayesian Regression</em> include:</p>
<blockquote>
<ul class="simple">
<li>Inference of the model can be time consuming.</li>
</ul>
</blockquote>
<div class="section" id="bayesian-ridge-regression">
<h3>3.1.7.1. Bayesian Ridge Regression<a class="headerlink" href="#bayesian-ridge-regression" title="Permalink to this headline">¶</a></h3>
<p><tt class="xref py py-class docutils literal"><span class="pre">BayesianRidge</span></tt> tries to avoid the overfit issue of
<a class="reference internal" href="#ordinary-least-squares"><em>Ordinary Least Squares (OLS)</em></a>, by adding the following prior on
<span class="math">\beta</span>:</p>
<div class="math">
<p><span class="math">p(\beta|\lambda) =
\mathcal{N}(\beta|0,\lambda^{-1}\bold{I_{p}})</span></p>
</div><p>The resulting model is called <em>Bayesian Ridge Regression</em>, it is
similar to the classical <tt class="xref py py-class docutils literal"><span class="pre">Ridge</span></tt>. <span class="math">\lambda</span> is an
<em>hyper-parameter</em> and the prior over <span class="math">\beta</span> performs a
shrinkage or regularization, by constraining the values of the weights
to be small. Indeed, with a large value of <span class="math">\lambda</span>, the
Gaussian is narrowed around 0 which does not allow large values of
<span class="math">\beta</span>, and with low value of <span class="math">\lambda</span>, the Gaussian is
very flattened which allows values of <span class="math">\beta</span>. Here, we use a
<em>non-informative</em> prior for <span class="math">\lambda</span>.
The parameters are estimated by maximizing the <em>marginal log likelihood</em>.
There is also a Gamma prior for <span class="math">\lambda</span> and <span class="math">\alpha</span>:</p>
<div class="math">
<p><span class="math">g(\alpha|\alpha_1,\alpha_2) = \frac{\alpha_2^{\alpha_1}}
{\Gamma(\alpha_1)} \alpha^{\alpha_1-1} e^{-\alpha_2 {\alpha}}</span></p>
</div><div class="math">
<p><span class="math">g(\lambda|\lambda_1,\lambda_2) = \frac{\lambda_2^{\lambda_1}}
{\Gamma(\lambda_1)} \lambda^{\lambda_1-1} e^{-\lambda_2 {\lambda}}</span></p>
</div><dl class="docutils">
<dt>By default <span class="math">\alpha_1 = \alpha_2 = \lambda_1 = \lambda_2 = 1.e^{-6}</span>, <em>i.e.</em></dt>
<dd>very slightly informative priors.</dd>
</dl>
<div class="figure align-center">
<a class="reference external image-reference" href="../auto_examples/linear_model/plot_bayesian_ridge.html"><img alt="auto_examples/linear_model/images/plot_bayesian_ridge.png" src="auto_examples/linear_model/images/plot_bayesian_ridge.png" /></a>
</div>
<p><em>Bayesian Ridge Regression</em> is used for regression:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">scikits.learn</span> <span class="kn">import</span> <span class="n">linear_model</span>
<span class="gp">>>> </span><span class="n">X</span> <span class="o">=</span> <span class="p">[[</span><span class="mf">0.</span><span class="p">,</span> <span class="mf">0.</span><span class="p">],</span> <span class="p">[</span><span class="mf">1.</span><span class="p">,</span> <span class="mf">1.</span><span class="p">],</span> <span class="p">[</span><span class="mf">2.</span><span class="p">,</span> <span class="mf">2.</span><span class="p">],</span> <span class="p">[</span><span class="mf">3.</span><span class="p">,</span> <span class="mf">3.</span><span class="p">]]</span>
<span class="gp">>>> </span><span class="n">Y</span> <span class="o">=</span> <span class="p">[</span><span class="mf">0.</span><span class="p">,</span> <span class="mf">1.</span><span class="p">,</span> <span class="mf">2.</span><span class="p">,</span> <span class="mf">3.</span><span class="p">]</span>
<span class="gp">>>> </span><span class="n">clf</span> <span class="o">=</span> <span class="n">linear_model</span><span class="o">.</span><span class="n">BayesianRidge</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">clf</span><span class="o">.</span><span class="n">fit</span> <span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">Y</span><span class="p">)</span>
<span class="go">BayesianRidge(n_iter=300, verbose=False, lambda_1=1e-06, lambda_2=1e-06,</span>
<span class="go"> fit_intercept=True, eps=0.001, alpha_2=1e-06, alpha_1=1e-06,</span>
<span class="go"> compute_score=False)</span>
</pre></div>
</div>
<p>After being fitted, the model can then be used to predict new values:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">clf</span><span class="o">.</span><span class="n">predict</span> <span class="p">([[</span><span class="mi">1</span><span class="p">,</span> <span class="mf">0.</span><span class="p">]])</span>
<span class="go">array([ 0.50000013])</span>
</pre></div>
</div>
<p>The weights <span class="math">\beta</span> of the model can be access:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">clf</span><span class="o">.</span><span class="n">coef_</span>
<span class="go">array([ 0.49999993, 0.49999993])</span>
</pre></div>
</div>
<p>Due to the Bayesian framework, the weights found are slightly different to the
ones found by <a class="reference internal" href="#ordinary-least-squares"><em>Ordinary Least Squares (OLS)</em></a>. However, <em>Bayesian Ridge
Regression</em> is more robust to ill-posed problem.</p>
<div class="topic">
<p class="topic-title first">Examples:</p>
<ul class="simple">
<li><a class="reference internal" href="../auto_examples/linear_model/plot_bayesian_ridge.html#example-linear-model-plot-bayesian-ridge-py"><em>Bayesian Ridge Regression</em></a></li>
</ul>
</div>
<div class="topic">
<p class="topic-title first">References</p>
<ul class="simple">
<li>More details can be found in the article <a class="reference external" href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.27.9072&rep=rep1&type=pdf">Bayesian Interpolation</a>
by MacKay, David J. C.</li>
</ul>
</div>
</div>
</div>
<div class="section" id="automatic-relevance-determination-ard">
<h2>3.1.8. Automatic Relevance Determination - ARD<a class="headerlink" href="#automatic-relevance-determination-ard" title="Permalink to this headline">¶</a></h2>
<p><tt class="xref py py-class docutils literal"><span class="pre">ARDRegression</span></tt> adds a more sophisticated prior <span class="math">\beta</span>,
where we assume that each weight <span class="math">\beta_{i}</span> is drawn in a
Gaussian distribution, centered on zero and with a precision
<span class="math">\lambda_{i}</span>:</p>
<div class="math">
<p><span class="math">p(\beta|\lambda) = \mathcal{N}(\beta|0,A^{-1})</span></p>
</div><p>with <span class="math">diag \; (A) = \lambda = \{\lambda_{1},...,\lambda_{p}\}</span>.
There is also a Gamma prior for <span class="math">\lambda</span> and <span class="math">\alpha</span>:</p>
<div class="math">
<p><span class="math">g(\alpha|\alpha_1,\alpha_2) = \frac{\alpha_2^{\alpha_1}}
{\Gamma(\alpha_1)} \alpha^{\alpha_1-1} e^{-\alpha_2 {\alpha}}</span></p>
</div><div class="math">
<p><span class="math">g(\lambda|\lambda_1,\lambda_2) = \frac{\lambda_2^{\lambda_1}}
{\Gamma(\lambda_1)} \lambda^{\lambda_1-1} e^{-\lambda_2 {\lambda}}</span></p>
</div><dl class="docutils">
<dt>By default <span class="math">\alpha_1 = \alpha_2 = \lambda_1 = \lambda_2 = 1.e-6</span>, <em>i.e.</em></dt>
<dd>very slightly informative priors.</dd>
</dl>
<div class="figure align-center">
<a class="reference external image-reference" href="../auto_examples/linear_model/plot_ard.html"><img alt="auto_examples/linear_model/images/plot_ard.png" src="auto_examples/linear_model/images/plot_ard.png" /></a>
</div>
<div class="topic">
<p class="topic-title first">Examples:</p>
<ul class="simple">
<li><a class="reference internal" href="../auto_examples/linear_model/plot_ard.html#example-linear-model-plot-ard-py"><em>Automatic Relevance Determination Regression (ARD)</em></a></li>
</ul>
</div>
<div class="section" id="id3">
<h3>3.1.8.1. Mathematical formulation<a class="headerlink" href="#id3" title="Permalink to this headline">¶</a></h3>
<p>A prior is introduced as a distribution <span class="math">p(\theta)</span> over the parameters.
This distribution is set before processing the data. The parameters of a prior
distribution are called <em>hyper-parameters</em>. This description is based on the
Bayes theorem :</p>
<div class="math">
<p><span class="math">p(\theta|\{X,y\})
= \frac{p(\{X,y\}|\theta)p(\theta)}{p(\{X,y\})}</span></p>
</div><dl class="docutils">
<dt>With :</dt>
<dd><ul class="first last simple">
<li><span class="math">p({X, y}|\theta)</span> the likelihood : it expresses how probable it is
to observe <span class="math">{X,y}</span> given <span class="math">\theta</span>.</li>
<li><span class="math">p({X, y})</span> the marginal probability of the data : it can be
considered as a normalizing constant, and is computed by integrating
<span class="math">p({X, y}|\theta)</span> with respect to <span class="math">\theta</span>.</li>
<li><span class="math">p(\theta)</span> the prior over the parameters : it expresses the
knowledge that we can have about <span class="math">\theta</span> before processing the
data.</li>
<li><span class="math">p(\theta|{X, y})</span> the conditional probability (or posterior
probability) : it expresses the uncertainty in <span class="math">\theta</span> after
observing the data.</li>
</ul>
</dd>
</dl>
<p>All the following regressions are based on the following Gaussian
assumption:</p>
<div class="math">
<p><span class="math">p(y|X,w,\alpha) = \mathcal{N}(y|X w,\alpha)</span></p>
</div><p>where <span class="math">\alpha</span> is the precision of the noise.</p>
<div class="topic">
<p class="topic-title first">References</p>
<ul class="simple">
<li>Original Algorithm is detailed in the book <em>Bayesian learning for neural
networks</em> by Radford M. Neal</li>
</ul>
</div>
</div>
</div>
<div class="section" id="stochastic-gradient-descent-sgd">
<h2>3.1.9. Stochastic Gradient Descent - SGD<a class="headerlink" href="#stochastic-gradient-descent-sgd" title="Permalink to this headline">¶</a></h2>
<p>Stochastic gradient descent is a simple yet very efficient approach
to fit linear models. It is particulary useful when the number of samples
(and the number of features) is very large.</p>
<p>The classes <tt class="xref py py-class docutils literal"><span class="pre">SGDClassifier</span></tt> and <tt class="xref py py-class docutils literal"><span class="pre">SGDRegressor</span></tt> provide
functionality to fit linear models for classification and regression
using different (convex) loss functions and different penalties.</p>
<div class="topic">
<p class="topic-title first">References</p>
<ul class="simple">
<li><a class="reference internal" href="sgd.html#sgd"><em>Stochastic Gradient Descent</em></a></li>
</ul>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
</div>
<div class="footer">
<p style="text-align: center">This documentation is relative
to scikits.learn version 0.6.0<p>
© 2010, scikits.learn developers (BSD Lincense).
Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.0.5. Design by <a href="http://webylimonada.com">Web y Limonada</a>.
</div>
</body>
</html>
distrib
>
Mandriva
>
2010.2
>
i586
>
media
>
contrib-backports
>
by-pkgid
>
e578866d55cd81fdb23827cdf3cec911
>
files
>
711
