\chapter{Gretl and Python}
\label{chap:gretlPython}

\section{Introduction}
\label{Python-intro}

According to \url{www.python.org}, \app{Python} is ``an easy to learn,
powerful programming language. It has efficient high-level data
structures and a simple but effective approach to object-oriented
programming. Python's elegant syntax and dynamic typing, together with
its interpreted nature, make it an ideal language for scripting and
rapid application development in many areas on most platforms.''

Indeed, \app{Python} is widely used in a great variety of
contexts. Numerous add-on modules are available; the ones likely to be
of greatest interest to econometricians include \app{pandas} for data
processing, \app{NumPy} (``the fundamental package for scientific
computing with Python''---see \url{www.numpy.org}); \app{SciPy} (which
builds on \app{NumPy}---see \url{www.scipy.org}); and
\app{Statsmodels} (\url{http://statsmodels.sourceforge.net/}). A
special mention goes to \app{gretl4py}, which makes many gretl methods
available in Python, and can be seen as a mirror feature to the one
we're describing here.

\section{Python support in gretl}
\label{sec:Python-support}

The support offered for \app{Python} in gretl is similar to that
offered for \app{Octave} (chapter~\ref{chap:gretlOctave}). You can
open and edit \app{Python} scripts in the gretl GUI.  Clicking
the ``execute'' icon in the editor window will send your code to
\app{Python} for execution. In addition you can embed \app{Python}
code within a gretl script using a \texttt{foreign} block, as
described in connection with \app{R}.

When you launch \app{Python} from within gretl one variable and
two convenience functions are pre-defined, as follows.
\begin{code}
gretl_dotdir
gretl_loadmat(filename, autodot=1)
gretl_export(M, filename, autodot=1)
\end{code}
The variable \verb|gretl_dotdir| holds the path to the user's ``dot
directory.''  The first function loads a matrix of the given
\texttt{filename} as written by gretl's \texttt{mwrite}
function, and the second writes matrix \texttt{M}, under the given
\texttt{filename}, in the format wanted by gretl.

Additionally, the \option{send-data} option makes the currently open
dataset available to the Python environment as a Pandas dataframe,
under the name \texttt{gretldata}. A short example is provided below,
in section \ref{sec:Python-send-data}.

By default the traffic in matrices goes via the dot directory on the
\app{Python} side; that is, the name of this directory is prepended to
\texttt{filename} for both reading and writing. (This is complementary
to use of the \textsl{export} and \textsl{import} parameters with
gretl's \texttt{mwrite} and \texttt{mread} functions, respectively.)
However, if you wish to take control over the reading and writing
locations you can supply a zero value for \texttt{autodot} (or give an
absolute path) when calling \verb|gretl_loadmat| and
\verb|gretl_export|: in that case the \texttt{filename} argument is
used as is.

Note that \verb|gretl_loadmat| and \verb|gretl_export| depend on
\app{NumPy}; they make use of the functions \texttt{loadtxt} and
\texttt{savetxt} respectively. Nonetheless, the presence of
\app{NumPy} is not an absolute requirement if you don't need
to use these two functions.

\section{Illustration 1: linear regression with multicollinearity}
\label{sec:Python-longley}

Listing~\ref{ex:python-longley} compares the numerical accuracy of
gretl's \texttt{ols} command with that of the function
\texttt{linalg.lstsq} in \app{NumPy}, using the notorious Longley test
data which exhibit extreme multicollinearity.  Unlike some
econometrics packages, \app{NumPy} does a good job on these data. The
script computes and prints the log-relative error in estimation of the
regression coefficients, using the NIST-certified values as a
benchmark;\footnote{See
  \url{http://www.itl.nist.gov/div898/strd/lls/data/Longley.shtml}.}
the error values correspond to the number of correct digits (with a
maximum of 15). The results will likely differ somewhat by computer
architecture and compiler.

\begin{script}[htbp]
  \scriptinfo{python-longley}{Comparing regression results with \app{Python}}
\begin{scode}
set verbose off

function matrix logrel_err (const matrix est, const matrix true)
  return -log10(abs(est - true) ./ abs(true))
end function

open longley.gdt -q
list LX = prdefl .. year
ols employ 0 LX -q
matrix b_gretl = $coeff

mwrite({employ} ~ {const} ~ {LX}, "alldata.mat", 1)

foreign language=python
   import numpy as np
   X = gretl_loadmat('alldata.mat', 1)
   # NumPy's OLS   
   b = np.linalg.lstsq(X[:,1:], X[:,0])[0]
   gretl_export(np.transpose(np.matrix(b)), 'py_b.mat', 1)
end foreign

# NIST's certified coefficient values
matrix b_nist = {-3482258.63459582, 15.0618722713733,
    -0.358191792925910E-01, -2.02022980381683,
    -1.03322686717359, -0.511041056535807E-01,
     1829.15146461355}'

matrix b_numpy = mread("py_b.mat", 1)
matrix E = logrel_err(b_gretl, b_nist) ~ logrel_err(b_numpy, b_nist)
cnameset(E, "gretl python")
printf "Log-relative errors, Longley coefficients:\n\n%#12.5g\n", E
printf "Column means\n%#12.5g\n", meanc(E)
\end{scode}
  Output:
\begin{outbit}
Log-relative errors, Longley coefficients:

       gretl      python
      12.844      12.850
      11.528      11.414
      12.393      12.401
      13.135      13.121
      13.738      13.318
      12.587      12.363
      12.848      12.852

Column means
      12.725      12.617
\end{outbit}
\end{script}

\section{Illustration 2: passing data via Pandas}
\label{sec:Python-send-data}

In this example, we assume that the user's Python environment has the
\app{statsmodels} module available, and we'll use the data from
\cite{mroz87} for estimating a simple probit model on women's labor
force participation. The dataset is available in gretl as an example
dataset as \texttt{mroz87.gdt}.

After having loaded the dataset, the probit model is estimated in
gretl using the native \cmd{probit} command. The data is then passed
to the Python environment, where \app{statsmodels} is used to estimate
the same model.

\begin{script}[htbp]
  \scriptinfo{python-probit}{Comparing probit results with \app{Python}}
\begin{scode}
set verbose off

open mroz87 -q
list LX = const WA WE HA HE
probit LFP LX --p-values

foreign language=python --send-data
	import statsmodels.formula.api as sf;
	f = 'LFP ~ WA + WE + HA + HE';
	model = sf.probit(formula=f, data=gretldata).fit();
	print(model.summary());
end foreign
\end{scode}
  Output:
\begin{outbit}
Model 1: Probit, using observations 1-753
Dependent variable: LFP
Standard errors based on Hessian

             coefficient    std. error      z       p-value 
  ----------------------------------------------------------
  const      -0.485099      0.407364     -1.191     0.2337  
  WA         -0.0105331     0.0125299    -0.8406    0.4006  
  WE          0.145865      0.0265928     5.485     4.13e-08 ***
  HA         -0.000646206   0.0126691    -0.05101   0.9593  
  HE         -0.0520389     0.0198418    -2.623     0.0087   ***

Mean dependent var   0.568393   S.D. dependent var   0.495630
McFadden R-squared   0.035600   Adjusted R-squared   0.025889
Log-likelihood      -496.5435   Akaike criterion     1003.087
Schwarz criterion    1026.207   Hannan-Quinn         1011.994

[... other output lines omitted ...]
  
                          Probit Regression Results                           
==============================================================================
Dep. Variable:                    LFP   No. Observations:                  753
Model:                         Probit   Df Residuals:                      748
Method:                           MLE   Df Model:                            4
Date:                Thu, 26 Mar 2026   Pseudo R-squ.:                 0.03560
Time:                        11:25:59   Log-Likelihood:                -496.54
converged:                       True   LL-Null:                       -514.87
Covariance Type:            nonrobust   LLR p-value:                 2.117e-07
==============================================================================
                 coef    std err          z      P>|z|      [0.025      0.975]
------------------------------------------------------------------------------
Intercept     -0.4851      0.407     -1.191      0.234      -1.284       0.313
WA            -0.0105      0.013     -0.841      0.401      -0.035       0.014
WE             0.1459      0.027      5.485      0.000       0.094       0.198
HA            -0.0006      0.013     -0.051      0.959      -0.025       0.024
HE            -0.0520      0.020     -2.623      0.009      -0.091      -0.013
==============================================================================
\end{outbit}
\end{script}

%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "gretl-guide"
%%% End: 

