\chapter{Special functions in \cmd{genr}}
\label{chap:genr}

\section{Introduction}
\label{genr-intro}

The \cmd{genr} command provides a flexible means of defining new
variables. At the same time, the somewhat paradoxical situation is
that the ``genr'' keyword is almost never visible in gretl scripts. 
For example, it is not really recommended to write a line such as 
\texttt{genr b = 2.5}, because there are the following alternatives:

\begin{itemize}
\item \texttt{scalar b = 2.5}, which also invokes the genr apparatus
  in gretl, but provides explicit type information about the variable
  \texttt{b}, which is usually preferable. (gretl's language hansl is
  statically typed, so \texttt{b} cannot switch from scalar to string
  or matrix, for example.)
  
\item \texttt{b = 2.5}, leaving it to gretl to infer the admissible or
  most ``natural'' type for the new object, which would again be a
  scalar in this case.

\item \texttt{matrix b = \{2.5\}}: This formulation is required if
  \texttt{b} is going to be expanded with additional rows or columns
  later on. Otherwise, gretl's static typing would not allow
  \texttt{b} to be promoted from scalar to matrix, so it must be a
  matrix right from the start, even if it is of dimension $1 \times 1$
  initially.  (This definition could also be written as \texttt{matrix
    b = 2.5}, but the more explicit form is recommended.)
\end{itemize}

In addition to \cmd{scalar} or \cmd{matrix}, other type keywords that
can be used to substitute the generic genr term are those enumerated
in the following chapter \ref{chap:datatypes}. In the case of an array
the concrete specification should be used, so one of \cmd{matrices}, 
\cmd{strings}, \cmd{lists}, \cmd{bundles}.\footnote{A recently added 
advanced datatype is an array of arrays, with the associated type 
specifier \cmd{arrays}.}

Therefore, there's only a handful of special cases where it is really
necessary to use the ``genr'' keyword:

\begin{itemize}
\item \texttt{genr time} --- Creates a time trend variable
  (1,2,3,\ldots) under the name \texttt{time}. Note that within an
  appropriately defined panel dataset this variable honors the panel
  structure and is a true time index. (In a cross-sectional dataset,
  the command will still work and produces the same result as
  \texttt{genr index} below, but of course no temporal meaning
  exists.)

\item \texttt{genr index} --- Creates an observation variable named
  \texttt{index}, running from 1 to the sample size.

\item \texttt{genr unitdum} --- In the context of panel data, creates
  a set of dummies for the panel groups or ``units''. These are named
  \dtk{du_1}, \dtk{du_2}, and so forth. Actually, this
  particular genr usage is not strictly necessary, because a list of
  group dummies can also be obtained as:

\begin{code}
series gr = $unit
list groupdums = dummify(gr, NA)
\end{code} 

  (The \texttt{NA} argument to the dummify function has the effect of not
  skipping any unit as the reference group, thus producing the full
  set of dummies.)

\item \texttt{genr timedum} --- Again for panel data, creates a set of
  dummies for the time periods, named \dtk{dt_1}, \dtk{dt_2},
  \ldots. And again, a list-producing variant without genr exists,
  using the special accessor \dollar{obsminor} which indexes time in
  the panel context and can be used as a substitute for \texttt{time} from
  above:

\begin{code}
series tindex = $obsminor
list timedums = dummify(tindex, NA)
\end{code} 

\item \texttt{genr markers} --- See section \ref{sec:more-panel} for 
  an explanation and example of this panel-related feature.
  
\end{itemize}

Finally, there also exists \texttt{genr dummy}, which produces a set
of seasonal dummies. However, it is recommended to use the
\texttt{seasonals()} function instead, which can also return centered
dummies.

The rest of this chapter discusses other special function aspects.

\section{Cumulative densities and p-values}
\label{sec:genr-cdf}

The two functions \cmd{cdf} and \cmd{pvalue} provide complementary
means of examining values from 17 probability distributions (as of 
July 2021), among which the most important ones:
standard normal, Student's $t$, $\chi^2$, $F$, gamma, and binomial.
The syntax of these functions is set out in the \GCR; here we expand
on some subtleties.

The cumulative density function or CDF for a random variable
is the integral of the variable's density from its lower limit
(typically either $-\infty$ or 0) to any specified value $x$.  The
p-value (at least the one-tailed, right-hand p-value as returned by
the \cmd{pvalue} function) is the complementary probability, the
integral from $x$ to the upper limit of the distribution, typically
$+\infty$.

In principle, therefore, there is no need for two distinct functions:
given a CDF value $p_0$ you could easily find the corresponding
p-value as $1-p_0$ (or vice versa).  In practice, with
finite-precision computer arithmetic, the two functions are not
redundant.  This requires a little explanation.  In gretl, as in
most statistical programs, floating point numbers are represented as
``doubles'' --- double-precision values that typically have a storage
size of eight bytes or 64 bits.  Since there are only so many bits
available, only so many floating-point numbers can be represented:
\textit{doubles do not model the real line}.  Typically doubles can
represent numbers over the range (roughly) $\pm 1.7977 \times
10^{308}$, but only to about 15 digits of precision.

Suppose you're interested in the left tail of the $\chi^2$ distribution
with 50 degrees of freedom: you'd like to know the CDF value for $x =
0.9$.  Take a look at the following interactive session:
\begin{code}
? scalar p1 = cdf(X, 50, 0.9)
Generated scalar p1 = 8.94977e-35
? scalar p2 = pvalue(X, 50, 0.9)
Generated scalar p2 = 1
? scalar test = 1 - p2
Generated scalar test = 0
\end{code}

The \cmd{cdf} function has produced an accurate value, but the
\cmd{pvalue} function gives an answer of 1, from which it is not
possible to retrieve the answer to the CDF question.  This may seem
surprising at first, but consider: if the value of \texttt{p1} above
is correct, then the correct value for \texttt{p2} is $1 - 8.94977
\times 10^{-35}$.  But there's no way that value can be represented as
a double: that would require over 30 digits of precision.

Of course this is an extreme example.  If the $x$ in question is not
too far off into one or other tail of the distribution, the \cmd{cdf}
and \cmd{pvalue} functions will in fact produce complementary
answers, as shown below:
\begin{code}
? scalar p1 = cdf(X, 50, 30)
Generated scalar p1 = 0.0111648
? scalar p2 = pvalue(X, 50, 30)
Generated scalar p2 = 0.988835
? scalar test = 1 - p2
Generated scalar test = 0.0111648
\end{code}
But the moral is that if you want to examine extreme values
you should be careful in selecting the function you need, in the
knowledge that values very close to zero can be represented as doubles
while values very close to 1 cannot.


\section{Retrieving internal variables (dollar accessors)}
\label{sec:genr-internal}

A very useful feature is to retrieve in a script various values
calculated by gretl in the course of estimating models or testing
hypotheses. Since they all start with a literal \$ character, they
are called ``dollar accessors''. The variables that can be retrieved
in this way are listed in the \GCR or in the built-in function help
under the Help menu. The dollar accessors can be used like other
gretl objects in script assignments or statements. Some of those
accessors are actually independent of any estimation or test and
describe, for example, the context of the running gretl program. 
But here we say a bit more about the special variables
\dollar{test} and \dollar{pvalue}.

These variables hold, respectively, the value of the last test
statistic calculated using an explicit testing command and the p-value
for that test statistic.  If no such test has been performed at the
time when these variables are referenced, they will produce the
missing value code.  Some ``explicit testing commands'' that work in
this way are as follows (among others): \cmd{add} (joint test for the
significance of variables added to a model); \cmd{adf} (Augmented
Dickey--Fuller test, see below); \cmd{arch} (test for ARCH);
\cmd{chow} (Chow test for a structural break); \cmd{coeffsum} (test
for the sum of specified coefficients); \cmd{coint} (Engle-Granger
cointegration test); \cmd{cusum} (the Harvey--Collier $t$-statistic);
\cmd{difftest} (test for a difference of two groups); \cmd{kpss} (KPSS
stationarity test, no p-value available); \cmd{modtest} (see below);
\cmd{meantest} (test for difference of means); \cmd{omit} (joint test
for the significance of variables omitted from a model); \cmd{reset}
(Ramsey's RESET); \cmd{restrict} (general linear restriction);
\cmd{runs} (runs test for randomness); and \cmd{vartest} (test for
difference of variances). In most cases both a \dollar{test} and a
\dollar{pvalue} are stored; the exception is the KPSS test, for which
a p-value is not currently available.

The \cmd{modtest} command (which must follow an estimation
command) offers several diagnostic tests; the particular test
performed depends on the option flag provided. Please see the \GCR\
and for example chapters~\ref{chap:var} and \ref{chap:timeseries} of
this \textit{Guide} for details.

An important point to notice about this mechanism is that the internal
variables \dollar{test} and \dollar{pvalue} are over-written each time
one of the tests listed above is performed.  If you want to reference
these values, you must do so at the correct point in the sequence of
gretl commands.


%%% Local Variables:
%%% mode: latex
%%% TeX-master: "gretl-guide"
%%% End:
