\documentclass[11pt,english]{article}
\usepackage{mathpazo}
\usepackage[a4paper]{geometry}
\geometry{verbose,tmargin=3cm,bmargin=3cm,lmargin=3cm,rmargin=3cm}
\usepackage{array}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{fancyvrb}
\usepackage{extra}
\usepackage[pdftex,hyperfootnotes=false]{hyperref}

\DefineVerbatimEnvironment%
{code}{Verbatim}
{fontsize=\small, xleftmargin=1em}

\newcommand{\noun}[1]{\textsc{#1}}
\newcommand{\dollar}[1]{\texttt{\$#1}}

\usepackage{babel}

\definecolor{steel}{rgb}{0.03,0.20,0.45}

\hypersetup{pdftitle={The extra package},
            pdfauthor={The gretl team},
            colorlinks=true,
            linkcolor=blue,
            urlcolor=red,
            citecolor=steel,
            bookmarksnumbered=true,
            plainpages=false
          }

\newcommand{\ArgRet}[2]{%
  {\it Arguments}: {#1}%
  \ifx&#2&%
  \else
  \par\smallskip\noindent {\it Return type}: \texttt{#2}
  \fi%
  \par\medskip\par%
  }

\begin{document}

\title{The extra package\\
(a collection of various convenience functions for hansl programming) }

\date{September 2024}

\author{The \noun{gretl} team\thanks{Currently coordinated by Sven
Schreiber.}}

\maketitle
\tableofcontents{}

\section{Usage}

This package is intended for hansl scripting, not for gretl's GUI.
(But of course other contributed function packages that make use of
functions in extra.gfn can provide GUI access for themselves.)

The usual one-time requirement is to do \cmd{pkg install extra.zip}
to get a copy on the local system (or install it via gretl's graphical
mechanism), and then in the respective hansl script have a line \cmd{include
extra.gfn}.

Note that functions that are exact lookalikes of \app{Matlab/Octave}
functions do not live here, but would go into the
\dtk{matlab_utilities} package.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Matrix-related functions}

\subsection{combinations}

\ArgRet{\texttt{matrix x}, \texttt{int h}}{matrix} This function
returns a matrix whose rows are all the possible subsets of $x$
containing $h$ elements; for $h>2$, a recursive algorithm is used.

For example: \texttt{combinations(\{1,2,3\}, 2)} returns
\begin{code}
         1   2
         1   3
         2   3
\end{code}

The argument \texttt{x} must be a (row or column) vector. The returned
matrix will have $n \choose k$ rows if successful, otherwise it will be a 1x1
matrix with an NA value.

\emph{Nota bene}: The recursive algorithm used may be a little slow if
the number of elements of $x$ is large.

\subsection{drawbootres}

\ArgRet{\texttt{const matrix U}, \texttt{bundle bparams} (optional),
  \texttt{int bootcode} (optional)}{matrix} Construct a new draw of
innovations (residuals) for bootstrapping, based on the input matrix
\texttt{U}, interpreted to be T-by-K (observations in rows). The
return value has the same shape. \texttt{U} can be original residuals
or some pre-processed input -- the pre-processing is not done here to
avoid doing it repeatedly during the bootstrap.

Two ways of choosing the bootstrap type are supported: Either by
adding a string element to the \texttt{bparams} bundle under the key
`btypestr', or by using the associated number code in the
\texttt{bootcode} argument, where the bundle-based choice takes
precedence. Both ways are optional, the general default is plain
resampling. Here is a brief description of the meanings of the codes
(see also the SVAR addon's documentation):

\begin{enumerate}
\item plain resampling -- A standard draw with replacement based on
  libgretl's \texttt{resample} function; so if this is all that's
  needed, you might be better off using \texttt{resample}
  directly. (Allowed string code: any word starting with ``re'',
  e.g. ``resampling''.)
\item wild, Normal -- A draw for the wild bootstrapping scheme, where
  each row of the input matrix is scaled randomly according to
  Gaussian noise.  (string code ``wildN'' or simply ``wild'')
\item wild, Rademacher -- Here the signs of all rows are flipped
  randomly.  (string code ``wildR'')
\item wild, Mammen -- Here the multiplicative factors for the rows are
  the ones proposed by Mammen (see the SVAR documenation; string code
  ``wildM'').
\item moving blocks -- Provides a draw of moving blocks of the input,
  see for example Brüggemann, Jentsch \& Trenkler (2016; Journal of
  Econometrics, vol. 191, issue 1, 69-85). By default a block length
  of 10\% of the input length \texttt{T} is used (rounded, at least
  2). The block length can be overridden by supplying the
  `moveblocklen' scalar element inside the \texttt{bparams} bundle
  argument. (Allowed string codes: any word starting with ``mov'', or
  ``MBB''.)

\end{enumerate}

Note that this function is included in extra's version 1.7 and made
public for convenience of potential users, but it may be moved to
another separate addon in the future.


\subsection{drill}

\ArgRet{\texttt{matrices} array, \texttt{matrix rowspec} (optional),
  \texttt{matrix colspec} (optional)}{matrix}

This function ``drills through'' a matrix array and returns a matrix;
if one sees a matrix array whose elements are equally sized as a 3-way
tensor, this function can be used for extracting what are known as
\emph{fibers} and/or \emph{slices} in tensor algebra, that is matrices with
subsets of the tensor entries.\footnote{See eg Kolda, T. G., and
  Bader, B. W. (2009). Tensor decompositions and applications. SIAM
  review, 51(3), 455--500.}

for example, \cmd{drill(x, 2, 3)} returns a vector with the
\texttt{[2,3]} elements of all matrices in the \texttt{x} array (a
``fiber''). Omitting one of rowspec, colspec or entering ``0'' means
to select all rows or columns respectively; the matrix thus obtained
is a ``slice''. Of course, at least one of \texttt{rowspec}a and
\texttt{colspec} must be specified.

\emph{Nota bene}: all matrices in the array must be of the same
dimensions.

\subsection{duplicate}

\ArgRet{\texttt{matrix vechA}}{matrix}

The input is a vector assumed to come from an operation like vech(A).
Returns vec(A), which is the result of pre-multiplying vech(A) with
the ``duplication'' matrix $D_m$. If vechA has several columns, each
column is treated separately as described above (and the results
stacked side-by-side).

\subsection{eliminate}

\ArgRet{\texttt{matrix vecA}}{matrix}

Each column of the input vecA is assumed to come from the operation
vec(A) on a square matrix, thus rows(vecA) must be a square number.
Returns vech(A), which is the result of pre-multiplying vec(A) with
the ``elimination'' matrix $L_m$.  If vecA has several columns, each
column is treated separately as described above (and the results
stacked side-by-side).

\subsection{mat2latex}

\ArgRet{\texttt{matrix X}, \texttt{bundle opts} (optional)}{string}

Produces a string containing the representation of matrix \texttt{X}
as a \LaTeX\ \texttt{tabular} environment. For example,
\begin{code}
eval mat2latex(mshape(seq(1,6), 2, 3))
\end{code}
produces
\begin{code}
\begin{tabular}{lccc}
\hline
 &	  &	 Col 1  &	 Col 2  &	 Col 3 \\ \hline
Row 1 & 1.000	 & 3.000	 & 5.000 \\
Row 2 & 2.000	 & 4.000	 & 6.000 \\
\hline
\end{tabular}
\end{code}
Note that, if a matrix possesses row or column names, they will be
automatically used as labels. Some features of the results can be
tweaked by setting appropriate keys in the \texttt{opts} bundle:
\begin{description}
\item[format]: a string, to be used for aligning columns. The default
  is to have the first column left-aligned, and the subsequent ones
  centered, as in ``\texttt{lccc}''.
\item[decimals] The number of decimals to use (default=3).
\item[nacode] The string to use for missing entries (default: empty).
\item[rnames] A string array for row headings.
\item[cnames] A string array for column headings.
\end{description}

For example, the code
\begin{code}
  open credscore.gdt
  xtab OwnRent Selfempl --quiet
  s = mat2latex($result, _(decimals=0))
  print s
\end{code}
% $
produces
\begin{code}
\begin{tabular}{lccc}
\hline
 &	   0  &	    1  &	 TOTAL \\ \hline
   0 & 60	 & 4	 & 64 \\
   1 & 35	 & 1	 & 36 \\
TOTAL & 95	 & 5	 & 100 \\
\hline
\end{tabular}
\end{code}
which looks, when compiled, as
\begin{center}
\begin{tabular}{lccc}
\hline
 &	   0  &	    1  &	 TOTAL \\ \hline
   0 & 60	 & 4	 & 64 \\
   1 & 35	 & 1	 & 36 \\
TOTAL & 95	 & 5	 & 100 \\
\hline
\end{tabular}
\end{center}

\subsection{nearPSD}

\ArgRet{\texttt{matrix} pointer \texttt{{*}m}, \texttt{scalar epsilon}
(optional)}{scalar}

Forces the matrix $m$ into the positive semi-definite
region. Algorithm ported from ``DomPazz'' in Stackoverflow, apparently
mimicking the nearPD() function in R. Because of re-scaling (to
correlation matrix), the \texttt{epsilon} criterion value should
implicitly apply to the correlation-based eigenvalues. The return
value 0 or 1 indicates whether \texttt{m} was altered or not.

\subsection{qformvech}

\ArgRet{\texttt{matrix Xt}}{matrix} This function relies on the
relation $vech(X'AX) = P (X \otimes X)' Q \; vech(A) = G \; vech(A)$,
where $P$ and $Q$ are certain interim results.  It takes the matrix
$X'$ and returns the matrix $G$ such that the right-hand side of the
equalities becomes feasible, which should be numerically more
efficient than the direct application of the left-hand side.

\subsection{zeroifclose}

\ArgRet{\texttt{matrix} pointer \texttt{{*}m}, \texttt{scalar thresh}
(optional)}{scalar}

Sets elements of \texttt{m} to zero if they are really close. The
return value 0 or 1 indicates whether \texttt{m} was altered or not.

The default value for the threshold has been \texttt{1e-12} since extra
version 0.7. In some applications smaller (in absolute value) but
mathematically truly non-zero results may occur, in which case a smaller
threshold can be chosen.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Other functions working without a dataset in place}

\subsection{get\_settings}

\ArgRet{\texttt{string key} (optional)}{bundle}

The returned bundle contains information on either one or (almost) all
of the gretl state variables that can be configured via the
\texttt{set} command (``libset variables''). The bundle members are
named for the keys of the libset variables and their values are either
strings or scalars.

If no argument is given, all libset variable values are represented,
with the exception of a few that take the form of matrices
(\texttt{initvals}, \texttt{initcurv}), and a few others of no
interest from a programming point of view (\texttt{echo},
\texttt{messages}, \texttt{verbose}). If an argument is given, it
should be the key for a particular libset variable, in which case the
returned bundle has a single element.

\subsection{multi\_instrings}

\ArgRet{\texttt{strings lookinhere}, \texttt{strings tofind}}{matrix}

Returns in a column vector the positions (indices) in `lookinhere'
where any of the strings from `tofind' occur. If there are duplicates
in `tofind' then the output may also contain duplicate indices. Use
\cmd{uniq()} or \cmd{values()} afterwards if needed.

\subsection{onemode}

\ArgRet{\texttt{matrix v}}{matrix}

Finds the mode of the empirical distribution of the input data.  If
the data are multi-modal, details of internal computer arithmetic can
influence which of the modes is actually found. Returns a 2-element
column vector with the modal value and its absolute frequency. If
\texttt{v} is an empty matrix (comprises only nan values) a
$1\times 1$ matrix with nan is returned.

\subsection{powerset}

\ArgRet{\texttt{strings S}}{strings (array)}

Computes the powerset of the input S, i.e. all possible combinations
of the string elements in S. (Including the empty set / empty string
\texttt{""}.) Each combination yields one string in the output
array. Being a set, the ordering is not defined and arbitrary.


\subsection{scores2x2}

\ArgRet{\texttt{matrix in, bool verbose} (optional)}{matrix}

Computes some standard score measures for a $2\times 2$ contingency
table of the form:

\begin{center}
\begin{tabular}{cccc}
\toprule
 &  & \multicolumn{2}{c}{Observed}\\
 &  & 1 & 0\\
\midrule
\multirow{2}{*}{Predicted} & 1 & h(its) & f(alse)\\
 & 0 & m(iss) & z(eros)\\
\bottomrule
\end{tabular}
\end{center}

\noindent and $n=h+f+m+z$ (total observations). Returns a column
vector with the elements listed in Table \ref{tab:scores2x2}. The
input is always sanitized by taking the upper 2x2 part, using absolute
values, and integer-ization. Warnings are issued if \texttt{verbose}
is 1.


\begin{table}[htbp]
\begin{tabular}{rlp{0.35\textwidth}c}
  \hline
  \textbf{Number} &\textbf{Acronym} &  \textbf{Description} &\textbf{Formula} \\
  \hline
  1 & POD & prob of detection & $\frac{h}{h+m}$ \\
2 & POFD & prob of false detection & $\frac{f}{f+z}$ \\
3 & HR & hit rate & $\frac{h+z}{n}$ \\
4 & FAR & false alarm rate & $\frac{f}{h+f}$ \\
5 & CSI & critical success index & $\frac{h}{h+f+m}$\\
6 & OR & odds ratio & $\frac{h \cdot z}{f \cdot m}$\\
7 & BIAS & bias score & $\frac{h+f}{h+m}$\\
8 & TSS & true skill stat ($POD-POFD$); also known as the
          Hanssen-Kuipers score & $\frac{h}{h+m} -\frac{f}{f+z}$. \\
9 & HSS & Heidke skill score & $2 \frac{h \cdot z - f \cdot m}{(h+m) \cdot (m+z)+(h+f) \cdot (f+z)}$ \\
10 & ETS & equitable threat score & $\frac{h \cdot z-f \cdot m}{(f+m) \cdot n+(h \cdot z-f \cdot m)}$ \\
11 & PRC & precision & $\frac{h}{h+f}$ \\
12 & FSC & $F$-Score & $2 \frac{PRC \cdot POD}{PRC+POD} = 2 \frac{h}{1+h+m}$\\ \hline
\end{tabular}
\caption{Elements returned by the \cmd{scores2x2} function}
\label{tab:scores2x2}
\end{table}

\subsection{splitfname}

\ArgRet{\texttt{string fn}}{strings (array)}

The idea is to take a file name or full path and extract three components:
\begin{enumerate}
\item The path prefix (may be empty; without the trailing / or \textbackslash)
\item The ``base'' component of the file name, without the extension
  and without the path prefix
\item The file extension (without the dot; may be empty)
\end{enumerate}
In principle this should work with both forward slashes and backslashes, and also
with doubled slashes.

Example:

Input string: \cmd{"/what/on/earth/isthisfile.gdt"}

Output equivalent to:

\cmd{defarray("/what/on/earth", "isthisfile", "gdt")}


\subsection{truncnorm}

\ArgRet{\texttt{int n, scalar m, scalar sigma, scalar below, scalar
above}}{matrix}

Generates $n$ truncated normal random values. Specify mean \texttt{m}
and standard deviation \texttt{sigma}, and the left/right truncation values
\texttt{below} and \texttt{above}. (Pass NA for any one of them to
skip the respective truncation.) Returns a column vector of values.


\subsection{WSRcritical}

\ArgRet{\texttt{int n, scalar prob }(optional)\texttt{, bool forcenorm}
(optional)}{matrix}

Concerns the distribution of Wilcoxon's signed rank test statistic for
\texttt{n} trials (at least 4). Tries to find the critical values
(low/hi) where the two-sided area to the outside is as close as
possible to the given \texttt{prob} (default: 0.05). (Note that
``outside'' means including the critical values themselves in the
exact/discrete case.) If we end up in the interior region not covered
by the exact table (for \texttt{prob} far away from 0 and also from
1), we fall back to the normal approximation. The function returns a
column vector \verb|{lo; hi; epv}|, where \texttt{epv} is the actual
probability mass (close to \texttt{prob} but not equal in general for
small samples). \texttt{lo} and \texttt{hi} can be non-integers in the
normal approximation case. The normal approximation instead of the
exact table values can be enforced with the \texttt{forcenorm}
argument (default: zero, do not enforce).

See also the sister function \cmd{WSRpvalue}.

\subsection{WSRpvalue}

\ArgRet{\texttt{int n, scalar W, bool forcenorm} (optional)}{scalar}

Concerns the distribution of Wilcoxon's signed rank test statistic for
\texttt{n} trials (at least 4), returns $P(X\geq W)$. In the interior
region not covered by the exact table, the true value is $\geq$ 12.5\%
(and $\leq$87.5\%) according to the table used,\footnote{Source of the
  table: Wilfrid J Dixon and Frank J. Massey, Jr., Introduction to
  Statistical Analysis, 2nd ed. (New York: McGraw-Hill, 1957), pp.
  443-444.} so typically based on such values H0 would not be
rejected. We fall back to the normal approximation in this region. In
the extreme outer regions not explicitly covered by the table, the
deviation from 0 or 1 will be smaller than 0.5\% = 0.005. We return
values 0.001 or 0.999 as an approximation here. The test statistic
\texttt{W} should usually be an integer, but in case of bindings it
could be fractional as well; in this case we also fall back to the
normal approximation.

The normal approximation instead of the exact table values can be
enforced with the \texttt{forcenorm} argument (default: zero, do not
enforce).

See also the sister function \cmd{WSRcritical}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Functions requiring a dataset}

\subsection{combine}

\ArgRet{\texttt{series a, series b}}{series}

This function takes as arguments two discrete series and computes all
the combinations of their values that occur in the selected sample of
the currently open dataset. These are stored in the resulting series.

For example, suppose you have a dataset of trade flows, with two
series \texttt{ic} and \texttt{ec} for the importing and exporting
countries, respectively. Then \cmd{combine(ic,ec)} will generate a
series in which each pair has a distinct encoding.

If the two input series are both string-valued, then the output series
will also be string-valued, as long as it's possible to assign unique
labels to each value.

\subsection{correspondence}

\ArgRet{\texttt{series a, series b}}{scalar}

This function takes two series and establishes if there's a 1-to-1
relationship between them, in which case it returns 2.  If there's a
1-to-n relationship such that \texttt{a} could be interpreted as a
(mathematical) function of \texttt{b}, it returns 1. If there's no
relationship -- for example several different values of series
\texttt{a} appear together with some value of \texttt{b} -- it returns
0.

One of the possible use cases is to check whether two discrete series encode
the same variable. For example, the code:
\begin{code}
  open grunfeld.gdt
  c = correspondence($unit, firm)
\end{code}
% $
sets \texttt{c} to 2, indicating that the variable \texttt{firm} is in fact the
panel cross-sectional identifier.

\subsection{fracorder}

\ArgRet{\texttt{series x, int order} (optional), \texttt{bool verbosity}
 (optional)}{matrix}

Meta function to invoke all the various ways in gretl to estimate the
order of fractional integration of the input series, namely the Local
Whittle estimator, the one by Geweke \& Porter-Hudak (GPH), and the
Hurst exponent minus $0.5$.  The first two are executed through
gretl's command \texttt{fractint}, the latter via
\texttt{hurst}.\footnote{Another estimation approach for the Hurst
  exponent is provided in the user-contributed function package
  \textsf{gen\_hurst}.}
% This function only works with gretl 2020c or higher.

Returns a matrix with three rows corresponding to the methods above;
the four columns contain (1) the point estimate, (2) its standard
error, (3) the test statistic for the null hypothesis of integration
order zero, (4) the associated p-value. For example, to obtain the
standard error of the Local Whittle estimator one picks the
1,2-element of the output matrix. The optional `verbosity' switch is
set to 0 (OFF) by default, otherwise the standard output of the
underlying commands is printed out.

The optional `order' argument only applies to the Local Whittle and GPH
estimators and overrides gretl's default lag order of $\min(T/2, T^{0.6})$.

For the Hurst method a minimum of 128 observations is required, and
test results are never available. Also note that by construction this
estimator can only take values between $-0.5$ and $0.5$.

\subsection{gap\_filler}

\ArgRet{\texttt{series x, int method }(optional)}{series}

Simple convenience function to crudely get rid of missing values
interspersed between valid observations. The function is meant to be
used with time series, or panel datasets with a time dimension. An
error is returned if the function is used with a cross-sectional
dataset.

Apart from the first argument (series), it accepts an integer
parameter as second argument, whose meaning is: 0: do nothing, leave
the gaps; 1: NAs are replaced with previous observations; 2: NAs are
replaced with a linear interpolation (this uses the internal function
\texttt{interpol()}). Returns the filled series.

The very existence of the ``0'' method for interpolation may look
bizarre at first sight, but it may make sense in the context of batch
processing, as in the following example (hopefully, self-explanatory):
\begin{code}
k = 1
loop foreach i X
   series z_$i = gap_filler($i, action[k++])
endloop
\end{code}

Note that the function only replaces NAs between valid observations;
therefore, if the origin series has missing values at the beginning or
the end of the sample, they will be in the returned series too.

\subsection{winsor}

\ArgRet{\texttt{series x, scalar p} (optional), \texttt{scalar
phi} (optional)}{series}

Returns a trimmed (``winsorized'') version
of the series, where outliers are replaced with implicit threshold
values. Truncation quantiles are determined according to relative
tail frequencies \texttt{p} and \texttt{phi}. Default lower and upper
frequencies are 0.05, but re-settable with \texttt{p}. Pass \texttt{phi}
in addition to \texttt{p} for an asymmetric trimming, then \texttt{p}
determines only the lower frequency and \texttt{phi} the upper.

\section{Authors}
\begin{itemize}
\item gap\_filler, eliminate, duplicate, truncnorm, powerset, drill,
  correspondence, combinations, qformvech, mat2latex, combine: Jack
  Lucchetti

\item nearPSD, zeroifclose, scores2x2, WSRcritical, WSRpvalue, onemode,
splitfname, multi\_instrings, fracorder, put\_outofsmpl:
Sven Schreiber

\item winsor: Sven Schreiber, original code JoshuaHe

\item drawbootres: Jack Lucchetti and Sven Schreiber

\end{itemize}

\section{Changelog }
\begin{itemize}
\item March 2024: add the get\_settings function
\item September 2023: add the combine function
\item July 2023: adopt the drawbootres function from the SVAR addon
  to make it publicly available
\item January 2023: introduce the put\_outofsmpl function; internal
  refactoring of the mat2latex() function
\item July 2022: introduce the mat2latex function
\item June 2022: retire the commute function (now in libgretl)
\item January 2022: retire the mat2list function (now in
  libgretl), update gap\_filler
\item October 2021: add qformvech
\item June 2021: add combinations, and increase gretl version requirement
  to 2020c
\item November 2020: add mat2list
\item September 2020: add fracorder, remove bwritejson
\item July 2020: add multi\_instrings and correspondence, add deprecation
  warning to bwritejson, efficiency improvement for zeroifclose
\item January 2020: add drill, bwritejson, onemode, splitfname;
  finally remove the retired sepstr2arr (use native strsplit instead);
  slightly revise gap\_filler; rearrange the documentation a little
\item October 2018: fix small commute bug; retire sepstr2arr; add
  powerset, eliminate, duplicate
\item February 2018: allow non-integer input in WSRpvalue
\item January 2018: add WSRcritical, WSRpvalue
\item December 2017: add scores2x2; switch to pdf help document
\item September 2017: add winsor
\item July 2017: initial release
\end{itemize}

\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:
