\chapter{Named lists and strings}
\label{chap:lists-strings}


\section{Named lists}
\label{named-lists}

Many gretl commands take one or more lists of series as
arguments.  To make this easier to handle in the context of command
scripts, and in particular within user-defined functions, gretl
offers the possibility of \textit{named lists}.  

\subsection{Creating and modifying named lists}

A named list is created using the keyword \texttt{list}, followed by
the name of the list, an equals sign, and an expression that forms a
list.  The most basic sort of expression that works in this context
is a space-separated list of variables, given either by name or
by ID number.  For example,
%
\begin{code}
list xlist = 1 2 3 4
list reglist = income price 
\end{code}

Note that the variables in question must be of the series type.

Two abbreviations are available in defining lists:
\begin{itemize}
\item You can use the wildcard character, ``\texttt{*}'', to create a
  list of variables by name. For example, \texttt{dum*} can be used to
  indicate all variables whose names begin with \texttt{dum}.
\item You can use two dots to indicate a range of variables. For
  example \texttt{income..price} indicates the set of variables whose
  ID numbers are greater than or equal to that of \texttt{income}
  and less than or equal to that of \texttt{price}.
\end{itemize}

In addition there are two special forms:
\begin{itemize}
\item If you use the keyword \texttt{null} on the right-hand side,
  you get an empty list.
\item If you use the keyword \texttt{dataset} on the right, you get
  a list containing all the series in the current dataset (except
  the pre-defined \texttt{const}).
\end{itemize}

The name of the list must start with a letter, and must be composed
entirely of letters, numbers or the underscore character.  The maximum
length of the name is 31 characters; list names cannot contain
spaces.  

Once a named list has been created, it will be ``remembered'' for the
duration of the gretl session (unless you delete it), and can be
used in the context of any gretl command where a list of
variables is expected.  One simple example is the specification of a
list of regressors:
%
\begin{code}
list xlist = x1 x2 x3 x4
ols y 0 xlist
\end{code}

To get rid of a list, you use the following syntax:
\begin{code}
  list xlist delete
\end{code}
Be careful: \texttt{delete xlist} will delete the series contained
in the list, so it implies data loss (which may not be what you want).
On the other hand, \texttt{list xlist delete} will simply ``undefine''
the \texttt{xlist} identifier; the series themselves will not be
affected.

Similarly, to print the names of the members of a list you have to
invert the usual print command, as in
\begin{code}
  list xlist print
\end{code}
If you just say \texttt{print xlist} the list will be expanded and
the values of all the member series will be printed.

Lists can be modified in various ways.  To \textit{redefine} an existing
list altogether, use the same syntax as for creating a list.  For
example
%
\begin{code}
list xlist = 1 2 3
xlist = 4 5 6
\end{code}

After the second assignment, \texttt{xlist} contains just variables 4,
5 and 6.

To \textit{append} or \textit{prepend} variables to an existing list,
we can make use of the fact that a named list stands in for a
``longhand'' list.  For example, we can do
%
\begin{code}
list xlist = xlist 5 6 7
xlist = 9 10 xlist 11 12
\end{code}
%
Another option for appending a term (or a list) to an existing list is
to use \texttt{+=}, as in
%
\begin{code}
xlist += cpi
\end{code}
%
To drop a variable from a list, use \texttt{-=}:
%
\begin{code}
xlist -= cpi
\end{code}
%

In most contexts where lists are used in gretl, it is expected
that they do not contain any duplicated elements.  If you form a new
list by simple concatenation, as in \texttt{list L3 = L1 L2}
(where \texttt{L1} and \texttt{L2} are existing lists), it's possible
that the result may contain duplicates.  To guard against this you can
form a new list as the union of two existing ones:
%
\begin{code}
list L3 = L1 || L2
\end{code}
%
The result is a list that contains all the members of \texttt{L1},
plus any members of \texttt{L2} that are not already in \texttt{L1}.

In the same vein, you can construct a new list as the intersection of
two existing ones:
%
\begin{code}
list L3 = L1 && L2
\end{code}
%
Here \texttt{L3} contains all the elements that are present in both
\texttt{L1} and \texttt{L2}.

You can also subtract one list from another:
%
\begin{code}
list L3 = L1 - L2
\end{code}
%
The result contains all the elements of \texttt{L1} that are not 
present in \texttt{L2}.

Indexing into a defined list is also possible, as if it were a vector:
% 
\begin{code}
list L2 = L1[1:4]
\end{code}
%
This leaves L2 with the first four members of \texttt{L1}. Notice that
the ordering of list members is path-dependent. 

\subsection{Lists and matrices}

There are two ways one can think of lists and matrices being
interchangeable: either you think of a list as a collection of
references to series, or you may consider the rectangle of data given
by the series that the list contains.

In the former case, a list may be translated into (or created from) a
one-dimensional matrix, that is a vector. Therefore, the
matrix in question must be interpretable as a vector containing ID
numbers of data series.  It may be either a row or a column
vector, and each of its elements must have an integer part that is
no greater than the number of variables in the data set.  For example:
%
\begin{code}
matrix m = {1,2,3,4}
list L = m
\end{code}
%
The above is OK provided the data set contains at least 4
variables. Conversely, the command
%
\begin{code}
matrix m = L
\end{code}
%
will create a row vector with the ID numbers of the series referenced
by \texttt{L}.

The latter case occurs when the matrix is assumed to contain valid
data. To create a matrix from the list, simply assing to a matrix the
list name surrounded by curly brackets, as in
%
\begin{code}
matrix m = { L }
\end{code}
%
Note the difference with the above: without the curly brackets, matrix
\texttt{m} would have been just a vector. Also note that any row
corresponding to one or more missing entries will be dropped, unless
the \dtk{skip_missing} set variable is set to \texttt{on}.

For the reverse operation, gretl provides the \cmd{mat2list} function,
which takes a matrix (say, \texttt{X}) as argument and creates new
series as well as a list containing them.  The row dimension of
\texttt{X} must equal either the length of the current dataset or the
number of observations in the current sample range.

The naming of the series in the returned list proceeds as
follows. First, if the optional prefix argument is supplied, the
series created from column $i$ of \texttt{X} is named by appending
\texttt{i} to the given string. Otherwise, if \texttt{X} has column
names set these names are used. Finally, if neither of the above
conditions is satisfied, the names are \texttt{column1},
\texttt{column2} and so on.

For example,
\begin{code}
matrix X = mnormal($nobs, 8)
list L = mat2list(X, "xnorm")
\end{code}
% $
will add to the dataset eight full-length series named
\texttt{xnorm1}, \texttt{xnorm2} and so on.

\subsection{Querying a list}

You can determine the number of variables or elements in a list
using the function \texttt{nelem()}.
%
\begin{code}
list xlist = 1 2 3
nl = nelem(xlist)
\end{code}

The (scalar) variable \texttt{nl} will be assigned a value of 3 since
\texttt{xlist} contains 3 members.

You can determine whether a given series is a member of a specified
list using the function \texttt{inlist()}, as in
%
\begin{code}
scalar k = inlist(L, y)
\end{code}
%
where \texttt{L} is a list and \texttt{y} a series. The series may
be specified by name or ID number. The return value is the (1-based)
position of the series in the list, or zero if the series is not
present in the list. 

\subsection{Generating lists of transformed variables}
\label{sec:transform-lists}

Given a named list of series, you are able to generate lists of
transformations of these series using the functions \texttt{log},
\texttt{lags}, \texttt{diff}, \texttt{ldiff}, \texttt{sdiff} or
\texttt{dummify}.  For example
%
\begin{code}
list xlist = x1 x2 x3
list lxlist = log(xlist)
list difflist = diff(xlist)
\end{code}

When generating a list of \textit{lags} in this way, you specify the
maximum lag order inside the parentheses, before the list name and
separated by a comma.  For example
%
\begin{code}
list xlist = x1 x2 x3
list laglist = lags(2, xlist)
\end{code}
%
or
%
\begin{code}
scalar order = 4
list laglist = lags(order, xlist)
\end{code}

These commands will populate \texttt{laglist} with the specified
number of lags of the variables in \texttt{xlist}.  You can give the
name of a single series in place of a list as the second argument to
\texttt{lags}: this is equivalent to giving a list with just one
member.

The \texttt{dummify} function creates a set of dummy variables coding
for all but one of the distinct values taken on by the original
variable, which should be discrete.  (The smallest value is taken as
the omitted catgory.)  Like lags, this function returns a list even if
the input is a single series.

Another useful operation you can perform with lists is creating
\emph{interaction} variables. Suppose you have a discrete variable
$x_i$, taking values from $1$ to $n$ and a variable $z_i$, which could
be continuous or discrete. In many cases, you want to ``split'' $z_i$
into a set of $n$ variables via the rule
\[
z^{(j)}_i =\left\{ 
    \begin{array}{ll}
      z_i & \mathrm{when} \quad x_i = j \\
      0 & \mathrm{otherwise;}
    \end{array}
    \right. 
\] 
in practice, you create dummies for the $x_i$ variable first and then
you multiply them all by $z_i$; these are commonly called the
\emph{interactions} between $x_i$ and $z_i$. In gretl you can do 
\begin{code}
  list H = D ^ Z
\end{code}
where \texttt{D} is a list of discrete series (or a single discrete
series), \texttt{Z} is a list (or a single series)\footnote{
  This construct does \emph{not} work if neither \texttt{D} nor
  \texttt{Z} are of the list type. However, a simple workaround is
  to turn \texttt{D} into a one-element list in this case, by writing
  \dtk{deflist(D) ^ Z}.}; all the interactions will be
created and listed together under the name \texttt{H}.

An example is provided in Listing~\ref{ex:interaction-lists}.

\begin{script}[ht]
  \scriptinfo{interaction-lists}{Usage of interaction lists}
Input:
\begin{scode}
open mroz87.gdt --quiet

# the coding below makes it so that
# KIDS = 0 -> no kids
# KIDS = 1 -> young kids only
# KIDS = 2 -> young or older kids

series KIDS = (KL6 > 0) + ((KL6 > 0) || (K618 > 0))

list D = CIT KIDS # interaction discrete variables 
list X = WE WA    # variables to "split"
list INTER = D ^ X

smpl 1 6

print D X INTER -o
\end{scode}

Output (selected portions):
\begin{outbit}
           CIT         KIDS           WE           WA     WE_CIT_0

1            0            2           12           32           12
2            1            1           12           30            0
3            0            2           12           35           12
4            0            1           12           34           12
5            1            2           14           31            0
6            1            0           12           54            0

      WE_CIT_1     WA_CIT_0     WA_CIT_1    WE_KIDS_0    WE_KIDS_1

1            0           32            0            0            0
2           12            0           30            0           12
3            0           35            0            0            0
4            0           34            0            0           12
5           14            0           31            0            0
6           12            0           54           12            0

     WE_KIDS_2    WA_KIDS_0    WA_KIDS_1    WA_KIDS_2

1           12            0            0           32
2            0            0           30            0
3           12            0            0           35
4            0            0           34            0
5           14            0            0           31
6            0           54            0            0

\end{outbit}
\end{script}

\subsection{Generating series from lists}

There are various ways of retrieving or generating individual series
from a named list. The most basic method is indexing into the
list. For example,
%
\begin{code}
series x3 = Xlist[3]
\end{code}
%
will retrieve the third element of the list \texttt{Xlist} under the
name \texttt{x3} (or will generate an error if \texttt{Xlist} has less
then three members).

In addition gretl offers several functions that apply to a list and
return a series.  In most cases, these functions also apply to single
series and behave as natural extensions when applied to lists, but
this is not always the case.

For recognizing and handling missing values, gretl offers
several functions (see the \GCR{} for details). In this context, it is
worth remarking that the \texttt{ok()} function can be used with a
list argument.  For example,
%
\begin{code}
list xlist = x1 x2 x3
series xok = ok(xlist)
\end{code}
%
After these commands, the series \texttt{xok} will have value 1 for
observations where none of \texttt{x1}, \texttt{x2}, or
\texttt{x3} has a missing value, and value 0 for any observations
where this condition is not met.

The functions \texttt{max}, \texttt{min}, \texttt{mean}, \texttt{sd},
\texttt{sum} and \texttt{var} behave ``horizontally'' rather than
``vertically'' when their argument is a list. For instance, the
following commands
\begin{code}
  list Xlist = x1 x2 x3
  series m = mean(Xlist)
\end{code}
produce a series \texttt{m} whose $i$-th element is the average of
$x_{1,i}, x_{2,i}$ and $x_{3,i}$; missing values, if any, are implicitly discarded.

\begin{table}
  \centering
  \begin{tabular}{lllllllll}
\hline
	& YpcFR & YpcGE	& YpcIT	& NFR		& NGE		& NIT        \\ 
\hline
\\ [-8pt]
1997	& 114.9 & 124.6	& 119.3	& 59830.635	& 82034.771	& 56890.372  \\ 
1998	& 115.3 & 122.7	& 120.0	& 60046.709	& 82047.195	& 56906.744  \\ 
1999	& 115.0	& 122.4	& 117.8	& 60348.255	& 82100.243	& 56916.317  \\ 
2000	& 115.6 & 118.8	& 117.2	& 60750.876	& 82211.508	& 56942.108  \\ 
2001	& 116.0	& 116.9	& 118.1	& 61181.560	& 82349.925	& 56977.217  \\ 
2002	& 116.3 & 115.5	& 112.2	& 61615.562	& 82488.495	& 57157.406  \\ 
2003	& 112.1 & 116.9	& 111.0	& 62041.798	& 82534.176	& 57604.658  \\ 
2004	& 110.3 & 116.6	& 106.9	& 62444.707	& 82516.260	& 58175.310  \\ 
2005	& 112.4 & 115.1	& 105.1	& 62818.185	& 82469.422	& 58607.043  \\ 
2006	& 111.9 & 114.2	& 103.3	& 63195.457	& 82376.451	& 58941.499  \\
\hline
  \end{tabular}
  \caption{GDP per capita and population in 3 European countries (Source: Eurostat)}
  \label{tab:EuroData}
\end{table}
In addition, gretl provides three functions for weighted
operations: \texttt{wmean}, \texttt{wsd} and \texttt{wvar}. Consider
as an illustration Table \ref{tab:EuroData}: the first three columns are GDP
per capita for France, Germany and Italy; columns 4 to 6 contain the
population for each country. If we want to compute an aggregate
indicator of per capita GDP, all we have to do is
\begin{code}
list Ypc = YpcFR YpcGE YpcIT
list N = NFR NGE NIT
y = wmean(Ypc, N)
\end{code}
so for example
\[
y_{1996} = \frac{114.9 \times 59830.635 + 124.6 \times 82034.771 +
  119.3 \times 56890.372} {59830.635 + 82034.771 + 56890.372} =
120.163
\]
See the \GCR\ for more details.

\section{Named strings}
\label{sec:named-strings}

First, a few words about so-called string literals. A string literal is
a sequence of characters enclosed in quotes, which in gretl means a
sequence of UTF-8 characters enclosed in straight double quotes
(\verb|"|). In many programming languages, backslash---a character which
never appears in ordinary text---is given a special role as the
``escape'' character in string literals, allowing so-called escape
sequences for producing special characters. Among the most common such
sequences are the following:
\begin{center}
\begin{tabular}{cl}
  \verb|\t| & tab \\
  \verb|\n| & newline \\
  \verb|\r| & carriage return \\
  \verb|\v| & vertical tab \\
  \verb|\"| & literal double quote \\
  \verb|\\| & literal backslash \\
\end{tabular}
\end{center}
Note that \verb|\"| permits embedding a double quote in a string
literal (it doesn't terminate the string), while if you want a literal
backslash you need to double it.

Gretl has a dual policy with regard to string literals.
\begin{enumerate}
\item In the context of the \texttt{printf} command and the
  \texttt{sprintf} function backslash is the escape character,
  supporting all of the sequences noted above.
\item In all other contexts backslash is treated as literal---with a
  single exception, namely that the sequence \verb|\"| inserts a literal
  double-quote into a string.
\end{enumerate}

The exception under clause 2 means that the command
\begin{code}
string s = "This is a \"string literal\""
\end{code}
gives \texttt{s} the following value
\begin{code}
This is a "string literal"
\end{code}

However, the exception has a consequence that should be noted: if you
want to produce a string that ends with a backslash you should use
\texttt{sprintf}, as shown below.
\begin{code}
string s = "foo\"           # don't do this!
string s = sprintf("foo\\") # do this instead
\end{code}
In the second form the doubled backslash inserts a literal backslash,
and the final double-quote terminates the string as intended.

We now move on to the various means of defining and manipulating string
variables in gretl. Here are a few examples.

%
\begin{code}
string s1 = "some stuff I want to save"
string s2 = getenv("HOME")
string s3 = s1 + 11
\end{code}
%
The first field after the type-name \texttt{string} is the name under
which the string should be saved, then comes an equals sign, then
comes a specification of the string to be saved. This may take any of
the following forms:

\begin{itemize}
\item a string literal; or
\item the name of an existing string variable; or
\item a function that returns a string (see below); or
\item any of the above followed by \texttt{+} and an integer offset.
\end{itemize}

The role of the integer offset is to use a substring of the preceding
element, starting at the given character offset.  An empty string is
returned if the offset is greater than the length of the string in
question.

To add to the end of an existing string you can use the operator
\verb|~=|, as in
%
\begin{code}
string s1 = "some stuff I want to "
string s1 ~= "save"
\end{code}
or you can use the \verb|~| operator to join two or more strings, as
in
\begin{code}
string s1 = "sweet"
string s2 = "Home, " ~ s1 ~ " home."
\end{code}

Let us again stress the point that if you wish to use backslash-escapes
to denote newlines, tabs and so on, you should use the \texttt{sprintf}
function instead. This function can also be used to produce a string
variable whose definition involves the values of other variables, as in
%
\begin{code}
scalar x = 8
foo = sprintf("var%d", x) # produces "var8"
\end{code}

\subsection{String variables and string substitution}

String variables can be used in two ways in scripting: the name of the
variable can be typed ``as is'', or it may be preceded by the ``at''
sign, \verb|@|. In the first variant the named string is treated as a
variable in its own right, while the second calls for ``string
substitution''. The context determines which of these variants is
appropriate. 

In the following contexts the names of string variables should be
given in plain form (without the ``at'' sign):

\begin{itemize}
\item When such a variable appears among the arguments to the
  \texttt{printf} command or \texttt{sprintf} function.
\item When such a variable is given as the argument to a function.
\item On the right-hand side of a \texttt{string} assignment.
\end{itemize}

Here is an illustration of the use of a named string argument with
\texttt{printf}:
%
\begin{code}
? string vstr = "variance"
Generated string vstr
? printf "vstr: %12s\n", vstr
vstr:     variance
\end{code}

String substitution can be used in contexts where a string variable is
not acceptable as such. If gretl encounters the symbol \verb|@|
followed directly by the name of a string variable, this notation is
treated as a ``macro'': the value of the variable is sustituted
literally into the command line before the regular parsing of the
command is carried out.

One typical use of string substitution is when you want the options used
with a particular command to vary depending on some condition. For
example,
%
\begin{code}
function void use_optstr (series y, list xlist, int verbose)
   string optstr = verbose ? "" : "--simple-print"
   ols y xlist @optstr 
end function

open data4-1
list X = const sqft
use_optstr(price, X, 1)
use_optstr(price, X, 0)
\end{code} 

When printing the value of a string variable using the \texttt{print}
command, the plain variable name should generally be used, as in
%
\begin{code}
string s = "Just testing"
print s
\end{code}
%
The following variant is equivalent, though clumsy.
%
\begin{code}
string s = "Just testing"
print "@s"
\end{code}
%
But note that this next variant does something quite different.
%
\begin{code}
string s = "Just testing"
print @s
\end{code}
%
After string substitution, the print command reads
%
\begin{code}
print Just testing
\end{code}
%
which attempts to print the values of two variables, \texttt{Just} and
\texttt{testing}.

A word of advice: it is considered bad practice to use string
substitution in cases where use of a regular string variable is
feasible.

\subsection{Built-in strings}

Apart from any strings that the user may define, some string variables
are defined by gretl itself.  These may be useful for people
writing functions that include shell commands.  The built-in strings
are as shown in Table~\ref{tab:pred-strings}.

\begin{table}[htbp]
\centering
\begin{tabular}{ll}
  \texttt{gretldir} & the gretl installation directory \\
  \texttt{workdir} & user's current gretl working directory \\
  \texttt{dotdir} & the directory gretl uses for temporary files \\
  \texttt{gnuplot} & path to, or name of, the \app{gnuplot} executable \\
  \texttt{tramo}& path to, or name of, the \app{tramo} executable \\
  \texttt{x12a} & path to, or name of, the \app{x-12-arima} executable \\
  \texttt{tramodir} & \app{tramo} data directory \\
  \texttt{x12adir} & \app{x-12-arima} data directory \\
\end{tabular}
\caption{Built-in string variables}
\label{tab:pred-strings}
\end{table}

To access these as ordinary string variables, prepend a dollar sign
(as in \verb|$dotdir|); to use them in string-substitution mode,
prepend the at-sign (\verb|@dotdir|).

\subsection{Reading strings from the environment}

It is possible to read into gretl's named strings, values that are
defined in the external environment.  To do this you use the function
\texttt{getenv}, which takes the name of an environment variable as
its argument.  For example:
%
\begin{code}
? string user = getenv("USER")
Generated string user
? string home = getenv("HOME")
Generated string home
? printf "%s's home directory is %s\n", user, home
cottrell's home directory is /home/cottrell
\end{code}
%
To check whether you got a non-empty value from a given call to
\texttt{getenv}, you can use the function \texttt{strlen}, which
retrieves the length of the string, as in
%
\begin{code}
? string temp = getenv("TEMP")
Generated string temp
? scalar x = strlen(temp)
Generated scalar x = 0
\end{code}

\subsection{Capturing strings via the shell}

If shell commands are enabled in gretl, you can capture the
output from such commands using the syntax 

\texttt{string} \textsl{stringname} = \texttt{\$(}\textsl{shellcommand}\texttt{)}

That is, you enclose a shell command in parentheses, preceded by
a dollar sign.

\subsection{Reading from a file into a string}

You can read the content of a file into a string variable using
the syntax

\texttt{string} \textsl{stringname} = \texttt{readfile(}\textsl{filename}\texttt{)}

The \textsl{filename} field may be given as a string variable.  For
example
%
\begin{code}
? fname = sprintf("%s/QNC.rts", $x12adir)
Generated string fname
? string foo = readfile(fname)
Generated string foo
\end{code}

\subsection{More string functions}

Gretl offers several functions for creating or manipulating
strings. You can find these listed and explained in the
\textit{Function Reference} under the category \textsf{Strings}.
    
%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "gretl-guide"
%%% End: 

