\chapter{String-valued series}
\label{chap:strval-series}

\section{Introduction}

By a string-valued series we mean a series whose primary values are
strings (though internally such series comprise an integer coding plus
a ``dictionary'' mapping from the integer values to strings). This
chapter explains how to create such series and describes the
operations that are supported for them.

\section{Creating a string-valued series}

This can be done in three ways:
\begin{itemize}
\item by reading such a series from a suitable source file;
\item by taking a suitable numerical series within gretl and adding
  string values using the \cmd{stringify()} function; and
\item by direct assignment to a series from an array of
strings.
\end{itemize}

In each case string values will be preserved when such a series is
saved in a gretl-native data file.

\subsection{Reading string-valued series}
\label{sec:reading}

The primary ``suitable source'' for string-valued series is a
delimited text data file (but see section~\ref{sec:other-imports}
below). Here's a little example. The following is the content of a
file named \texttt{gc.csv}:
%
\begin{code}
city,year
"Bilbao",2009
"Toruń",2011
"Oklahoma City",2013
"Berlin",2015
"Athens",2017
"Naples",2019
\end{code}
%
A script to read this file and its output are shown in
Listing~\ref{listing:cities}, from which we can see a few things.

\begin{script}[htbp]
  \caption{Working with a string-valued series}
  \label{listing:cities}
 Input:
\begin{scodebit}
open gc.csv --quiet
print --byobs
print city --byobs --numeric
printf "The third gretl conference took place in %s.\n", city[3]
\end{scodebit}

Output:
\begin{outbit}
? print --byobs

          city         year

1       Bilbao         2009
2        Toruń         2011
3 Oklahoma C..         2013
4       Berlin         2015
5       Athens         2017
6       Naples         2019

? print city --byobs --numeric

          city

1            1
2            2
3            3
4            4
5            5
6            6

The third gretl conference took place in Oklahoma City.
\end{outbit}
\end{script}

\begin{itemize}
\item By default the \cmd{print} command shows us the string values
  of the series \texttt{city}, and it handles non-ASCII characters
  provided they're in UTF-8 (but it doesn't handle longer strings
  very elegantly).
\item The \verb|--numeric| option to \cmd{print} exposes the
  integer codes for a string-valued series.
\item The syntax \texttt{seriesname[obs]} yields a string when a
  series is string-valued.
\end{itemize}

If you want to access the numeric code for a particular string-valued
observation you can get it by ``casting'' the series in question to a
vector (by wrapping the identifier in curly brackets). So, for
example,
\begin{code}
printf "The code for '%s' is %d.\n", city[3], {city}[3]
\end{code}
gives
\begin{code}
The code for 'Oklahoma City' is 3.
\end{code}

The numeric codes for string-valued series are always assigned thus:
reading the data file row by row, the first string value is assigned
1, the next \textit{distinct} string value is assigned 2, and so on.

\subsection{Assigning string values to a numeric series}
\label{sec:stringify}

This is done via the \cmd{stringify()} function, which takes two
arguments, the name of a series and an array of strings. For this to
work two conditions must be met:

\begin{enumerate}
\item The series must have only integer values and the smallest value
  must be 1 or greater.
\item The array of strings must have at least $n$ distinct members,
  where $n$ is the largest value found in the series.
\end{enumerate}

The logic of these conditions is that we're looking to create a
mapping as described above, from a 1-based sequence of integers to a
set of strings. However, we're allowing for the possibility that the
series in question is an incomplete sample from an associated
population. Suppose we have a series that goes 2, 3, 5, 9, 10. This is
taken to be a sample from a population that has at least 10 discrete
values, 1, 2, \dots{}, 10, and so requires at least 10 value-strings.

Here's (a simplified version of) an example that one of the authors
has had cause to use: deriving US-style ``letter grades'' from a
series containing percentage scores for students. Call the percentage
series $x$, and say we want to create a series with values \texttt{A}
for $x \geq 90$, \texttt{B} for $80 \leq x <90$, and so on down to
\texttt{F} for $x<60$. Then we can do:
\begin{code}
series grade = 1 # F, the least value
grade += x >= 60 # D
grade += x >= 70 # C
grade += x >= 80 # B
grade += x >= 90 # A
stringify(grade, strsplit("F D C B A"))
\end{code}
%
The way the \texttt{grade} series is constructed is not the most
compact, but it's nice and explicit, and easy to amend if one wants to
adjust the threshold values. Note the use of \cmd{strsplit()} to
create an on-the-fly array of strings from a string literal; this is
convenient when the array contains a moderate number of elements with
no embedded spaces. An alternative way to get the same result is to
define the array of strings via the \cmd{defarray()} function, as in
\begin{code}
stringify(grade, defarray("F","D","C","B","A"))
\end{code}

The inverse operation of \cmd{stringify()} is performed by the
\cmd{strvals()} function: this retrieves the array of distinct string
values from a series (or returns an empty array if the series is not
string-valued).

\subsection{Assigning from an array of strings}
\label{sec:array-assign}

Given an array of strings whose length matches the full length of the
current dataset you can assign directly to a series result, provided
these conditions are satisfied: the dataset is not sub-sampled, and if
the assignment is to a pre-existing series it is not already
string-valued.

Here's a trivial example:
\begin{code}
nulldata 6
strings S = defarray("a", "b", "c", "b", "a", "d")
series sx = S
print sx --byobs
\end{code}
Here's a second example where we create a string-valued series
using the ``observation markers'' from the current dataset, after
grabbing them as an array via the \texttt{markers} command:
\begin{code}
open data4-10
markers --to-array=S
series state = S
print state --byobs
\end{code}
And here's a third example where we construct the array of strings by
reading from a text file:
\begin{code}
nulldata 8
series sv = strsplit(readfile("ABCD.txt"))
print sv --byobs
\end{code}
This will work fine if the content of \texttt{ABCD.txt} is something
like
\begin{code}
A B C D D C B A
\end{code}
(containing 8 space-separated values, with or without line breaks). If
the strings in question contain embedded spaces you would have to make
use of the optional second argument to \texttt{strsplit}.

\section{Permitted operations}

One question that arises with string-valued series is, what exactly
are you allowed to do with them? The optimal policy may be debatable,
but here we set out the current state of things.

\subsection{Setting values per observation}

You can set particular values in a string-valued series either by
string or numeric code. For example, suppose (in relation to the
example in section~\ref{sec:stringify}) that for some reason student
number 31 with a percentage score of 88 nonetheless merits an
\texttt{A} grade. We could do
\begin{code}
grade[31] = "A"
\end{code}
or, if we're confident about the mapping,
\begin{code}
grade[31] = 5
\end{code}
Or to raise the student's grade by one letter:
\begin{code}
grade[31] += 1
\end{code}

What you're \textit{not} allowed to do here is make a numerical
adjustment that would put the value out of bounds in relation to the
set of string values. For example, if we tried \texttt{grade[31] = 6}
we'd get an error.

On the other hand, you \textit{can} implicitly extend the set of
string values. This wouldn't make sense for the letter grades example
but it might for, say, city names. Returning to the example in
section~\ref{sec:reading} suppose we try
%
\begin{code}
dataset addobs 1
year[7] = 2023
city[7] = "Gdańsk"
\end{code}
%
This will work: we're implicitly adding another member to the string
table for \texttt{city}; the associated numeric code will be the next
available integer.\footnote{So please be careful: one may
  inadvertently add a new string value by mistyping a string that's
  already present.}

\subsection{Logical product of two string-valued series}

The operator \verb|^| can be used to produce what we might call the
logical product of two string-valued series, as in
\begin{code}
series sv3 = sv1 ^ sv2
\end{code}
The result is another string-valued series with value $s_i.s_j$ at
observations where \texttt{sv1} has value $s_i$ and \texttt{sv2} has
value $s_j$. For example, if at a given observation \texttt{sv1} has
value ``\texttt{A}'' and \texttt{sv2} has value ``\texttt{X}'', then
\texttt{sv3} will have value ``\texttt{A.X}''. The set of strings
attached to the resulting series will include all such string
combinations even if they are not all represented in the given sample.

\subsection{Assignment to a string-valued series}

In an assignment statement where the left-hand side (LHS) term is an
existing string-valued series two general conditions must be
met. First, the right-hand side (RHS) term must be a series (either
numeric or string-valued) and second, the assignment operator must be
plain ``\texttt{=}''; inflected operators such as \texttt{+=} and
\texttt{*=} are not supported.

When the RHS series is numeric, all its values must be either integers
between 1 and the number of strings attached to the LHS series, or
\texttt{NA}. This is required to preserve the integrity of the LHS.
When the RHS series is itself string-valued there are two cases to
consider: there's no sample restriction in place, or there is such a
restriction. In the unrestricted case the LHS series is in effect
destroyed and replaced by a clone of the RHS. Otherwise string values
on the RHS are written into the LHS only within the current sample
range. If an RHS string is already present on the left its numerical
code is adjusted if necessary to match the LHS string table; if it is
not present on the left it is appended to the LHS string table.

\subsection{Missing values}

We support one exception to the general rule, never break the mapping
between strings and numeric codes for string-valued series: you can
mark particular observations as missing. This is done in the usual
way, e.g.,
\begin{code}
grade[31] = NA
\end{code}
Note, however, that on importing a string series from a delimited text
file any non-blank strings (including ``NA'') will be interpreted as
valid values; any missing values in such a file should therefore be
represented by blank cells.

\subsection{Copying a string-valued series}

If you make a copy of a string-valued series, as in
\begin{code}
series foo = city
\end{code}
the string values are \textit{not} copied over: you get a purely
numerical series holding the codes of the original series. But if you
want a full copy with the string values that can easily be arranged:
\begin{code}
series citycopy = city
stringify(citycopy, strvals(city))
\end{code}

\subsection{String-valued series in other contexts}

String-valued series can be used on the right-hand side of assignment
statements at will, and in that context their numerical values are
taken. For example,
%
\begin{code}
series y = sqrt(city)
\end{code}
%
will elicit no complaint and generate a numerical series 1, 1.41421,
\dots{}. It's up to the user to judge whether this sort of thing
makes any sense.

Similarly, it's up to the user to decide if it makes sense to use a
string-valued series ``as is'' in a regression model, whether as
regressand or regressor---again, the numerical values of the series
are taken. Often this will not make sense, but sometimes it may: the
numerical values may by design form an ordinal, or even a cardinal,
scale (as in the ``grade'' example in section~\ref{sec:stringify}).

More likely, one would want to use \cmd{dummify} on a string-valued
series before using it in statistical modeling. In that context
gretl's series labels are suitably informative. For example, suppose
we have a series \texttt{race} with numerical values 1, 2 and 3 and
associated strings ``White'', ``Black'' and ``Other''. Then the hansl
code
\begin{code}
list D = dummify(race)
labels
\end{code}
will show these labels:
\begin{code}
Drace_2: dummy for race = 'Black'
Drace_3: dummy for race = 'Other'
\end{code}

Given a series such as \texttt{race} you can use its string values in a
sample restriction, as in
\begin{code}
smpl race == "Black" --restrict
\end{code}
(although \texttt{race == 2} would also be acceptable).

\subsection{Accessing string values}

We have mentioned above two ways of accessing string values from a
given series: via the syntax
\begin{code}
seriesname[obs]
\end{code}
to obtain a single such value; and via the \texttt{strvals()} function
to obtain an array holding all its distinct values. Here we note a
third option: direct assignment from a string-valued series to an
array of strings, as in
%
\begin{code}
strings S = sv
\end{code}
%
where \texttt{sv} is a suitable series. In this case you get an array
holding \textit{all} the \texttt{sv} strings for observations in the
current sample range, not just the distinct values as with
\texttt{strvals}.

\section{String-valued series and functions}

We first offer a few words on built-in functions that can be applied
to string-valued series. The five functions \texttt{substr},
\texttt{strsub}, \texttt{regsub}, \texttt{tolower} and
\texttt{toupper} all perform transformations on
strings---respectively, extraction of a substring, replacement of a
substring, replacement via regular expression, conversion to all
lower-case and to all upper-case (see the \GCR\ for details). These
functions work on single strings, arrays of strings and also
string-valued series. Note that when applied to a string-valued series
these functions may reduce the number of distinct strings attached to
the series. For example, some string values that are originally
distinct may ``collapse'' into identity when converted to all
lower-case. This possibility is handled by adjustment of the integer
codes as needed.

A special case is presented by the built-in \texttt{strvsort}
function: this does not return a modified string-valued series but
rather modifies such a series in place. It puts the string values into
alphabetical order and recalculates the integer codes so as to
preserve the original association between observation number and
string. If, for example, the first observation had a string value of
``\texttt{X}'', coded as 1, it will still have value ``\texttt{X}''
but its code will reflect the position of ``\texttt{X}'' in the
alphabetized ordering. This can be particularly useful if a dataset
comprises several series having the same string values, but occurring
in various orders. The effect of running \texttt{strvsort} on such
series will be to impose a common numerical encoding.

User-defined hansl functions can also deal with string-valued series.
If you supply such a series as an argument to a hansl function its
string values will be accessible within the function. One can test
whether a given series \texttt{arg} is string-valued as follows:
\begin{code}
if nelem(strvals(arg)) > 0
  # yes
else
  # no
endif
\end{code}

It's also possible, since gretl version 2023c, to put something like
the code that generated the \texttt{grade} series in
section~\ref{sec:stringify} into a function, and return the
stringified series, as in the following (where we assume that
\texttt{x} contains percentage scores):
\begin{code}
function series letter_grade (series x)
  series grade
  # define grade based on x and stringify it, as shown above
  return grade
end function
\end{code}
%
An alternative means of achieving the same effect---and the only means
available prior to gretl 2023c---is to to define \texttt{grade} as a
series at the level of the caller and pass it in ``pointer'' form to
\verb|letter_grade()|, as in
%
\begin{code}
function void letter_grade (series x, series *grade)
  # define grade based on x and stringify it
end function

# caller
...
series grade
letter_grade(x, &grade)
\end{code}

As you'll see from the account above, we don't offer any very fancy
facilities for string-valued series. We'll read them from suitable
sources and we'll create them natively via \cmd{stringify}---and
we'll try to ensure that they retain their integrity---but we don't,
for example, take the specification of a string-valued series as a
regressor as an implicit request to include the dummification of its
distinct values.

\section{Other import formats}
\label{sec:other-imports}

In section~\ref{sec:reading} we illustrated the reading of
string-valued series with reference to a delimited text data
file. Gretl can also handle several other sources of string-valued
data, including the spreadsheet formats \texttt{xls}, \texttt{xlsx},
\texttt{gnumeric} and \texttt{ods} and (to a degree) the formats of
\textsf{Stata}, \textsf{SAS} and \textsf{SPSS}.

\subsection{Stata files}

Stata supports two relevant sorts of variables: (1) those that are of
``string type'' and (2) variables of one or other numeric type that
have ``value labels'' defined. Neither of these is exactly equivalent
to what we call a ``string-valued series'' in gretl.

Stata variables of string type have no numeric representation; their
values are literally strings, and that's all. Stata's numeric
variables with value labels do not have to be integer-valued and their
least value does not have to be 1; however, you can't define a label
for a value that is not an integer. Thus in Stata you can have a
series that comprises both integer and non-integer values, but only
the integer values can be labeled.\footnote{Verified in Stata 12.}

This means that on import to gretl we can readily handle variables of
string type from Stata's \texttt{dta} files. We give them a 1-based
numeric encoding; this is arbitrary but does not conflict with any
information in the \texttt{dta} file. On the other hand, in general
we're not able to handle Stata's numeric variables with value labels;
currently we report the value labels to the user but do not attempt to
store them in the gretl dataset. We could check such variables and
import them as string-valued series if they satisfy the criteria
stated in section~\ref{sec:stringify} but we don't at present.

\subsection{SAS and SPSS files}

Gretl is able to read and preserve string values associated with
variables from SAS ``export'' (\texttt{xpt}) files, and also from SPSS
\texttt{sav} files. Such variables seem to be on the same pattern as
Stata variables of string type.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "gretl-guide"
%%% End:
