\chapter{Named lists and strings}
\label{chap-persist}

%%% Should this chapter deal with saving models too?

\section{Named lists}
\label{named-lists}

Many \app{gretl} commands take one or more lists of series as
arguments.  To make this easier to handle in the context of command
scripts, and in particular within user-defined functions, \app{gretl}
offers the possibility of \textit{named lists}.  

\subsection{Creating and modifying named lists}

A named list is created using the keyword \texttt{list}, followed by
the name of the list, an equals sign, and an expression that forms a
list.  The most basic sort of expression that works in this context
is a space-separated list of variables, given either by name or
by ID number.  For example,
%
\begin{code}
list xlist = 1 2 3 4
list reglist = income price 
\end{code}

Note that the variables in question must be of the series type: you
can't include scalars in a named list.

Two special forms are available:
\begin{itemize}
\item If you use the keyword \texttt{null} on the right-hand side,
  you get an empty list.
\item If you use the keyword \texttt{dataset} on the right, you get
  a list containing all the series in the current dataset (except
  the pre-defined \texttt{const}).
\end{itemize}

The name of the list must start with a letter, and must be composed
entirely of letters, numbers or the underscore character.  The maximum
length of the name is 15 characters; list names cannot contain
spaces.  

Once a named list has been created, it will be ``remembered'' for the
duration of the \app{gretl} session, and can be used in the context of
any \app{gretl} command where a list of variables is expected.  One
simple example is the specification of a list of regressors:
%
\begin{code}
list xlist = x1 x2 x3 x4
ols y 0 xlist
\end{code}

To get rid of a list, you can use the following syntax:
\begin{code}
  list xlist delete
\end{code}
Be careful: \texttt{delete xlist} will delete the variables contained
in the list, so it implies data loss (which may not be what you want).
On the other hand, \texttt{list xlist delete} will simply ``undefine''
the \texttt{xlist} identifier and the variables themselves will not be
affected.

Lists can be modified in two ways.  To \textit{redefine} an existing
list altogether, use the same syntax as for creating a list.  For
example
%
\begin{code}
list xlist = 1 2 3
xlist = 4 5 6
\end{code}

After the second assignment, \texttt{xlist} contains just variables 4,
5 and 6.

To \textit{append} or \textit{prepend} variables to an existing list,
we can make use of the fact that a named list stands in for a
``longhand'' list.  For example, we can do
%
\begin{code}
list xlist = xlist 5 6 7
xlist = 9 10 xlist 11 12
\end{code}
%
Another option for appending a term (or a list) to an existing list is
to use \texttt{+=}, as in
%
\begin{code}
xlist += cpi
\end{code}
%
To drop a variable from a list, use \texttt{-=}:
%
\begin{code}
xlist -= cpi
\end{code}
%

In most contexts where lists are used in \app{gretl}, it is expected
that they do not contain any duplicated elements.  If you form a new
list by simple concatenation, as in \texttt{list L3 = L1 L2}
(where \texttt{L1} and \texttt{L2} are existing lists), it's possible
that the result may contain duplicates.  To guard against this you can
form a new list as the union of two existing ones:
%
\begin{code}
list L3 = L1 || L2
\end{code}
%
The result is a list that contains all the members of \texttt{L1},
plus any members of \texttt{L2} that are not already in \texttt{L1}.

In the same vein, you can construct a new list as the intersection of
two existing ones:
%
\begin{code}
list L3 = L1 && L2
\end{code}
%
Here \texttt{L3} contains all the elements that are present in both
\texttt{L1} and \texttt{L2}.


\subsection{Lists and matrices}

Another way of forming a list is by assignment from a matrix.  The
matrix in question must be interpretable as a vector containing ID
numbers of (series) variables.  It may be either a row or a column
vector, and each of its elements must have an integer part that is
no greater than the number of variables in the data set.  For example:
%
\begin{code}
matrix m = {1,2,3,4}
list L = m
\end{code}
%
The above is OK provided the data set contains at least 4 variables.

\subsection{Querying a list}

You can determine whether an unknown variable actually represents a list
using the function \texttt{islist()}.
%
\begin{code}
series xl1 = log(x1)
series xl2 = log(x2)
list xlogs = xl1 xl2
genr is1 = islist(xlogs)
genr is2 = islist(xl1)
\end{code}

The first \texttt{genr} command above will assign a value of 1 to
\texttt{is1} since \texttt{xlogs} is in fact a named list.  The second
genr will assign 0 to \texttt{is2} since \texttt{xl1} is a data
series, not a list.  

You can also determine the number of variables or elements in a list
using the function \texttt{nelem()}.
%
\begin{code}
list xlist = 1 2 3
nl = nelem(xlist)
\end{code}

The (scalar) variable \texttt{nl} will be assigned a value of 3 since
\texttt{xlist} contains 3 members.

You can display the membership of a named list just by giving its name,
as illustrated in this interactive session:
%
\begin{code}
? list xlist = x1 x2 x3
Added list 'xlist'
? xlist
 x1 x2 x3
\end{code}
%
Note that \texttt{print xlist} will do something different, namely
print the values of all the variables in \texttt{xlist} (as should be
expected).

\subsection{Generating lists of transformed variables}

Given a named list of variables, you are able to generate lists of
transformations of these variables using the functions \texttt{log},
\texttt{lags}, \texttt{diff}, \texttt{ldiff}, \texttt{sdiff} or
\texttt{dummify}.  For example
%
\begin{code}
list xlist = x1 x2 x3
list lxlist = log(xlist)
list difflist = diff(xlist)
\end{code}

When generating a list of \textit{lags} in this way, you specify the
maximum lag order inside the parentheses, before the list name and
separated by a comma.  For example
%
\begin{code}
list xlist = x1 x2 x3
list laglist = lags(2, xlist)
\end{code}
%
or
%
\begin{code}
scalar order = 4
list laglist = lags(order, xlist)
\end{code}

These commands will populate \texttt{laglist} with the specified
number of lags of the variables in \texttt{xlist}.  You can give the
name of a single series in place of a list as the second argument to
\texttt{lags}: this is equivalent to giving a list with just one
member.

The \texttt{dummify} function creates a set of dummy variables coding
for all but one of the distinct values taken on by the original
variable, which should be discrete.  (The smallest value is taken as
the omitted catgory.)  Like lags, this function returns a list even if
the input is a single series.


\subsection{Generating series from lists}

Once a list is defined, \app{gretl} offers several functions that
apply to the list and return a series. In most cases, these functions
also apply to single series and behave as natural extensions when
applied to a list, but this is not always the case.

For recognizing and handling missing values, \app{Gretl} offers
several functions (see the \GCR{} for details). In this context, it is
worth remarking that the \texttt{ok()} function can be used with a
list argument.  For example,
%
\begin{code}
list xlist = x1 x2 x3
series xok = ok(xlist)
\end{code}
%
After these commands, the series \texttt{xok} will have value 1 for
observations where none of \texttt{x1}, \texttt{x2}, or
\texttt{x3} has a missing value, and value 0 for any observations
where this condition is not met.

The functions \texttt{max}, \texttt{min}, \texttt{mean}, \texttt{sd},
\texttt{sum} and \texttt{var} behave horizontally rather than
vertically when their argument is a list. For instance, the following
commands
\begin{code}
  list Xlist = x1 x2 x3
  series m = mean(Xlist)
\end{code}
produce a series \texttt{m} whose $i$-th element is the average of
$x_{1,i}, x_{2,i}$ and $x_{3,i}$; missing values, if any, are implicitly discarded.

\begin{table}
  \centering
  \begin{tabular}{lllllllll}
\hline
	& YpcFR & YpcGE	& YpcIT	& NFR		& NGE		& NIT        \\ 
\hline
\\ [-8pt]
1997	& 114.9 & 124.6	& 119.3	& 59830.635	& 82034.771	& 56890.372  \\ 
1998	& 115.3 & 122.7	& 120.0	& 60046.709	& 82047.195	& 56906.744  \\ 
1999	& 115.0	& 122.4	& 117.8	& 60348.255	& 82100.243	& 56916.317  \\ 
2000	& 115.6 & 118.8	& 117.2	& 60750.876	& 82211.508	& 56942.108  \\ 
2001	& 116.0	& 116.9	& 118.1	& 61181.560	& 82349.925	& 56977.217  \\ 
2002	& 116.3 & 115.5	& 112.2	& 61615.562	& 82488.495	& 57157.406  \\ 
2003	& 112.1 & 116.9	& 111.0	& 62041.798	& 82534.176	& 57604.658  \\ 
2004	& 110.3 & 116.6	& 106.9	& 62444.707	& 82516.260	& 58175.310  \\ 
2005	& 112.4 & 115.1	& 105.1	& 62818.185	& 82469.422	& 58607.043  \\ 
2006	& 111.9 & 114.2	& 103.3	& 63195.457	& 82376.451	& 58941.499  \\
\hline
  \end{tabular}
  \caption{GDP per capita and population in 3 European countries (Source: Eurostat)}
  \label{tab:EuroData}
\end{table}
In addition, \app{gretl} provides three functions for weighted
operations: \texttt{wmean}, \texttt{wsd} and \texttt{wvar}. Consider
as an illustration Table \ref{tab:EuroData}: the first three columns are GDP
per capita for France, Germany and Italy; columns 4 to 6 contain the
population for each country. If we want to compute an aggregate
indicator of per capita GDP, all we have to do is
\begin{code}
list Ypc = YpcFR YpcGE YpcIT
list N = NFR NGE NIT
y = wmean(Ypc, N)
\end{code}
so for example
\[
y_{1996} = \frac{114.9 \times 59830.635 + 124.6 \times 82034.771 +
  119.3 \times 56890.372} {59830.635 + 82034.771 + 56890.372} =
120.163
\]
See the \GCR\ for more details.

\section{Named strings}
\label{sec:named-strings}

For some purposes it may be useful to save a string (that is, a
sequence of characters) as a named variable that can be reused.
Versions of \app{gretl} higher than 1.6.0 offer this facility, but
some of the refinements noted below are available only in \app{gretl}
1.7.2 and higher.

To \textit{define} a string variable, you can use either of two
commands, \texttt{string} or \texttt{sprintf}.  The \texttt{string}
command is simpler: you can type, for example,
%
\begin{code}
string s1 = "some stuff I want to save"
string s2 = getenv("HOME")
string s3 = s1 + 11
\end{code}
%
The first field after \texttt{string} is the name under which the
string should be saved, then comes an equals sign, then comes a
specification of the string to be saved. This can be the keyword
\texttt{null}, to produce an empty string, or may take any of the 
following forms:

\begin{itemize}
\item a string literal (enclosed in double quotes); or
\item the name of an existing string variable; or
\item a function that returns a string (see below); or
\item any of the above followed by \texttt{+} and an integer offset.
\end{itemize}

The role of the integer offset is to use a substring of the preceding
element, starting at the given character offset.  An empty string is
returned if the offset is greater than the length of the string in
question.

To add to the end of an existing string you can use the operator
\texttt{+=}, as in
%
\begin{code}
string s1 = "some stuff I want to "
string s1 += "save"
\end{code}
or you can use the \verb|~| operator to join two or more strings, as
in
\begin{code}
string s1 = "sweet"
string s2 = "Home, " ~ s1 ~ " home."
\end{code}

Note that when you define a string variable using a string literal, no
characters are treated as ``special'' (other than the double quotes
that delimit the string).  Specifically, the backslash is not used as
an escape character.  So, for example,
%
\begin{code}
string s = "\"
\end{code}
%
is a valid assignment, producing a string that contains a single
backslash character.  If you wish to use backslash-escapes to denote
newlines, tabs, embedded double-quotes and so on, use \texttt{sprintf}
instead.

The \texttt{sprintf} command is more flexible.  It works exactly as
\app{gretl}'s \texttt{printf} command except that the ``format''
string must be preceded by the name of a string variable.  For
example,
%
\begin{code}
scalar x = 8
sprintf foo "var%d", x
\end{code}

To use the \emph{value} of a string variable in a command, give the
name of the variable preceded by the ``at'' sign, \verb|@|.  This
notation is treated as a ``macro''.  That is, if a sequence of
characters in a \app{gretl} command following the symbol \verb|@| is
recognized as the name of a string variable, the value of that
variable is sustituted literally into the command line before the
regular parsing of the command is carried out.  This is illustrated in
the following interactive session:
%
\begin{code}
? scalar x = 8
 scalar x = 8
Generated scalar x (ID 2) = 8
? sprintf foo "var%d", x
Saved string as 'foo'
? print "@foo"
var8
\end{code}
%
Note the effect of the quotation marks in the line 
\verb|print "@foo"|.  The line
%
\begin{code}
? print @foo
\end{code}
%
would \textit{not} print a literal ``\texttt{var8}'' as above.  After
pre-processing the line would read
%
\begin{code}
print var8
\end{code}
%
It would therefore print the value(s) of the variable \texttt{var8},
if such a variable exists, or would generate an error otherwise.

In some contexts, however, one wants to treat string variables as
variables in their own right: to do this, give the name of
the variable without the leading \verb|@| symbol.  This is the
way to handle such variables in the following contexts:

\begin{itemize}
\item When they appear among the arguments to the commands \texttt{printf} and
  \texttt{sprintf}.
\item On the right-hand side of a \texttt{string} assignment.
\item When they appear as an argument to the function taking
  a string argument.
\end{itemize}

Here is an illustration of the use of named string arguments with
\texttt{printf}:
%
\begin{code}
string vstr = "variance"
Generated string vstr
printf "vstr: %12s\n", vstr
vstr:     variance
\end{code}
%
Note that \texttt{vstr} should not be put in quotes in this context.
Similarly with
\begin{code}
? string vstr_copy = vstr
\end{code}

\subsection{Built-in strings}

Apart from any strings that the user may define, some string variables
are defined by \app{gretl} itself.  These may be useful for people
writing functions that include shell commands.  The built-in strings
are as shown in Table~\ref{tab:pred-strings}.

\begin{table}[htbp]
\centering
\begin{tabular}{ll}
  \texttt{gretldir} & the \app{gretl} installation directory \\
  \texttt{workdir} & user's current \app{gretl} working directory \\
  \texttt{dotdir} & the directory \app{gretl} uses for temporary files \\
  \texttt{gnuplot} & path to, or name of, the \app{gnuplot} executable \\
  \texttt{tramo}& path to, or name of, the \app{tramo} executable \\
  \texttt{x12a} & path to, or name of, the \app{x-12-arima} executable \\
  \texttt{tramodir} & \app{tramo} data directory \\
  \texttt{x12adir} & \app{x-12-arima} data directory \\
\end{tabular}
\caption{Built-in string variables}
\label{tab:pred-strings}
\end{table}

\subsection{Reading strings from the environment}

In addition, it is possible to read into \app{gretl}'s named strings,
values that are defined in the external environment.  To do this you
use the function \texttt{getenv}, which takes the name of an environment
variable as its argument.  For example:
%
\begin{code}
? string user = getenv("USER")
Saved string as 'user'
? string home = getenv("HOME")
Saved string as 'home'
? print "@user's home directory is @home"
cottrell's home directory is /home/cottrell
\end{code}
%
To check whether you got a non-empty value from a given call to
\texttt{getenv}, you can use the function \texttt{strlen}, which
retrieves the length of the string, as in
%
\begin{code}
? string temp = getenv("TEMP")
Saved empty string as 'temp'
? scalar x = strlen(temp)
Generated scalar x (ID 2) = 0
\end{code}

The function \texttt{isstring} returns 1 if its argument is the name
of a string variable, 0 otherwise.  However, if the return is 1
the string may still be empty.

At present the \texttt{getenv} function can only be used on the
right-hand side of a \texttt{string} assignment, as in the above
illustrations.

\subsection{Capturing strings via the shell}

If shell commands are enabled in \app{gretl}, you can capture the
output from such commands using the syntax 

\texttt{string} \textsl{stringname} = \texttt{\$(}\textsl{shellcommand}\texttt{)}

That is, you enclose a shell command in parentheses, preceded by
a dollar sign.

\subsection{Reading from a file into a string}

You can read the content of a file into a string variable using
the syntax

\texttt{string} \textsl{stringname} = \texttt{readfile(}\textsl{filename}\texttt{)}

The \textsl{filename} field may be given as a string variable.  For
example
%
\begin{code}
? sprintf fname "%s/QNC.rts", x12adir
Generated string fname
? string foo = readfile(fname)
Generated string foo
\end{code}
%
The above could also be accomplished using the ``macro'' variant
of a string variable, provided it is placed in quotation marks: 

\verb|string foo = readfile("@x12adir/QNC.rts")|

\subsection{The \texttt{strstr} function}

Invocation of this function takes the form

\texttt{string} \textsl{stringname} = 
\texttt{strstr(}\textsl{s1}\texttt{,} \textsl{s2}\texttt{)}

The effect is to search \textsl{s1} for the first occurrence of
\textsl{s2}.  If no such occurrence is found, an empty string is
returned; otherwise the portion of \textsl{s1} starting with
\textsl{s2} is returned.  For example:
%
\begin{code}
? string hw = "hello world"
Saved string as 'hw'
? string w = strstr(hw, "o")
Saved string as 'w'
? print "@w"
o world
\end{code}
%


    
%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "gretl-guide"
%%% End: