Gekko Timeseries and Modeling Software Timeseries handling, and solving of large-scale economic models Thu, 24 Jun 2021 09:25:34 +0000 en-GB hourly 1 Gekko Timeseries and Modeling Software 32 32 Functions in Gekko 3.0 (UFCS) Mon, 26 Aug 2019 10:34:31 +0000

Regarding functions (both in-built and user-defined) in Gekko 3.0, these implement so-called UFCS (Uniform Function Call Syntax). First, some background.

In object-oriented languages, a method has the form x.f(a, b), where x is the object, and f() is the method, with arguments a and b (this is also called a member function). In contrast, a function has the form f(a, b), where a and b are again the arguments (this is also called a free function). When designing the syntax for Gekko 3.0, the distinction between methods and functions came up. For instance, consider a function returning the length of a Gekko list. If the list is called #m, we could use either #m.length() or length(#m). Both make sense, and both have their advantages and drawbacks. Gekko 2.0 and earlier only had free functions, but it became clear that objects methods were needed, in order to implement method chaining and fluent interfaces so that we can write, for instance, #m = #m1.append(‘a’).except(#m2).sort(). This means taking the list #m1, adding the string ‘a’, removing elements from the list #m2, and finally sorting the list alphabetically. Using free functions, this would read like #m = sort(except(append(#m1, ‘a’), #m2)), which is much harder (non-fluent) to read, because it reads backwards and from the inside out.

So regarding Gekko 3.0 there was no doubt that object methods had to be implemented. But there was a really nagging question. In some cases where objects are involved, a free function also makes sense, and choosing between implementing for instance the length of a list as an object method or free function seemed arbitrary. This is also seen in computer languages like C#, Java, Python and others, where it is not always obvious whether some functionality is implemented as an object method or a free function. For instance, in Python, the length of a list is len(m), whereas in C# it would be m.Count(). So for Gekko 3.0, choosing between length(#m) and #m.length() was not obvious, and contemplating whether a given functionality should be an object method or free function was both time-consuming and a bit frustrating. But then the author discovered the UFCS idea, which is used in, for instance, the D language. The idea is really simple: a given free function f(a, b) can always be written in the alternative form a.f(b), where the first argument of f(a, b) is moved outside the parentheses, and then followed by a dot. This is purely syntactic sugar: when Gekko 3.0 sees a function like a.f(b), it is first transformed into a “normal” free function f(a, b), and then executed. So when implementing funtions in Gekko 3.0, if the user is careful about putting the “object” (or primary argument) first in the argument list, calling the function like an object method becomes possible. Therefore, Gekko users can also design user-defined functions that can be called with an object-oriented flavor.

Consider a user-defined function like rescale(x, %v) which rescales the values of the timeseries x with the scaling factor %v. Because the timeseries is the first argument, this can alternatively be written as x.rescale(%v), with a more object-oriented feeling. All in all, there are many advantages regarding UFCS, and this author wonders why it is not standard in all computer languages. Bjarne Stoustrup (the father of C++) has proposed to implement UFCS in the C++ language too, so the idea probably has no obvious drawbacks.

Another question regarding functions in Gekko 3.0 is the use of optional time parameters in functions. For instance, the above-mentioned rescale(x, %v) function could allow a time period, for instance rescale(x, %t1, %t2, %v), where %t1 and %t2 represent the time period — which could be written x.rescale(%t1, %t2, %v), too. But the function should also be callable without time paramaters, in which case the enclosing time period could be used (either the local period in a Gekko command, or the global Gekko time period). Since periods are fundamental in Gekko, functions in Gekko 3.0 allow an alternative (and better) way of stating time periods: rescale(<%t1 %t2>, x, %v), or written with UFCS: x.rescale(<%t1 %t2>, %v). Note here that the Gekko UFCS implementation counts the first argument as the first argument that is not a <>-period. This special treatment of <>-periods is convenient, because in that way, the periods are always written as the first argument, like Gekko commands. But this is not the end of the conveniences. When defining a user-defined function that can be called with, for instance, x.rescale(<%t1 %t2>, %v), the %t1 and %t2 time parameters are obvisously used inside the function (why else use them as arguments?). But what if the rescale() function is defined with time parameters, but called without them, like x.rescale(%v)? In that case, Gekko will hook the time parameters inside the function to the enclosing time period. This is similar to a command like PRT <2020 2025> x, which uses the period 2020-25, whereas PRT x uses the global time period. One nice thing about optional time parameters is that they can always later on be added to a function, without breaking existing code.

]]> 0
String interpolation Fri, 07 Jun 2019 22:06:47 +0000

Strings are represented with single quotes in Gekko (for instance: ‘this is a string’). Some languages use double quotes, and other languages allow both variants (or use single quotes for characters, and double quotes for strings). The use of single quotes for strings in Gekko has at least one convenience: double quotes are allowed inside a Gekko strings, and will be treated just like any other character (this is sometimes practical).

Anyway, this post is not about single quotes versus double quotes. It is about strings inside strings, or how strings can be used to compose other strings. The strict way of doing this is, for instance, %c = ‘red’; %s = ‘a ‘ + %c + ‘ car’;. This will be the same as ‘a red car’, but using the ‘+’ for chopping up a string is not very readable, especially if there are many inserts like %c. In Gekko versions prior to 3.0, it was possible to use %s = ‘a %c car’ instead. Allowing such syntax is of course tempting, and it is even one of the reasons sigils (type symbols like ‘%’) are used in some computer languages. But as the case often is: the devil is in the details. What if %c is not surrounded by blanks? It may be surrounded by ‘%’, ‘|’, ‘{‘, ‘}’, and so on, and it is not always easy to be sure that this works as intended in all corner cases.

Therefore, it was decided to use string interpolation instead, where string interpolation means inserting a string inside another string by means of certain placeholders, in practice implementing something akin to the f-string concept known from Python. So in Gekko 3.0, you should use %s = ‘a {%c} car’, where the {}-curlies is the placeholder. In Gekko 3.0, any expression may be put inside these {}-curlies, as long as the inside evaluates to a string or value. Therefore, the following is possible: %c1 = ‘r’; %c2 = ‘ed’; %s = ‘a {%c1 + %c2} car’;, where the inside of the {}-curly evaluates to ‘red’. Of course, ‘a {%c1}{%c2} car’ could be used instead, so the example is just to illustrate that expressions are allowed. Another example could be %t = 2020; TELL ‘The {%t} value is: {y[%t]/1000}’;, where y is a timeseries. You may even use {}-curlies inside {}-curlies, for instance: %s = ‘a{‘x{‘y’}z’}c’; which is that same as %s = ‘axyzc’;.

This is all very good, but the best part of it is probably that names and strings in Gekko 3.0 are very easy to convert from one to another. For instance, in Gekko 3.0, x{%i}z is a variable name, and if %i = ‘y’, the name is xyz. If you need to convert this name into a string, you just add the single quotes: ‘x{%i}z’, which is the same as ‘xyz’. Conversely, the string ‘x{%i}z’ can be converted into x{%i}z by simply removing the quotes. In practice, this conversion arises quite often, for instance because a command or function only accepts a string rather than a name. An example could be a command like y = y + x{%i}z; where it is not certain if x{%i}z exists (and if it does not, y should be unchanged). There is an exist() function that checks for just that, but this function only accepts strings. So the syntax becomes IF (exist(‘x{%i}y’) == 1); y = y + x{%i}z; END; which easy to understand. Another example is that a list of strings like #m = (‘a{%i}b’, ‘x{%i}y’) can alternatively be written as a naked list, in the form #m = a{%i}b, x{%i}y, where the parentheses and quotes are dropped.

All in all, using string interpolation with {}-curlies inside strings is convenient and consistent, and as an extra convenience, the same kind of syntax is used regarding name-substitution, that is, {}-curlies inside variable names. You can even tell Gekko to format the {}-curlies in particular ways, for instance number of decimals (if the inside is a value), cf. OPTION string interpolate format.





]]> 0
Array-series Fri, 10 May 2019 16:46:52 +0000

Array-series have not been described in this blog until now. The concept of array-series arose as a consequence of database integration on the one side, and GAMS integration on the other side. When interfacing with a particular Danish online database (Statistikbanken), there was the question of dimensionality. The database consists of a number of tables that are SQL-like in nature, that is, with columns describing dimensions or other attributes, and rows that represent “records”, or observations. For instance, a population table may consist of columns sex, age, and year, combined with rows that represent records/observations, for instance that there were 35082 Danish women of age 40 in the year 1990. To store such information, one could construct a number of timeseries containing these numbers, for instance pop_w_40 representing the 40-year old Danish women, with the value pop_w_40[1990] = 35082.

Using names like that could be called name-composition, since parts of the names represent the dimensions. This can work ok, but there is the question of whether to use for instance a ‘_’ delimiter in the series names, or a more terse name like popw40. Regarding the latter variant, there is the problem of name collisions. Consider for instance an input-output matrix, where anqz could represent deliveries from a sector ‘n’ to a sector ‘qz’. This is ok, if there is not a sector ‘nq’ and ‘z’, with a delivery between them, because in that case, the name would also become anqz. Using underscores, the names would be different: a_n_qz versus a_nq_z, but such names are not exactly aesthetical.

Instead of this, Gekko 2.4 and especially 3.0 proposes an array-series alternative, using brackets, where a_n_qz would become a[‘n’, ‘qz’], and a_nq_z would become a[‘nq’, ‘z’], and in the population case, pop_w_40 would become pop[‘w’, ‘40’]. The array-series indexes are strings, but these strings may also be written in ‘naked’ form, losing the quotes: a[n, qz], a[nq, z] and pop[q, 40]. This is possible because it would not make sense for a timeseries to be used as an index regarding an array-series.

Using for instance pop[w, 40] instead of pop_w_40 or popw40 is perhaps more readable, especially if there are many dimensions. But there are other advantages regarding array-series. If pop is defined as an array-series, it will function as a container/storage for all the sub-array-series, bundling them together inside one object. So pop[w, 40] is an integrated part of the array-series pop, whereas pop_w_40 is just a stand-alone timeseries that does not belong logically to anything (other than residing in a databank). Therefore, an array-series like pop may be thought of af kind of a complete table, with each sub-arrayseries representing a row in that table. This makes it possible to perform operations on the whole table/array-series as one object, for instance copying, deleting, printing it, etc.

In addition to this, array-series allow optional domain-restrictions on the dimension, so that, for instance,  setting pop[f, 40] would issue an error, since the domain corresponding to the first dimension does not contain the element ‘f’ (women are called ‘w’). This counteracts misspellings etc., and restricts the dimension elements, so that they cannot just attain any string value. All the information regarding an array-series (number of elements, dimensions, domains, etc.) can be shown with the DISP command, in contrast to using name-composition, where such meta-information is only implicit.

Array-series support easy summing up of elements, for instance sum(#a, pop[w, #a]), where #a is a list of strings representing ages in the pop-array-series. If #a represents the age domain, restrictions (filtering) can be set with ‘$’, for instance sum(#a, pop[w, #a] $ (#a in #a4049)), where #a4049 is a subset of #a, for instance the age group 40-49. This is very similar to GAMS syntax, where the dimensions of variables and parameters are typically assigned to sets. The following syntax is also legal in Gekko, and looks even more like GAMS: sum(#a, pop[w, #a] $ (#a4049[#a])), but the $ (… in …) syntax is probably easier to read. Using ‘in’ in this way is used in Python and other languages, for instance (in Python): “if x in y:” or “for x in y:”, where the first one checks if x is a member of y, and the second one loops x over the elements of y.

Gekko also supports for instance printing of a whole arrayseries, for instance “PRT pop;”. This will print all the elements of the arrayseries, instead of having to state, for instance, “PRT pop[#s, #a];” explicitly. Simple mathematical operations on array-series are also possible, for instance “PRT pop*income;” where pop and income are two compatible array-series with the same dimensions. This will be further developed, providing a kind of algebra regarding array-series.

It should be mentioned that Gekko’s array-series are sparse: it is not the potential dimensionality of the array-series that matters regarding memory use, but only the number of concrete sub-series stored inside the array-series. So a Gekko array-series can be thought of as a table where the columns are the dimensions (excluding the time dimension), and where each row represents a sub-array-series. These sub-array-series are just normal timeseries, with data in the time dimension. For instance:

  Sex        Age     Value
  w          40      Timeseries
  w          41      Timeseries
  w          42      Timeseries

So pop[w, 40] is a (sub-)timeseries, defined over some time period (for instance 1990-2019), and pop[f, 41] is another timeseries, defined over the same time period. In a software system like GAMS, the data would be represented like this:

  Sex        Age     Year     Value
  w          40      1990     35082
  w          40      1991     35222
  w          40      1992     33596

Above, the the last column is the actual data. So in GAMS, pop(‘f’, ‘40’, ‘1990’) = 35082, whereas in Gekko, pop[f, 40] is a (sub-)series, where pop[f, 40][1990] = 35082. When summing up and printing, Gekko has some options to control how to handle elements (combinations) that do not exist. For instance, in 1990 there are no women of age 108 or more. There is an option “OPTION series array calc missing” to control this, where the default value is that Gekko will abort with an error, if an element is missing. Instead, the option can be set so that a missing element is represented as zero, thus not being counted when summing up. When printing, the “OPTION series array print missing” can be used, so that missing elements can be printed as missing values, zeroes, or alternatively simply skipped.

A table like the one shown above where the values (data cells) are additive can also be understood as (part of) a multidimensional OLAP-cube, cf. the illustration accompanying the blog post. At some point, dataframes (known from R (dplyr), Python (pandas) etc.) will also be introduced in Gekko. Dataframes are much like SQL tables, and they would ease the way data could be exchanged between Gekko and R, Python, etc.

When designing the array-series syntax, the AMPL syntax was also considered, but the GAMS syntax is actually quite compact (but on the downside less similar to mathematical notation than AMPL). For summing up with restrictions ($-operator), GAMS both supports the syntax corresponding to sum(#a, pop[w, #a] $ (#a in #a4049)), or the alternative sum(#a $ (#a in #a4049), pop[w, #a]), where in the first case, the loop is run over all elements in #a (setting some of these elements to 0), whereas in the second case, the loop itself only runs over a subset of #a, thus skipping some of the elements (both these variants are shown with Gekko-syntax). The latter variant is not implemented yet. The result would be the same, but the latter variant would run more efficiently (faster).

All in all, array-series are very convenient, both for data-manipulation (if data is multi-dimensional), for interfacing with databases or other software packages, and last but not least: for modeling. Gekko does not yet implement array-series in the modeling part of the software, but this will come sooner or later (hopefully sooner: until then, Gekko interfaces tightly with GAMS, and Gekko reads and writes gdx files and can also understand GAMS equations).

Note: a database or GAMS gdx file may contain data that is not in the time-domain, for instance parameters, constants, etc. Gekko will represent such data as an array-series where the sub-series are timeless. For convenience, it is possible to define a timeless timeseries in Gekko (such a series essentially behaves like a value (val scalar)).

]]> 0
Use of {}-curlies Fri, 03 May 2019 10:28:37 +0000

In Gekko, {}-curlies have been used for quite a long time now. The idea is that variable names can be composed by means of the {}-curlies, creating a means of composing names dynamically, in a sense forwarding from a string to a variable name corresponding to that string. For instance, if %s = ‘x’, the expression {%s} would refer to the variable name x, not the string ‘x’. In many computer languages, such name-composition is not possible, and the possibility of name-composition is one of the more popular features of Gekko (it should be mentioned that AREMOS, EViews and others implement this feature, too).

In Gekko 2.x, the use of {}-curlies had become a bit fuzzy, since they could sometimes be omitted, and sometimes not. In addition, Gekko 2.x contained the monstrosity of the distinction between strings and names, originally inspired by the AREMOS distinction between strings and literals. This monstrosity (= the name type in Gekko 2.x) has been eradicated in Gekko 3.0, and hence whenever a string is supposed to be used as a name, the user must always enclose the string inside {}-curlies, no exceptions. Therefore, if %s = ‘x’, a statement like PRT %s will just print the string ‘x’ on the screen, whereas a statement like PRT {%s} will look for the variable x (a timeseries), and print that timeseries on the screen.

In principle, with a string inside a {}-curly, the logic is straightforward. The {}-curly performs a forwarding operation, looking for the variable corresponding to the string inside the {}-curly. So for instance, if %s = ‘x’, the statement PRT {%s} is exactly the same as PRT x. Now we can try inserting: PRT {%s} becomes PRT {‘x’}, which becomes PRT x, from which it is seen that the {}-curlies are in a sense quote-eaters, transforming {‘x’} into just x. This is a bit of a limited way of looking at {}-curlies, but there is some truth to that interpretation. This is also seen regarding name-composition, for instance the name a{%s}b, where a similar logic can be applied: a{%s}b becomes a{‘x’}b, which becomes axb, and again: the curlies have eaten a pair of quotes.

This also means that {}-curlies with a string inside can be understood as simply a sequence of characters (without quotes), just as if {…} had been the name abc. This leads us to the so-called abc-test. If in a particular command it is natural to use the name abc, then you should also use for instance {%s}, whereas if it is natural to use the string ‘abc’, you should use omit curlies and just use the raw string, %s.

Beware though that the {}-logic is not quite that simple when it comes to lists of strings. In order to use {}-curlies, the inside of such a curly must evaluate to a string or to a list of strings (actually integer values are also allowed, but let us think of them as strings). We have already covered the case where the inside of the {}-curly is a string, so let us move on to the list case. Consider the list #m = (‘x’, ‘y’), or defined with a naked list as “#m = x, y;”. If we issue a “PRT #m;”, we will just get the strings ‘x’ and ‘y’ printed on the screen, whereas “PRT {#m};” prints the variables x and y (two timeseries), just like the statement “PRT x, y;”. There is more to it though, because you may combine a list with fixed parts, as for instance “PRT a{#m}b;”, which is the same as “PRT axb, ayb;”. You may even use several lists inside one name (like for instance a{#m1}b{#m2}c: in that case, the lists are combined, so if #m1 has n1 elements, and #m2 has n2 elements, the resulting print will show n1·n2 variables.

So to sum up: if the inside of a {}-curly is a string, the {}-curly returns the characters of that string (without quotes), possibly glueing them together with other name parts. The result of this will be a variable name. If the inside of a {}-curly is a list of strings, Gekko will also use the characters of these string elements (without quotes), and possibly glueing these together with other name parts. The result of this will be a list of variable names.

Image note: the image depicts a so-called Tuborg hat, used on trucks delivering Tuborg beer. In Danish, {}-curlies are often called “tuborg” because of the similar shape.

]]> 0
Lists and naked lists Fri, 03 May 2019 10:14:28 +0000

As mentioned in the overview post regarding Gekko 3.0, lists were generalized in Gekko 3.0, using primarily Python as the inspiration. In Gekko 2.x, lists could only contain strings, so in that sense, they were quite simple. Gekko 3.0 allows any variable to be an element of a list, including other lists. Therefore, lists can be nested, and can contain strings, dates, values, series, etc. So in Gekko 3.0, lists may contain series objects, but it should be noted that since series typically live in databanks, it is often more convenient to store names of these series (in the form of strings), rather that the series themselves.

The general form of a Gekko 3.0 list is “(element1, element2, … , element_n)”, so the list elements are put inside parentheses, and separated with commas. For instance, (‘a’, ‘b’, ‘c’) is a list of three strings, (1, 2, 3) is a list of three values, and (‘a’, (1.5, 2010q3)) is a nested list with mixed types (string, value and date). This is consistent, but the following uses become cumbersome:

#m = ('a', 'b', 'c');
x = (1, 2, 3);

In the first line, we create a list of three strings and put this into a list called #m, and in the second line, we create a list of three values, and put these values into the series x. This syntax may not seem like a big deal, but definitions of lists of strings in Gekko can be quite long, and assigning values to a series is also such an frequent occurrence that the mandatory parentheses become a nuisance.

Therefore, Gekko 3.0 operates with the concept of naked lists. Naked means that they are without parentheses, and that any strings inside them are without quotes. Using such naked lists, we may write:

#m = a, b, c;
x = 1, 2, 3;

For long Gekko command files with many lines like that, the possibility of using naked lists is a relief. Naked lists may seem simple in cases like above, but beneath the surface, there are some details to take note of. First of all, a naked list can only be put after the equals sign in an assignment, or in a FOR loop (for instance: “FOR string %i = a, b, c; … END;”). Apart from that, naked list elements can only be of the following kind:

  • normal names like a1 or 1a, even -a1 is legal (becomes ‘-a1’)
  • series names with bank, frequency and index (for array-series), like b:a!q[i, j]
  • Integers like 2 or 007, minus may be used
  • Floating point values like 1.2 or 1.2e5, minus may be used. Missing value can be stated with m().
  • {}-curlies  may be used, for instance {%s}, a{%s}b, {#m}, a{#m}b, and any expression is legal inside the {}-curlies. In general, {}-curlies can be thought of as producing sequences of characters, but there are some special rules regarding these, see this blog post.

In addition to this, there are some special rules. A list like #m = 12, 02 will become a list of the two strings ‘12’ and ‘02’, and not a list of the two values 12 and 2. And a list like #m = 12, 1e5 will also become a list of two strings ‘12’ and ‘1e5’, and not the two values 12 and 100000. This is because both these lists could be codes (name-parts used for name-composition), for instance these name-parts could appear in variable names like x02y or x1e5y. If 02 and 1e5 were converted into 2 and 100000, respectively, such codes would become scrambled if used as strings afterwards (we would get: x2y and x100000y, which would be unexpected). This cannot be the case with a simple integer like 12, and a value like 1.2 or 1.2e5 could not be used as a code anyway, since such codes/name parts normally do not contain dots (R allows dots in names, and uses $ where all other languages use dots, but such strangeness of R is not going to be emulated in Gekko). So the special rules are to ensure that information is not lost or scrambled when using the list elements to compose names. List files use the very same rules, by the way, to keep the syntax and logic of naked lists and list files similar.

Like normal lists, naked lists allow the use of rep (repetition), for instance “#m = 1, 2 rep 10, 3;”, and like normal lists, a singleton list can be stated with the use of a trailing comma — for instance “#m = a,;” becomes a list with one element, namely the string ‘a’. If the comma is omitted, Gekko will fail with a type error (it will look for the series a, and even if such a series is found, it cannot be assigned to a list).

As stated in the post on the release of Gekko 3.0, life in Gekko 2.x was easier in some respects, for instance that an expression like “LIST m = a, #m2, b;” in Gekko 2.x is unequivocal. Since Gekko 2.x only allows strings inside lists (and not, for instance, series or lists), the statement has to mean creating a list consisting of first the string ‘a’, then the elements of the list #m2 (these elements can only be strings in Gekko 2.0), and finally the string ‘b’. In Gekko 3.0, this gets more complicated, since a 3.0 assignment like “#m = (a, #m2, b);” actually means finding the series a (as an object) and putting it into the first position in the list, then finding the list #m2 and putting it (the list itself, not its elements) into the second position in the list, and finally putting the series b into the last position of the list. So the list becomes nested: the second element of the list is another list. Nested lists have many uses, and it would be a cardinal sin to disallow them in Gekko. So now the question arises: In Gekko 3.0, how to create a list where strings and list elements are combined? There are basically three ways:

#m = a, {#m2}, b;
#m = list(); #m = #m.append('a').extend(#m2).append('b');
#m = ('a',) + #m2 + ('b',);
  • In the first variant where a naked list is used, the {}-curly makes sure that the elements (strings) of #m2 are added one by one, just like the statement “PRT a, {#m2}, b;” would do.
  • In the second variant, an empty list is created first, and then elements are added with the append() and extend() functions. Note that extend() is used for #m2. If append() was used for #m2, the list itself would be added, not its elements. The append() and extend() functions follow Python logic closely.
  • In the last variant, the ‘+’ operator is used, since this operator is understood as extend() regarding two lists. So between lists, you can just use ‘+’ to join their *elements* together, and for instance (‘a’,) creates a one-element list with the string ‘a’ (note the trailing comma).

Perhaps the first variant is easiest to use for such cases of joining strings and list elements together. But it should also be noted that the += operator can be practical, since it allows a naked list on the right-hand side:

#m = list();
#m += a,;
#m += #m2;
#m += b,;

This is yet another way of combining lists and strings in Gekko 3.0, where the += operator adds elements at the end of the list. Note the trailing commas in the two one-element naked lists. Trailing commas are borrowed from Python where they are prevalent regarding tuple definitions.

Besides trailing commas, there is another thing to remember about naked lists: they do not allow variables with type symbols ‘%’ or ‘#’. So do not expect a naked list like “#m = a, %s, #m, b;” to work: it will fail with a syntax error. Since a naked list like “#m = a, b, c;” is understood as “#m = (‘a’, ‘b’, ‘c’);”, a naked list like “#m = a, %s, #m, b;” should logically be understood as “#m = (‘a’, ‘%s’, ‘#m’, ‘b’);”, since the type symbols (sigils) are always just understood as a kind of special characters alongside normal characters like ‘a’, ‘b’, etc. (cf. this post). But such use would probably just create a lot of confusion, so therefore type symbols are not allowed in naked lists, except if they are located inside {}-curlies. As mentioned before, “#m = a, {#m2}, b;” is perfectly legal, and actually any expression can be put inside {}-curlies. But outside these curlies, type symbols cannot be used in naked lists. In any case, in practical use, long lists of names are typically lists of series names, not scalar or collection names, so the restriction if probably not inconvenient in any practical way. And if the user really wants a list like #m = (‘a’, ‘%s’, ‘#m’, ‘b’), it can just be stated that way (with parentheses and quotes). Such a list could have its uses, for instance “PRT {#m};” would print the four variables inside the quotes.

Lists and naked lists will probably take some getting used to for Gekko 2.x users. The syntactical complications stem from the fact that lists in Gekko 3.0 allow all kinds of elements, including other lists, and Gekko 3.0 tries to comply rather tightly with the list logic of Python. Of course, one-dimensional non-nested list are in a sense easier to use and understand, but multidimensional nested lists are immensely practical for a lot of purposes. For instance, a list like ((1, 2), (3, 4)) can be thought of as representing a 2 x 2 matrix, and a list like ((‘a’, ‘b’), (‘c’, ‘d’)) can be thought of as representing a 2 x 2 table, and arrays of higher dimensions than 2 can be constructed in a similar way, and in this way different kinds of variables/data can be bundled together in very flexible ways. Elements can be fetched with indexes, for instance if #m =  ((1, 2), (3, 4)), the index #m[2][1] would fetch the value 3.

If the user would rather like to fetch elements by names, like for instance #m[‘second row’][‘first col’], a map variable could be used instead. But this would be the subject of a another blog post…

]]> 0
The dynamic problem Thu, 02 May 2019 07:59:47 +0000

In Gekko 2.0/2.2/2.4, and also in some other software packages, series expressions are run in an outer loop. Consider a series expression like y = x1 + x2 + x3. In AREMOS, such an expression is run n times in an outer loop, where n is the number of periods. Imagine that the series values over the period 2010-12 are given as the vectors x1 = (1, 2, 4), x2 = (3, 5, 7), and x3 = (6, 8, 9). Then in AREMOS and Gekko 2.x, the following is run:

y[2010] = x1[2010] + x2[2010] + x3[2010] = 1 + 3 + 6 = 10
y[2011] = x1[2011] + x2[2011] + x3[2011] = 2 + 5 + 8 = 15
y[2012] = x1[2012] + x2[2012] + x3[2012] = 4 + 7 + 9 = 20

In contrast, Gekko 3.0 treats an expression like y = x1 + x2 + x3 more vector-like:

y  =  (1, 2, 4) + (3, 5, 7) + (6, 8, 9)  =  (1, 2, 4) + (9, 13, 16)  =  (10, 15, 20)

So Gekko 3.0 first adds x2 and x3 together, creating an temporary vector with values (9, 13, 16). When this temporary vector is added to x1, the final result (10, 15, 20) is obtained. If an expression like x1 + x2 + x3 is fed to a function that later on uses a lag on that expression, we have to be sure in advance that the temporary series x1 + x2 + x3 is calculated over a sufficiently long time period. This is the lag problem.

Another problem is the dynamic problem. Imagine an expression like x = x[-1] + 1. In AREMOS and Gekko 2.x, such an expression is performed in the following way over the period 2010-12:

x[2010] = x[2009] + 1
x[2011] = x[2010] + 1
x[2012] = x[2011] + 1

Imagine that x over the period 2009-12 is (3, 5, 8, 11). Since the equation accumulates, only the 2009 value (x[2009] = 3) is relevant, and the result is (4, 5, 6) over 2010-12, growing by 1 for each period.

But in Gekko 3.0, an expression like x = x[-1] + 1 is run differently. Over the period 2010-12, we have x = (5, 8, 11) and x[-1] = (3, 5, 8). It is seen that the lag simply shifts the values one position to the right in the vector. The expression is thus evaluated as follows:

x  =  x[-1] + 1  =  (3, 5, 8) + (1, 1, 1)  =  (4, 6, 9)

The first values is actually the same as in AREMOS and Gekko 2.x, but the following values are different. Because the values are not calculated period for period, the expression does not accumulate dynamically, augmenting the value by 1 each period. This problem is alleviated in Gekko 3.0 with an optional <dyn> option. So in Gekko 3.0, you can use x <dyn> = x[-1] + 1, and in that case, the statement is run period for period in an outer loop. This option can also be set via a “BLOCK series dyn = yes; …  ; END;”.

It would perhaps be tempting to set <dyn> mode on all assignments, completely emulating the way AREMOS and Gekko 2.x works. But using such an outer loop is a tremendous waste of effort, since Gekko can add two vectors/arrays like (3, 5, 7) + (6, 8, 9)  into a new vector (9, 13, 16) very efficiently compared to an outer loop. Another complication is that a naive implementation of an outer loop should not encompass for example a scalar or matrix expression like %x = %x + 1 or #x[1, 1] = #x[1, 1] + 1, running this expression n times (that would be a grave error: a scalar or matrix expression should of course only be run 1 time).

The dynamic problem only arises when the left-hand side variable is a timeseries that is also present on the right-hand side (and is lagged there). For an assignment like x = x[-1] + 1, this is self-evident, and in order to implement dynamic behavior on all assignments, the following was tried. When variables were accessed on the right-hand side, these variables were rememeberd, if they were of series type. Next, if the left-hand side variable was identical to one of the right-hand variables from the remembered list, the assignment was aborted and done dynamically instead. This seemed reasonalby watertight, but still quite cumbersome, complicating the internal source code quite a lot. When this was almost implemented, the following problem arose: how about an expression like x = x[-1] + #m, where #m is a list of values? This would readily be recognized as an assignment that needed to be run dynamically. But then, for each single period executed in an outer loop, Gekko would need to somehow pick the right element of #m (the first one in the first period, the second one in the second period, etc.). This would add another layer of complexity, and would be brittle and error-prone, too.

So, in the end, it was decided that the users have to indicate “manually” if certain assignment expressions are to be run dynamically, period by period. Either via the <dyn> option, or with a “BLOCK series dyn = yes; … ; END;”. Using a block makes sure that the dynamic behavior is switched off again, not accidently infecting parts of the code where this behavior is not needed or intended.

All in all, Gekko has quite a lot of operators regarding dynamic assignments, for instance x ^= 1 or x %= 1, either adding 1 (absolute) or 1% (relative) each period, but sometimes the dynamics may be more complicated than that, especially if model equations are used in command files. In theses cases, the <dyn> option or the above-mentioned BLOCK can be used, and the prevalence of such cases is probably not high enough to warrant general AREMOS-like behavior, always calculating series assignments period for period.

]]> 0
Gekko 3.0 Wed, 01 May 2019 13:34:57 +0000

The official Gekko 3.0 is now released.

This is a long post, but the intent is to try to explain what Gekko 3.0 really is about. Which is actually not so easy to boil down exactly: it is perhaps best to think of it as a long-term vision, borne out of the realization that Gekko 2.0 (or adaptations hereof) was not really up to task of realizing that vision. And what is that vision? It can perhaps be summed up as trying to move Gekko into the realm of more general data-wrangling and -manipulation, while still keeping a firm timeseries-oriented foundation.

Gekko versions up to and including 1.8 were inspired by PCIM, versions 2.0-2.2 were inspired by AREMOS, Gekko 2.4 was inspired by GAMS, and it could perhaps be said that Gekko 3.0 is still inspired by GAMS, but also to a large extent by Python. Does a Gekko 3.0 program look like Python? No. But take for instance a look at lists in Gekko 3.0, including the use of trailing comma for singleton lists, or the in-built list functions (most of these are borrowed from Python). It should be mentioned that R, Matlab and Julia have also been looked at regarding version 3.0, but the main inspiration probably stems from Python, especially regarding the list and map variable types. Also, the syntax of Python has been an inspiration (the syntax of R is quite peculiar and somehow feels old-fashioned compared to Python).

Similar to the birth of Gekko 2.0, the birth of 3.0 was a prolonged affair. Around the summer of 2017, it was evident that there was some unfortunate fuzziness regarding the Gekko 2.0 syntax. Sometimes Gekko would accept a name (like x), sometimes a bank and name (like b:x), sometimes an expression (like x + y), and sometimes the ‘%’ or ‘#’ type symbols should be omitted. In order to save a few keystrokes or symbols here and there, the syntax logic had become needlessly complicated in Gekko 2.0, and a string scalar like for instance %x could refer to a timeseries in some contexts, or a string in other contexts. So it was quite evident that the syntax needed some sort of cleanup.

It was also clear that it was natural to allow other variable types than timeseries into the databanks, so that scalars, lists, matrices, etc. could be stored together with timeseries. In addition, the so-called array-series (series callable by index, for instance x[‘a’] instead of xa, or x[‘a’, ‘b’] instead of xab) inspired by GAMS syntax needed to be further developed, because in reality the array-series were squeezed into Gekko 2.4 as an afterthought. Since databanks store all kinds of variables in Gekko 3.0, it was natural to also generalize lists, so that these can also store all kinds of variables (including lists), and there is even a databank-like variable type (map) in Gekko 3.0 that stores variables by name (such a map is called a dictionary in Python, or a named list in R).

As a last but not least point, the way timeseries were handled internally had also grown problematic. In versions prior to 3.0, series statements were handled differently from other assignments. A series assignment like “SERIES y = x1 + x2;” would be run n times in an outer loop, where n was the number of periods in the time period (for instance y[2010] = x1[2010] + x2[2010], followed by y[2011] = x1[2011] + x2[2011], etc.). In Gekko 3.0, a statement like y = x1 + x2  runs more like vector addition, without any outer time loop (more on this here). This means that expressions containing series variables can flow much more smoothly in and out of functions and procedures, and it is no longer necessary to sustain an artificial distinction between series expressions and non-series expressions (or series expressions versus array-series expressions). In Gekko 3.0, an expression is an expression, and expressions can be used anywhere. So assignments in Gekko 3.0 are in reality of only one kind: assigning a right-hand side expression (of any kind) to a left-hand side variable (of any kind). This does not mean that you can assign, say, a list to a scalar (which will issue a type error), but syntactically, all assignments are equal in Gekko 3.0.

So Gekko 3.0 started out as a modest wish of cleaning up the syntax, handling series and array-series more like objects/vectors, redesigning the databank structure, and allowing more flexible use of lists etc. But soon the complications began to creep in. For instance, if lists are no longer just lists of strings, a Gekko 2.0 statement like “LIST m = a, #m1, b;”” is no longer straightforward to interpret. How can we know that a and b are supposed to represent strings (they could be series objects), and how can we know that it is the elements of a list #m1 that are to be added, and not the list itself? So the flexibility of the 3.0 lists somehow had to be reflected in the syntax (see more on lists and so-called naked lists here). Likewise, in Gekko 2.0, a statement like “PRT b:#m;” is pretty straightforward: find the list #m, and print the series corresponding to the string names in #m, where all of these series are taken from the b databank. But in Gekko 3.0, a databank may contain lists, so b:#m actually refers to the list #m in the databank b. Therefore, in Gekko 3.0, we need to use {}-curlies to do this: “PRT b:{#m};”. This indicates that it is not the list #m itself that we want to have printed: it is the variable names referred to by the string elements in #m that we are interested in (and these names are to be taken from the b databank). Note incidentally, that “PRT {b:#m};” is still different: in that case we will be fetching the #m list from the b databank, and printing the variables corresponding the these string names (see more on {}-curlies here).

Syntax questions also arose regarding the special type symbols (‘%’ and ‘#’, also called sigils, more on those here and here) and where they should be used, and regarding the choice of frequency indicator (‘!’ symbol, more on this here). Worth noting is also that the syntax of and logic of wildcards had to be adjusted too, to reflect the fact that variable names in databanks may now contain ‘%’, ‘#’ and ‘!’ symbols. All this also had to comply rather tightly with GAMS syntax, in order to ease the use of array-series and being able to interface with GAMS in a convenient way.

All in all, these changes had many impacts throughout the Gekko source code. The source code has become quite a lot clearer in Gekko 3.0, but large parts of it had to be rewritten, especially those parts that accepted series, series expressions or wildcards as input. Stumbling blocks on the way was also the lag problem, and the dynamic problem (regarding the latter: in Gekko 3.0, an expression like “x = x[-1] + 1;” will not accumulated period by period, unless a <dyn> option is used). But all in all, treating series expressions more like vector algebra is a big advantage that will benefit the further development of Gekko tremendously. Without this, user-defined functions and procedures would never become a smooth experience, and perhaps this is the very reason that neither EViews nor AREMOS implement fully featured user-defined functions?

For the casual user, the syntax and functionality changes of Gekko 3.0 are probably not earth-shattering. The user has to remember to always use type symbols ‘%’ and ‘#’ for scalars and collections, and {}-curlies must be used in some cases where they could be omitted in Gekko 2.0. There are other small differences, and the more prevalent {}-curlies will probably take some getting used to. But on the flip side, Gekko 3.0 is much more flexible regarding what can be put into, for instance, a list or a databank, and there are a lot of new functions and capabilities that benefit from the tightened syntax of 3.0. But what is perhaps most important of all: the changes in 3.0 make the use of functions, procedures and larger systems of command files a much more seamless experience, and interfacing with packages like for instance Matlab, GAMS, R, Python, Julia etc. will be much easier to implement in Gekko 3.0 than woud have been the case with the 2.0 series.

To sum it up, to this author, Gekko 3.0 feels like a fully consistent system regarding syntax, functionality and internal logic. In contrast, the 2.0/2.2/2.4 versions are less consistent, and also have some quirks and peculiarities that would sooner or later show themselves, limit the capabilities of the software, and hamper the future interoperability of Gekko and other software packages. There was a need for modernization of the core parts of Gekko, which involved rewriting large parts of the Gekko source code, but if this was not done now, it might not have been feasible to do later on. Both because the core parts of the Gekko 2.0/2.2/2.4 source code would grow more and more complicated with exceptions to rules and exceptions to exceptions to rules etc., but also because the users would begin to build larger systems on 2.0/2.2/2.4 that would later on become a major headache to migrate to a modernized Gekko.

In the future, Gekko will probably implement dataframes, table objects and other convenient datastructures too, and combined with the flexible and consistent nature of Gekko 3.0, it is the hope that general data wrangling, -manipulation and -aggregation could become one of Gekko’s strong points, too.

]]> 0
Rewriting a software system Mon, 04 Jun 2018 14:23:01 +0000

Gekko 3.0 is due to be officially released in the autumn of 2018. Version 3.0 entails a rewrite of large parts of Gekko, and the reader may ask him/herself why this is really necessary?

As explained in this blog post, Gekko started out (in 2008) just interpreting characters and strings, and later on tokens (‘words’). In 2010, a real parser was used (ANTLR), where the parser spits out a tree structure that is converted into runnable C# code. In 2014, when Gekko 2.0 was introduced, both the parser and converter was rewritten from scratch, mostly because Gekko 2.0 introduced a lot of syntax changes. Gekko 3.0 also introduces some syntax changes, but not that many – it is more a question of new functionality. Still, both the parser and converter is rewritten again for Gekko 3.0. Why?

In principle, the new functionality of Gekko 3.0 could have been introduced using the ‘old’ parser/converter of Gekko 2.0. But as mentioned in this blog post, a new rewrite was deemed essential for the long-time well-being of Gekko. Continuing to build upon the engine of 2.0 would have made the source code a mess in the longer run, with work-arounds and exceptions to general rules (and exceptions to these exceptions).

Rewriting the parser and converter from the ground up is a major hassle, and as a software system develops, it becomes more and more difficult and time-consuming to do it. Also, if the syntax changes, the users will have to rewrite code, too. This partly explains why some computer languages somehow seem frozen in time. An example could be R, which has a fantastic array of user-supplied libraries, but where some of the syntax and structures seems pretty strange and arcane at first sight. R has a lot of advantages, and a lot of mature libraries for statistics and plotting, but in the very long run a language like Python may win out simply because of its clean syntax and general “easy” and modern feel.

It is probably the case that the longer a software system lives on, the harder it gets to rewrite its core parts, including the syntax. And as mentioned, if the syntax changes, the users will have to rewrite code, too. Rewriting the core parts of Gekko 3.0 had a bit of a suicidal feel to it at the beginning, but there was a simultaneous feeling that this rewrite would be important, and perhaps the last chance before the users start to build really large systems with user-defined functions etc.

So is this the last major rewrite of the inner parts of Gekko? Regarding the core syntax, it may be. It is my belief that the syntax is now (version 3.0) really logical and consistent, and has become, in a sense, the best version of itself. For timeseries, it deals smoothly with timeseries of differing frequencies, handles the time dimension and sample settings equally smoothly, uses a very convenient databank scheme, and handles multidimensional timeseries as an inbuilt feature.

So how is version 3.0 doing? Since the winter 2017/18, some adventurous people have been using it, and in that sense it is alive and doing well. But in order to release it for general use, a lot of testcases from version 2.2/2.4 need to be checked/fixed in 3.0, and this is mainly what is going on at the moment. The hard parts regarding 3.0 have been done, but a lot of details (and a translator from 2.2/2.4 to 3.0) need to be finished.

The reward? Consistent syntax and data structures, and a healthy foundation regarding the further development of Gekko (instead of a syntax with ambiguities and gothas).

]]> 0
The lag problem Thu, 09 Nov 2017 16:01:26 +0000 One of the main reasons for the modernization of the parser for Gekko 3.0 was that it was deemed important to handle timeseries as objects in Gekko. Timeseries are already objects in Gekko 2.0/2.2, but they are not always handled as such during calculations. To understand this, think about two timeseries x1 and x2 as vectors of data. We may say that x1 = (1, 2, 3) and x2 = (2, 3, 4) over the period 2001-2003. If we need to create the timeseries y = x1 + x2 over the same period, we can just do a vector addition: (3, 5, 7) = (1, 2, 3) + (2, 3, 4). But this is not how it is done in Gekko 2.0/2.2. In that version, a time loop is performed, setting t = 2001, t = 2002, t = 2003, respectively. So first it is asked what the values of x1 and x2 are in 2001, these values are summed, and the result is put into y (in 2001). This is repeated for 2002 and 2003. This may sound innocent enough, but there are drawbacks to this strategy. For instance, it becomes less transparent how timeseries are transferred in and out of functions, and especially the use of lags becomes a bit of a headache. You might have a function f(x) returning 2*x[-1], and that method might be called with arguments f(x1+x2). To do this, a temporary series storing the result of x1+x2 would need to be generated (in a time loop) before running the function, and all this becomes a bit messy. The pinnacle of messiness was implementing moving averages and moving sums via such time loops, because these timeloops become nested if, for instance, the moving average is nested (like movavg(movavg(x, 2), 3)), but also just movavg(pch(x), 2) became complicated. Such problems are the probably reason that AREMOS does not allow lags in user functions.

Instead of this, it was decided to do timeseries expressions more encapsulated, like vector algebra. To return to the expression y = x1 + x2, Gekko 3.0 will not calculate the first element (3 = 1 + 2) and put this into y, then the next element (5 = 2 + 3) and put this into y, and so on. Instead, Gekko 3.0 will first calculate the right-hand side as a whole, and assess that this is equal to (3, 5, 7). Then, afterwards, Gekko will think about what to do with that sequence of values (in this case, put it into the timeseries y, over the period 2001-2003).

Handling timeseries expressions as a kind of vector algebra provides clarity and robustness to the internals of Gekko, but there are some issues to this way of handling them, the most pressing being how to deal with periods and lags (or leads). Imagine two large x1 and x2 vectors, containing 50 annual observations, and imagine that the user wants to “PRINT <2001 2003> x1 + x2;”. If x1 and x2 both run from 1954 to 2003, adding them vector-wise creates a 50-element timeseries running from 1953-2003. This is a lot of waste of space and effort to calculate, if in the end we only want to print the three observations 2001-2003. So, the adding of x1 and x2 in Gekko should be smarter than that, exploiting the fact that only a small part of the result is going to be used. An obvious fix would be to just operate on the 2001-2003 parts of x1 and x3, omitting the rest. This is fine, but complexities creep in regarding the setting of this “relevancy period”. How about “PRINT <2001 2003>  (x1+x2)[-10];”? This time, only calculating x1+x2 over the period 2001-2003 will fail, since the result is going to be lagged 10 periods. So we should rather calculate x1+x2 over the period 1991-1993, which would, in the next step, conform with the 10 period lag. So the sum x1+x2 needs to know what is going to happen to it afterwards, whether it is going to be lagged/leaded, weather some observation is going to be picked out, etc. (for instance, (x1+x2)[1991]). So there are some dynamics regarding the time period in play, and in principle this is a hard problem. In principle, the addition x1 + x2 should not be performed until actually needed, which is called lazy or deferred evaluation in computer languages (R, for instance, uses lazy evaluation as default). Lazy evaluation is possible in C# (the language used for Gekko), but it is probably an overkill to use for this purpose.

In Gekko 3.0, the following will be implemented. When performing “PRINT <2001 2003>  (x1+x2)[-10];”, Gekko will tell the addition component that there is going to be a 10 period lagging afterwards, and hence the time period will be adjusted regarding this addition. Such calls may be nested, like “PRINT <2001 2003> (x1+x2)[-10][-1];”, and the logic needs to be able to deal with that, too. In 95% of cases, such logic is simple enough, but it is not watertight. What to do with this function: randomlag(x1+x2)? We could imagine that the randomlag function lags the argument with a random lag between -1 to -50, so there would be no way to know in advance which period the addition component should calculate x1 + x2 over. The only real solution to this would be using lazy evaluation (hard), or revert to the old looping of Gekko 2.0/2.2 (messy).

What would probably happen in such a case in Gekko 3.0 is that Gekko would issue an error, and advise the user to decorate the randomlag method with an indicator of the potential maximal lagging/leading going on inside of the method. If the user does not feel up to this task, there will be an option to perform full-sample calculations of x1+x2, that is, having Gekko simply calculate the vector sum over the full common sample of the two timeseries. This will cost some performance, but Gekko will run.

And, as noted, such cases where a function performs unknown lagging/leading are probably very rare. And even so, randomlag(x1), or randomlag(x1[-2]) would not be a problem, it is only when such a method is fed with an expression that the problem arises. The lag problem caused some delays in the deployment of Gekko 3.0, but the issue seems fixed for now. And the benefits of treating timeseries as a kind of encapsulated vectors will be evident in the long run, both in Gekko command files, but also in the internal Gekko source code (which can be simplified quite a lot).

]]> 0
Choice of frequency indicator in Gekko Thu, 09 Nov 2017 15:10:48 +0000 To denote a timeseries with a particular frequency, AREMOS uses the dot, for instance x.a is x in its annual version, x.q is quarterly, x.m is monthly, etc. In Gekko 3.0, the dot is already quite busy, since it is used to pick out elements in collections. For instance, #m.x will pick out the timeseries x from the MAP #m (think of #m as a kind of mini-databank), or alternatively #m[‘x’] can be used synonymously. Hence, the dot can be used to pick out elements from 1-dimensional collections, and this is going to be used a lot. Imagine an estimation procedure returning a map #m with all kinds of info, like #m.%r2, #m.residuals, #m.#beta — that is, a scalar with R2, a series with the residuals, a matrix (vector) with the estimated parameters, etc. Now, the timeseries also implement a map-like functionality in the form of so-called array-timeseries. Here, x[‘a’] will pick out the element ‘a’ from the 1-dimensional array-series x, whereas y[‘a’, ‘z’] picks out the element (‘a’, ‘z’) from the two-dimensional array-series y. For one-dimensional timeseries it would be quite obvious to say that x.a is short-hand for x[‘a’], just as #m.x is short-hand for #m[‘x’]. But this interacts badly with the use of dots as frequency indicator, and it would probably create confusion to use dots for both frequencies and element access. A way out of this could be to interpret x.a, x.q and x.m as dimensions of a more general array-timeseries x of some sorts (storing the frequencies inside), but it would be stretching the logic somewhat, especially if x is supposed to have other dimensions than frequencies. For instance, the user would not be able to use the notation x.q[‘a’, ‘b’] for accessing a quarterly array-timeseries, since this would be equivalent to x[‘q’][‘a’, ‘b’] which is not the same as as x[‘q’, ‘a’, ‘b’], because the former would be an array-timeseries with another array-timeseries inside.

In the light of the above, for Gekko 3.0 the frequency indicator will be ‘!’, for instance x!a for x in its annual version, x!q for quarterly, x!m for monthly, etc. At the moment, the ‘!’ is not used for other things, since logical difference uses ‘<>’ instead of ‘!=’, and Gekko will generally use the keyword NOT instead of ‘!’ to denote logical negation.

For AREMOS users, using ‘!’ will probably take a little effort getting used to, but another advantage of using ‘!’ instead of dot for frequencies is that the ‘!’ can be thought of as a sigil alongside the other sigils ‘%’ and ‘#’ (cf. this post and this post). The only difference is that the ‘!’ sigil must be at the end of the timeseries name, followed by the letter indicating the frequency. Hence, the “real” name of the annual timeseries x is really x!a, but if Gekko is running with annual frequency set, a statement like “PRINT x, y;” will intenally be translated into “PRINT x!a, y!a;”, so in most cases the user can omit the frequency indicator. The indicator only comes into play when frequencies are mixed, for instance printing or plotting mixed frequencies, or converting frequencies from one type to another (COLLAPSE/INTERPOLATE). For array-timeseries, the syntax x!q[‘a’, ‘b’] can be used, that is, the element (‘a’, ‘b’) of the quarterly array-timeseries x!q.

It is believed that the improved clarity of using x!a instead of x.a outweighs the disadvantage of getting the users used to the ‘!’. If it turns out to be awkward using ‘!’, Gekko may revert to dots (if possible logically and syntactically), and in that case converting existing command files from ‘!’ to ‘.’ will be an easy search and replace. On the contrary, if Gekko 3.0 starts out using dots for frequencies, and later on changes it to ‘!’, converting existing command files would be more of horror, since dots are used for a lots of other things in Gekko.

]]> 0