Chapter 2: Yodl User Guide

This section describes the yodl program from the point of a meta-user, one who is interested in how macro files work, or one who wants to write a new converter. If you're just interested in using Yodl with the pre-existing converters and macro files, skip this chapter and continue with the macro package description (chapter 4).

The Yodl program the main converter of the Yodl package. The basic usage of the yodl program, yodl's built-in macros, and the syntax of the Yodl language is described here.

2.1: Using the yodl program

Yodl reads one or more input files, interprets the commands therein, and writes one output file. The program is started as:
yodl options inputfile [inputfile...]
In this specification, the options are optional. Most options have `long variants' also, which are mentioned in the following list. In this list, -x, --optionname are two alternate ways to specify option x. If -x takes an argument, it may be specified immediately following the -x, but separating blanks may also be used. Options not taking arguments can be combined (e.g., -a -b -c may be combined to -abc). Arguments specified with long options should be separated from the long option using a = character.

The following options are currently available:

The inputfile elements on the command line specify which files Yodl should process. All names are supplied with an extension (this extension is defined in the installation of Yodl and is usually .yo). The files are then searched for in the directories mentioned in the include-path. Files may also be specified using absolute pathnames.

Note that all filenames on the command line are input files. To define an output file, either use the --output option or redirect the output.

2.2: The Yodl grammar

The grammar which is used by Yodl mixes `real' text that should appear on the output with markups: commands for Yodl. The markups must follow a certain grammar, which is described in this section. Yodl therefore falls in the category of `markup languages', in contrast to `WYSIWYG'-programs. As a consequence, Yodl promotes concept-oriented writing.

Basically, Yodl only does `something special' when it encounters the name of a builtin function or the name of a user-defined macro, followed by a parenthesized argument. Sometimes a function or macro requires multiple arguments, which must then be specified in sequence. All required parameter lists, however, must be specified within the same input file. It is not allowed to split the activation of a builtin function or macro over multiple input files. Plain text, on the other hand, may be split over multiple files.

In this section the elements of Yodl's grammar are briefly discussed.

2.2.1: Language elements

At the lowest level, Yodl's lexical scanner returns small pieces of information to its parser. These pieces of information are called tokens, and consist of elements like a blank space, a non-blank character, or an end-of-ile flag. These tokens are at too small an aggregation level to be useful for the current user-guide, so here we concentrate our discussion on the next aggregation level: compound elements and conceptual elements.

Compound elements relate to the basic tokens as words in a sentence to the individual letters of the words. These compound elements are identifiers, names, numbers, parameter lists and arguments.

Conceptual elements are found at the next higher aggregation levels: builtin functions are the buildin blocks for all of Yodl's functionality, symbols and counters are Yodl's variables, and (user defined) macros extend Yodl's functionality beyond those of the basic builtin functions.

In the coming sections these basic and conceptual elements are discussed in greater detail.

2.2.1.1: Identifiers and Names

Identifiers are names that can have a special meaning in the Yodl language. E.g., the sequence INCLUDEFILE is an identifier: when followed by a filename in parentheses, Yodl will take some special action (in this case, read the file as a Yodl-source file).

Identifiers may consist of uppercase or lowercase characters. No other characters may appear in them.

In particular, note that this diverts from the well known definition for identifiers used in most programming languages: identifiers may not contain underscores, nor digits. Yodl, therefore, won't accept identifiers like run4 or under_score.

Names are sequences of characters, not containing white space characters. (i.e., any series of characters not containing spaces, tabs or newlines). Names are allowed with certain builtin functions, liek the INCLUDEFILE function, expecting the name of a file as its argument.

2.2.1.2: Numbers

Numbers consist of digits and an optional minus sign. They are most often used for so-called counters. In some contexts (e.g. with the builtin function VERBOSITY 3.1.69, hexadecimal numbers are allowed. Hexadecimal numbers have 16 `digits': the familiar 0-9, but also a-f (or A-F), representing the decimal values 10 until 15, respectively. Hexadecimal values are usually prefixed by 0x, for example 0x4e.

In other contexts (in particular, with character tables 2.3), octal numbers or character constants are allowed too.

An octal number only consists of the digits 0-7. In Yodl, octal values must consist of three digits, and must be preceded by a backslash.

Character constants may very well be considered numerical values. Character constants consist of a character value between single quotes, for example 'a'.

Refer to section 2.3 for more detailed information about the use of octal values and character constants.

Yodl has no concept of floating point values nor does it have facilities for performing floating point arithmetic.

2.2.1.3: Arguments to builtin functions and macros

Arguments that are passed to Yodl builtin functions or user-defined macros must be surrounded by a pair of matching open- and close-parentheses. Each parameter list contains exactly one argument.

An argument is recognized when it immediately follows the name of a builtin function or user-defined macro. Some functions or macros expect multiple arguments. In those cases, the required number of arguments must be provided, possibly separated from each other by white-space.

For example, the following shows how to call the builtin function DEFINECOUNTER, expecting two arguments: 


    DEFINECOUNTER(MyCounter)()
    DEFINECOUNTER(MyCounter)    ()
    DEFINECOUNTER(MyCounter)(12)

Yodl recognizes the arguments of macros when delimited by ( and ) characters. As long as the numbers of opening and closing parentheses match, Yodl correctly recognizes the list. E.g., given a hypothetical macro somemacro, the following code sample shows the macro followed by one argument: 


    somemacro(Here is a chunk of text.)
    somemacro(Here is a some (more) text.)
An error occurs when the parentheses aren't balanced: i.e., when the argument consists of more opening than closing parentheses or vice versa To handle such situations, Yodl offers a `literal-character' mechanism (see the CHAR macro in 3.1.4) and a `global substitution' mechanism (see the SUBST macro in 3.1.62). For example, to send the text 
here's a ")" closing parenthesis
as an argument to our hypothetical macro somemacro, the following can be used: 

    COMMENT(-- Alternative 1: using CHAR --)
    somemacro(here's a "CHAR(41)" closing parenthesis)

    COMMENT(-- Alternative 2: using SUBST --)
    SUBST(closepar)(CHAR(41))
    somemacro(here's a "closepar" closing parenthesis)
Both methods have disadvantages: the CHAR method requires you to remember that an ASCII 41 is a closing parenthesis. The SUBST method defines a string closepar that is always expanded to a closing parenthesis, wherever it may occur in the text. But whatever method is used, it should be clear by now that unbalanced arguments can be handled by Yodl. Also, remember that unbalanced parenthesis pairs are only relevant in argument lists. Yodl handles parentheses in normal text as ordinary characters.

2.2.1.4: Builtin functions

The building blocks of Yodl's functionality are its builtin functions. Builtin functions exists to manipulate all of Yodl's builtin types (character tables, counters, macros and symbols) and to do basic bookkeeping and flow-control: it is possible to test values of counters and symbols, to include other input files, to generate warning and error messages, and to start child- or subprocesses. Each builtin function is described in a separate subsection of section BUILTIN 3.1.

2.2.1.5: Character translation tables

Character translation tables exist to perform conversion specific transformations. For example, in html, the \'e is written as é, but in LaTeX it's written as \'e. Rather than using a potentially long if-else ladder to determine how to set a particular character, a character translation table can be used. The character translation table of a particular conversion is then activated only for that type of conversion.

Character table translations are used very late during the processing of Yodl's input s: it is the output generator that handles the character translations. Consequently, macros or builtin function calls that might appear in a character's redefinition element of a character table are not expanded. In practice this never is a point of concern. In section 2.3 the use of character translation tables is discussed in detail.

2.2.1.6: Counters

Some document languages (notably LaTeX) automatically prefix numbers when typesetting sections, subsections, tables, figures etc.. Other document languages (e.g. html) don't.

Therefore, a macro package that converts a Yodl document to LaTeX doesn't need to provide the numbering of sections etc.. However, if you do want the numbering and if you want to convert documents to, say, html, then you must take care of the numbering yourself.

Counters exist to make this possible. Counters can be incremented, can be given a particular value, can be given a new value temporarily and can be removed. They always contain integral values, which may be negative.

Section 2.5 describes the use of counters in more detail.

2.2.1.7: Macros

Macros are comparable to builtin functions, but they can be defined in Yodl input files. Macros add functionality to Yodl exceeding the basic functionality of the builtin functions. Macros can have arguments, and they are used in exactly the same way as builtin functions are used.

When Yodl encounters a macro, it acts as follows:

Defining macros is described in section 3.1.11. Macros may be defined, deleted, renamed, and temporarily given other definitions.

2.2.1.8: Nousermacros

When Yodl is started using the -w flag on the command line, then warnings are generated when Yodl encounters a possible macro name, followed by a parenthesized argument, without finding a macro by that name. Yodl then prints something like cannot expand possible user macro.

Examples of such sequences are, The necessary file(s) are here, or see the manual page for sed(1). The candidate macros are file and sed, as these names could very well have been `valid' user macros followed by their arguments.

A nousermacro can be defined to suppress these warnings, by informing Yodl that file and sed aren't macros. Nousermacros may be defined and undefined. See sections 3.1.41 and 3.1.16 for details).

2.2.1.9: Symbols

Yodl symbols contain text. They were introduced to allow the flexible expansion of text, the length and/or content of which cannot be determined in advance. In particular, symbols are useful to store a series of LaTeX document options, or a series of html body options. In earlier versions of Yodl complex and confusing constructions using nested definitions of macros were used for this. These macros were not only confusingly complex, but they also suffered from a hard-coded maximum. Symbols solve these drawbacks, and now that they are available, they are used for all natural situations in which an initially unknown piece of text must be stored. National language specific strings are another useful area in which symbols can be used. The symbol CONTENTSHEADING can be set to the name of the contents heading (e.g., Contents in English, Inhoud in Dutch, Contenido in Spanish, and macros can simply insert the value of the symbol CONTENTSHEADING at the appropriate location.

Symbols can be defined 3.1.12, removed 3.1.17, (temporarily 3.1.57 or permanently 3.1.61) be given another value; pushed symbol values can be restored 3.1.51 at a later point. Of course, their values can also be inserted 3.1.63 into Yodl's output file.

2.2.2: Line continuation

To make the typing of input easier, Yodl allows you to end a line with a backslash character \ and to continue it on the next line. That way you can split long lines to fit your screen. When processing its input, Yodl treats these lines as one long line, and ignores the final \ character. This feature only works when the \ character is the last one on the line (only a physical newline may follow).

When the line following the one with the \ character has leading spaces, then these are omitted. This allows you to `indent' a file as you wish, while the space characters of the indentation are ignored by the Yodl program.

A trivial example is the following: 


    Grandpa and\ 
    grandma are sitting on the sofa.
Due to the occurrence of the \ character in the sequence and\, Yodl combines the lines to 

    Grandpa andgrandma are sitting on the sofa.
Note that the spaces before grandma are ignored, since this is the second line following a \ character.

If you do want one or more spaces while joining lines with \, put the spaces before the \ character.
Summarizing:

The question is of course, how do you accomplish that a line really ends with a \, when you do not want Yodl to merge it with the following line? In such a case, type a space character following your \: Yodl won't combine the lines. Or set the \ character as CHAR(\) or CHAR(92) (see section 3.1.4 for the CHAR macro).

When Yodl processes input files, and the white-space level exceeds zero (see section 3.1.35), then all lines are processed as if they terminated by a \. This behavior was implemented first with Yodl version 2.00. It can be suppressed using Yodl's -k flag.

2.2.3: The +identifier sequence

There may be situations in which you must type a macro name right after a sequence of characters, while Yodl should recognize this. Imagine that someone wrote a great macro footnote for you (someone did, in fact, see the next chapter), to typeset footnotes. If you'd type in a document: 

The C Programming Languagefootnote(as defined by 
Kernighan and Ritchie) ...

then of course Yodl would fail to see the start of a macro in the sequence Languagefootnote. You could say 

The C Programming Language footnote(as defined by 
Kernighan and Ritchie) ...

but that would introduce a space between Language and the footnote. Probably you don't want that, since spaces between a word and a footnote number look awful and because of the fact that the footnote number might be typeset on the following line.

For these special situations, Yodl recognizes the +identifier sequence as the start of a macro, while the + sign is effectively ignored. In the above example you could therefore use 

The C Programming Language+footnote(as defined by 
Kernighan and Ritchie) ...

The +identifier recognition only works when the identifier following the + sign is a macro. In all other situations, a + is just a plus-sign.

(The +identifier sequence furthermore plays an important role in macro packages. If you're interested, see the file shared.yo which is by default installed to /usr/local/lib/yodl.)

2.2.4: Preventing macros from being expanded

One more feature of the Yodl language remains to be described. In the previous section it was described how a macro may be called immediately following alphabetical characters. What about the opposite situation where we do not want a macro to be expanded in a particular situation? The NOUSERMACRO builtin command (cf. section 3.1.41) may be used to suppress the interpretation of a character sequence (e.g., file(...)) as a macro, but what if a macro should not be expanded in the occasional situation? For this case various solutions are available:

2.3: Character tables

The Yodl language provides a way to define character translation tables, to activate them, and to deactivate them. A character translation table defines how a character in the input will appear in the output.

There are two main reasons for the need of character translation tables. First, a document language becomes much easier to use when you can type an asterisk as * instead of $*$ or \verb/*/ (these are sequences from the LaTeX document language). Hence, a mechanism that expands a * in the input to to \verb/*/ on the output, saves the users a lot of typing.

Second, forcing users to type weird sequences won't work if you're planning on converting the same Yodl document to a different output format. If the user types \verb/*/ in the input to typeset an asterisk in the output, how should he or she arrive at a single * in the output in another output format?

The solution is of course to define the translation for an input character like * given the output format.

2.3.1: Defining character tables

The built-in macro DEFINECHARTABLE defines a character translation table. It takes two arguments: the name of the table and the character translations. Hence, each table is defined by its own name.

As an example of a table, consider the following fragment. It defines a table that translates the upper case characters A to E to their lower case equivalents: 

DEFINECHARTABLE(tolower)(
    'A' = "a"
    'B' = "b"
    'C' = "c"
    'D' = "d"
    'E' = "e"
)

Each DEFINECHARTABLE statement must have a non-empty second parameter. "Empty" character tables cannot be defined, though one non-translation table is built-in.

The syntaxis of the second argument is as follows:

Translations which are not specified in the table are left to the default, which is to output the character as-is.

Note that the character table translation is something that the yodl program does as one of its last actions, just before sending text to the output file. The expansion text is not further processed by yodl, except for the conversion of C-type escape sequences to ordinary characters. The expansion text should therefore not be protected by, e.g., NOTRANS (unless of course you want some character to generate the text NOTRANS on the output).

2.3.2: Using character tables

A defined translation table is activated by the macro USECHARTABLE. This macro takes one argument, which may be:

The default mapping, selected when an empty argument is given, means that Yodl enters its `zero translation state', meaning no character translation at all.

2.3.3: Pushing and popping character tables

Besides the previously described macro USECHARTABLE(), Yodl has one other mechanism of activating and deactivating character translation tables. This mechanism uses a stack, and hence, the related macros are appropriately named PUSHCHARTABLE() and POPCHARTABLE().

Using the push/pop mechanism is handy when a table must be temporarily activated, but when it is not known which table exacty is active prior to the temporary activation. E.g., imagine that you need to use a character table called listing to typeset a listing, but that you do not know the current table. The pushing and popping mechanism is then used as follows: 

COMMENT(First, we save the current table on the stack and
        we activate our "listing" table.)
PUSHCHARTABLE(listing)

COMMENT(Now the text is question is typeset.)
...

COMMENT(The previously active table is re-activated, whatever its name.)
POPCHARTABLE()

2.4: Sending literal text to the output

The Yodl program has several built-in macros that can be used to send literal text to the output file. The macros are listed in the chapter 3.1 and are briefly described here: To illustrate the need for the distinction between NOTRANS and NOEXPAND, consider the following. The HTML converter (described in chapter 4) must be able to send HTML commands to the output file, but must also be able to send literal text (e.g., a source file listing). The HTML commands of course must be neither translated with any character table, nor must they be expanded in regard to macros. In contrast, a source file listing must be subject to character translations: the &, < and > characters can cause difficulties. Two possible macros for a HTML converter are: 

    COMMENT(--- htmlcommand(cmd) sends its argument as a HTML command 
                to the output ---)
    DEFINEMACRO(htmlcommand)(1)(NOTRANS(ARG1))
        
    COMMENT(--- verb(listing) sends the listing to the output ---)
    DEFINECHARTABLE(list)(
        '&'     =   "&amp;"
        '<'     =   "&lt;"
        '>'     =   "&gt;"
    )
    
    DEFINEMACRO(verb)(1)(
        USECHARTABLE(list)
        NOTRANS(<listing>)
        NOEXPAND(ARG1)
        NOTRANS(</listing>)
        USECHARTABLE(standard)
    )
In this example it is assumed that a character translation table standard exists, defining the `normal' translations. This table is re-activated in the verb macro.

The currently defined verb macro for, e.g., html conversion looks like this: 


    DEFINEMACRO(verb)(1)(
        PUSHSUBST(0)
        NOTRANS(<pre>)
        XXnl()
        NOEXPAND(ARG1)
        XXnl()
        NOTRANS(</pre>)
        XXnl()
        POPSUBST()
    )
Here, the surrounding PUSHSUBST(0) and POPSUBST() pair prevent the interpretation of multi-char SUBST definitions like 

    SUBST(\"e)(+NOTRANS(&euml;))
inside verb environments, causing unexpected translations in pieces of code like 

    cout << "\"evil\" code\n";
which, if the PUSHSUBST command isn't used, is converted to 

    cout << "&eacute;vil\" code\n";

2.5: Counters

Some document languages (notably LaTeX) automatically prefix numbers when typesetting sections, subsections, tables, figures etc.. Other document languages (e.g. HTML) unfortunately don't.

Therefore, a macro package that converts a Yodl document to LaTeX doesn't need to provide the numbering of sections etc.. However, if you do want the numbering and if you want to convert documents to, say, HTML, then you must take care of the numbering yourself.

This section describes the counters in Yodl: how to create counters, how to use them, etc..

2.5.1: Creating counters

Before a counter can be used, it must be created using DEFINECOUNTER or PUSHCOUNTER. These functions expects two arguments: the name of the counter and an (optional) integral additive expression. When provided, the expression may contain binary + and unary or binary - operators; its operands may be integral values or the names of exising counters. Additive expressions may not contain white space characters. If an additive expression is provided it is evaluated and its value is then used to initialize the named counter. If the second argument is empty then the counter is initialized to zero (0).

For example, let's say that our macro package should provide two sectioning commands: section and subsection. The sections should be numbered 0, 1, 2, etc., and the subsections 1.1, 1.2, 1.3 etc.. For this we can define two counters: 


    DEFINECOUNTER(sectcounter)()\//       initialized to 0
    DEFINECOUNTER(subsectcounter)(1)\//   initialized to 1

2.5.2: Using counters

The builtin function COUNTERVALUE(somecounter) expands to the value of somecounter. E.g., if the current value is 2, then the value 2 is inserted into the output object. It is an error to use COUNTERVALUE on a non-existing counter or on a counter not having a defined value (see below).

Yodl has several functions to modify and/or to set the values of counters. The counter's value, named expression below, is an additive expression: it may contain binary + and unary or binary - operators. The operands may be integral values of the names of exising counters. Additive expressions may not contain blank spaces.

The functions modifying values of counters are:

Given the numbering requirements of the hypothetical commands section and subsection (see the previous section), we can now complete the definitions: 

DEFINECOUNTER(sectcounter)
DEFINECOUNTER(subsectcounter)

DEFINEMACRO(section)(1)(\ 
SETCOUNTER(subsectcounter)(0)\ 
USECOUNTER(sectcounter) ARG1)

DEFINEMACRO(subsection)(1)(\ 
COUNTERVALUE(sectcounter).USECOUNTER(subsectcounter) ARG1)