TCW02: Approaching the Son of ODD: source markup for P5

This paper proposes a change of direction for the markup of the source of the TEI Guidelines; it is put forward as part of the TEI Council’s internal workgroup on the P5 revision of the TEI. We propose that:

• The ODD format should be revised to be entirely independent of the SGML/XML notation for DTDs
• The ODD format and the current TEI DTD for tag documentation (TSD) should be combined into a single standard TEI tagset useable as a pizza topping.
• The new notation should be based on one of the XML schema languages.
• Additional processors should be created to make not only XML DTDs (and possibly SGML as well), but also at least one XML schema format.
• Where possible, data types should be converted to use the datatype library of the W3C.
• The Pizza Chef should be rewritten to allow user choice of DTD or schema output.

[Most of this section is from a paper scheduled for delivery at the XML Europe 2002 conference by Sebastian Rahtz].

The original TEI project concentrated on abstract models of markup, which the editors used SGML to instantiate, that being the best method available to them at the time. However, they did their best to insulate themselves from future changes by formalizing the markup constraints in a small SGML-based markup language, rather than directly writing DTD code. In this respect, they followed the literate programming WEB model developed and popularized by Donald Knuth in which a single document is produced which contains both formal code and its documentation. This single source can then be processed to produce several outputs:

• SGML or XML documentation for the SGML DTD modules, containing embedded DTD fragments (in the same way that Pascal code is intermingled with prose in the Web system); this corresponds to the output from Knuth’s Weave processor.
• SGML or XML documentation for the elements, attributes, and other items in the markup language constituting an alphabetic reference manual.
• the actual DTD modules defining the markup language; this corresponds to Knuth’s Tangle processor.

The TEI literate programming system (jocularly named ODD, for One Document Does it all) as originally specified is documented in an internal TEI working paper (). This design underwent several modifications during the process of implementing and using it for production of the TEI Guidelines. C.M. Sperberg-McQueen proposed a more general system called

(see ), but this was not completed or implemented in the TEI. Other XML/SGML-based literate programming schemes are neatly summarized at .

The ODD system consists of a small suite of additional declarations for special purpose elements which are embedded within a special subset of the same basic TEI DTD designed for authoring of new documents.

Amongst the basic building blocks of the ODD extension to TEI markup are the tagDoc, a special-purpose element for documentation of a single TEI element, and the classDoc, an element which documents a single TEI class. Elements in the TEI scheme are assigned to one or more classes of three distinct kinds: model classes, which group elements that have similar structural properties, i.e. they can appear at the same point in the document hierarchy; attribute classes, i.e. they share common attributes; and semantic classes, i.e. they have similar semantic properties. As a simple example, we present the tagdoc for the bibl element:

<tagDoc id=”BIBL” usage=”opt”> <gi>bibl</gi> <name>bibliographic citation</name> <desc>contains a loosely-structured bibliographic citation of which the sub-components may or may not be explicitly tagged. </desc> <attList/> <exemplum> <p class=”eg”> <bibl>Blain, Clements and Grundy: Feminist Companion to Literature in English (Yale, 1990)</bibl> </p> </exemplum> <exemplum> <p class=”eg”> <bibl> <title level=”a”>The Interesting story of the Children in the Wood</title>. In <author>Victor E Neuberg</author>, <title>The Penny Histories</title>. <publisher>OUP</publisher> <date value=”1968″>1968</date>. </bibl> </p> </exemplum> <remarks/> <part type=”base” name=”core”/> <classes names=”CLBIBL DECLABL”/> <dataDesc>Contains phrase-level elements, together with any combination of elements from the <term>biblPart</term> class</dataDesc> <elemDecl> %om.RO; (#PCDATA | %m.phrase; | %m.biblPart; | %m.Incl;)*</elemDecl> <ptr target=”COBITY”/><ptr target=”HD3″/><ptr target=”TETA”/> <ptr target=”CCAS2″/> </tagDoc>

The start of this is straightforward; a description of the element, a list of attributes (here empty), and some examples. The next part is more interesting, as it places bibl into the overall scheme of the TEI. Firstly, the part element says to which module of the TEI bibl belongs (it forms part of the core module), and secondly the classes element indicates of which classesbibl is a member. In this case, bibl is a member of the semantic class bibl and the attribute class declarable. The last part of the tagDoc supplies the intended content model in a heavily parameterized form, and some pointers to further relevant parts of the Guidelines.

When run through the DTD generator, the following declaration is produced from this tagdoc element:

<!ELEMENT %n.bibl; %om.RO; (#PCDATA | %m.phrase; | %m.biblPart; | %m.Incl;)*> <!ATTLIST %n.bibl; %a.global; %a.declarable; TEIform CDATA ‘bibl’ >

Note here how the name of the element itself has been parameterized (%n.bibl;), thus permitting its renaming to another language. The attribute list has been generated from a variety of sources: (TEIform) has been generated, the value of which preserves the canonical name. The attribute , using information from the classes element of the tagDoc. The values CLBIBL and DECLABL serve as pointers to the appropriate classDoc elements, on which these same values are used as identifiers: CLBIBL, for example, links to the following classDoc element:

<classDoc id=”CLBIBL” type=”model”> <class>bibl</class> <desc>bibliographic elements. </desc> <attList/> <remarks/> <part type=”base” name=”core”/> <classes names=”INTER COMMON”/> <ptr target=”COBI”/> </classDoc>

Although this defines no attributes itself, it does declare itself to be a member of the class COMMON, which is in turn a member of the class GLOBAL, within which the global attributes are actually declared; similarly, the class DECLABL supplies another group of attributes. The same class mechanism also determines which DTD parameter entities the current element will be in: the classDoc for CLBIBL creates an entity %m.bibl; which includes bibl (or, more precisely, %n.bibl;).

An example of an element which has `local’ attributes is formula:

<tagDoc id=”FORMULA” usage=”rwa”> <gi>formula</gi> <desc>contains a mathematical or other formula.</desc> <attList> <attDef usage=”opt”><attident>notation</attident> <desc>supplies the name of a previously defined notation used for the content of the element.</desc> <datatype>%formulaNotations;</datatype> <valDesc>The name of a formal notation previously declared in the document type declaration.</valDesc> <default>#REQUIRED</default> <remarks/> </attDef> </attList> <part type=”top” name=”ft”/> <classes names=”PHRASE”/> <dataDesc>The content model for this element is specified by the parameter entity <term>formulaContent</term>, the default value of which is <mentioned>CDATA</mentioned>.</dataDesc> <elemDecl> %om.RR; %formulaContent;</elemDecl> <ptr target=”FTFOR” type=”div1″/> </tagDoc>

The result of this in the DTD is:

<!ELEMENT %n.formula; %om.RR; %formulaContent;> <!ATTLIST %n.formula; %a.global; notation %formulaNotations; #REQUIRED TEIform CDATA ‘formula’ >

in which the attDef element is translated into DTD notation.

The main body of the Guidelines is a conventional TEI document with chapters and sections which describe different parts of the TEI in great detail. Mixed in with informal prose are references to the tagDoc and classDoc elements, by means of dtdFrag, tagDecl, and claDecl elements; so the chapter describing formulae ends with:

<p>The formula element itself is defined as follows: <dtdFrag id=”DFTFOR” n=”Formulae”> <tagDecl tagDoc=”FORMULA”/> </dtdFrag> </p>

This inserts a summary description of the element at this point in the text; the DTD files themselves are created by a variant on the above, e.g.

<dtdFrag id=”DVE” n=”Base Tag Set for Verse” file=”teivers2.dtd”> <commDecl>First, declare the default text structure, which is the same for verse as it is for other kinds of text. (But declare it only if verse is the only base.)</commDecl> <msection keywords=”%TEI.singleBase;”> <entDecl entDoc=”TEISTR”/> <peRef n=”TEI.structure.dtd”/> </msection> <commDecl>Finally, declare the elements specific to this base tag set</commDecl> <dtdRef dtdFrag=”DVEST”/> <dtdRef dtdFrag=”DVESE”/> </dtdFrag>

This determines in what order elements are placed in separate files of the DTD suite, and in what marked sections (using msection) etc. There are many small extra features of the system, but this covers the essential operations.

It should be clear from this brief overview of TEI literate programming that the current model has three main characteristics:

• A class system for classifying elements and attributes which is essentially independent of SGML or XML;
• Markup for describing attributes and DTD files which is expressed in XML, but is quite closely tied to DTD notation (e.g. the datatype of %formulaNotations; in formula)
• A bottom-level element, elemDecl, which contains literal DTD code

The original aim of the TEI editors in trying to make the scheme independent of the current SGML/XML instantiation is not, in our view, completely successful; nevertheless, it provides a plausible framework for the generation of versions of the TEI Guidelines expressed using other metalanguages.

The main difficulty with the current ODD scheme is the elemDecl element, dataType and default elements, all of which contain pure DTD notation. The string element is also used to build up element content models. Section lists the range of values used throughout P4 for all these elements. The problem is exacerbated by the fact that the elemDecl contents do not correspond to the published DTDs, since there we imply that all element names are indirect, using the %n..; notation. Thus the content model

should actually be

if the DTD were to easily derived. In practice, the %n… is added by an ad hoc Perl script when the DTDs have been generated, which is embarassing and error-prone.

If the TEI Guidelines were maintained in complete DTD format, we would be in a better position, as we could use DTD-processing software to analyze the markup. As it stands, we have no effective into the element markup models, and no real tools by which we can check them for correctness in markup or references; if, for instance, someone changes %m.Incl; to %m.incl; by mistake, it will not get caught during early validation of the master document, only when a DTD is generated and some sample documents processed.

The default and dataType lists in , used for constructing attribute lists, show a combination of DTD markup (NMTOKEN) and literal values (unspecified), as well as a few occurences of abstract data types (ISO-date).

Finally, we may note that the fact that the eventual TEI DTDs are constructed from different types of data makes it necessary for the TEI Pizza Chef to use an entirely new program (

) to e.g. remove unused elements from subset DTDs. Ideally, we would like to do this work from the TEI source itself, since this would allow us to construct ad hoc subsets of the TEI Guidelines documentation to go with the DTD subset.

We propose that the content of all elemDecl element, dataType, default and string should replaced by full XML markup. This would have several immediate advantages:

• We could perform deeper validation of the TEI Guidelines
• We could representations of the constraint model in forms other than DTD much more easily
• We could write a new Pizza Chef which extracted documention and constraint subsets at the same time
• We could experiment with using other XML vocabularies alongside the TEI, by using XML namespaces; obvious examples are MathML and SVG. Selective validation should be possible.
• We would have an opportunity to add more extensive datatyping and subtler constraints than are currently possible, possibly allowing us to re-express some of the descriptive text as formal constraints.

The downsides are:

• The replacement language will be unfamiliar, and is likely to be much more verbose (DTD is very succinct), which imposes a burden on the TEI editors, workgroup authors, and readers
• The process of getting back to where we are now (DTDs) becomes more complex and opaque (ie we can no longer directly compare the DTD with the Guidelines text)

We suggest that these are fairly short-term problems which should not stop us proceeding.

What markup should we use for the new work? We have three choices:

• Use an existing notation for specifying constraints. UML is the obvious choice.
• Use fragments of an existing schema language, such as W3C schema, or RelaxNG.
• Invent a new language of our own

The first choice imposes the most problems, as we have no prior experience to show that we can model the TEI using eg UML (although a preliminary investigation by Leonor Barroca, of the Open University, shows that it should be possible). In addition, although UML has an XML-based interchange format, it is not per se XML, which imposes extra complexity on us.

Our natural course is to invent a new language, but it seems an intolerable burden on the TEI to replicate the many man-years of labour recently devoted to the subject in the XML community. Therefore it seems sensible to adopt the notation, if not the full panoply, of an existing language.

As a preparation for this paper, we undertook a complete conversion of the P4 sources to a form based on RelaxNG. This demonstrates the feasibility of the idea, and points towards some of the work which will have to be done. It is discussed in the following sections.

The major job in the conversion is the extraction of elemDecl elements from the source, and converting each one to an equivalent fragment of Relax NG. This was done with a Perl script which converted, for example, a content model of

%om.RO; (#PCDATA | %m.dictionaryParts; | %m.phrase; | %m.inter; | %m.Incl;)*

to an intermediate XML form of

<schema> <group> <text/> <seqbar/> <element name=”%m.dictionaryParts;”/> <seqbar/> <element name=”%m.phrase;”/> <seqbar/> <element name=”%m.inter;”/> <seqbar/> <element name=”%m.Incl;”/> </group> <repstar/> </schema>

This can be converted, using an XSLT program, to

<rng:zeroOrMore xmlns:rng=”http://relaxng.org/ns/structure/1.0″ xmlns:ext=”http://www.tei-c.org/relaxng/1.0″> <rng:choice> <rng:text/> <rng:ref ext:class=”entity” name=”m.dictionaryParts”/> <rng:ref ext:class=”entity” name=”m.phrase”/> <rng:ref ext:class=”entity” name=”m.inter”/> <rng:ref ext:class=”entity” name=”m.Incl”/> </rng:choice> </rng:zeroOrMore>

An extra bit of data has been added here (in its own namespace), the `rend’ attribute; this is put in to allow us to recreate the DTD at a later stage. In our use of RelaxNG, <rng:ref name="foo"/> would turn simply into the name `foo’ in a DTD, whereas <rng:ref class="entity" name="foo"/> turns into `%foo;’. There are a number of small inelegancies like this in the system we have developed, dictated either by the need to allow for back-translation to DTD, or by the presence of partial content models like | foo | bar.

When the fragments of content model have been put back into the main text, we can constrast the old declaration of the bibl element:

with the new

Note the `omit’ attribute which is added to allow for recreating SGML DTDs.

The remaining task was to rewrite the existing DTD generator to reconstruct content models from the Relax NG elements, and to write a new processor which extracts all the Relax fragments and creates a set of useable schema files. This was straightforward, but ugly, as we did nothing to change the datatypes, for instance, from their SGML notations. Similarly, the Guidelines use such SGML/XML literals as #IMPLIED or #REQUIRED, which need interpreting. Ideally, these would be re-expressed in some more neutral form. However, a relatively simple XSLT script suffices to produce schemas which are acceptable to the Relax NG processors. This final stage was debugged by comparing the results with conversions of the TEI DTDs made by James Clark’s rngconv program, which gave many useful hints.

The TEI Consortium’s Board and Council now need to decide:

• Whether to change the markup language for P5 at all, but to continue with DTD language internally, and derive schemas from the DTDs.
• If we proceed, which schema or other language to use
• Whether the printed Guidelines should show DTD content models or schemas (or some other abstract notation)
• What timescale to use for the work