TMQL Introduction

TMQL Introduction

Robert Barta rho@bond.edu.au

This introduction is a tour de force through TMQL, the upcoming Topic Map query language.

Introduction

Work in TMQL has begun long ago, having been kicked off by a number of proposals. After quite some background work―actually a prototype has been implemented and a lot of formal research regarding optimization has been done), the editors have produced another official draft specification.

We assume that your are familiar with Topic Maps per se and how to create maps. The following TMQL preview builds on an existing presentation, but it breaks somewhat with the TM tradition to use 'operas' as running use case. Instead, we will use the more general theme 'music', and will assume that we have a TM data store which contains various albums, (female or male) musicians and music groups (which are all artists and have persons as members). All these topics are connected via various associations, such as is-produced-by or is-part-of.

As TMQL also can be used to generate Topic Map content, obviously a notation for this has to be chosen. As we speak, uhm write, the exact form of CTM, the compact topic map notation is being defined. We here use AsTMa= 2.0 for the time being.

Setting Off

If you have used SQL before, then you will not be completely puzzled by the following query:

select $album
where
   is-produced-by (production: $album, producer: tom-waits)

This query (technically a query expression) will return all albums where Tom Waits is known to be a producer. tom-waits is an identifier of a topic which we happen to know to uniquely pinpoint that topic about that person in the map we query. The query processor will try to find an association of type is-produced-by and will check whether the topic tom-waits is playing the role producer there. If so, it will bind the variable $album to the topic playing the role production in that very same association. It will so work through the whole map and will collect all these variable bindings. Because we asked so in the select clause, the query processor will return a list of the bindings for that variable. What exactly we will get returned into the application, we will discuss a bit later.

If we wanted to make the query more watertight to return albums only (and not something else produced), then we will have to add another constraint to the WHERE clause:

select $album
where
   is-produced-by (production: $album, producer: tom-waits) &
   $album isa album

If we are not fixated on Tom Waits and would instead want a list of all albums together with their producers, we can extend the wishlist in the SELECT clause:

select $album, $producer
where
   is-produced-by (production: $album, producer: $producer) &
   $album isa album

Controlling What is Returned

In the SELECT clauses we have used so far, we asked for whole topics. Any query processor will hand over a complete topic data structure (most probably according to TMDM) into the application. If an application were interested, say, only in the name of such a topic, it would have to use some API outside the scope of TMQL to navigate to that name.

To let the TMQL processor do the work, we can tweak the SELECT clause by adding a path expression:

select $album / name
where
   $album isa album

This process of converting a characteristics (be it a name or an occurrence) into a string is called atomification, as literals as numbers, strings, etc. are referred to as atoms. In most cases, this is highly convenient, in others this is not.

Generally, a topic can have any number of names―all with different type or scope―, so we actually would get a whole list of names for each individual album. If we were not interested in all names, but only in those in, say, english (en), then we want to filter the characteristic first based on the scope and only then atomify the name:

select $album / name [ @ en ]
where
   $album isa album

Path expressions are also a convenient way to impose a sorting order on the list of tuples we return:

select $album, $producer
where
   is-produced-by (production: $album, producer: $producer)
order by
   $album / name [ @ en ]

The ordering can also include more than one ordering criterion, like in

select $album, $producer
where
   is-produced-by (production: $album, producer: $producer)
order by
   $producer / name [ @ en ] desc, $album

As you would expect, TMQL makes it possible to make the list of returned tuples unique and to select only slices out of the whole result set. It could look like this:

select $album
order by
   $album / name
unique offset 10 limit 20

Identifying Things

You may correctly argue that identifying topics with their (internal) map identifier (the TMDM model calls this source locators) is not an immensely robust idea if that identifier may change any second. Less brittle solutions involve to use either subject locators to directly address a subject, or subject indicators for indirect identification.

In the case, for instance, that there is a subject indicator, you can use that instead of an internal identifier:

select $album
where
   is-produced-by (production: $album, producer: http://www.u2.com/ ~)

select $album
where
   is-produced-by (production: $album, producer: $producer) &
   is-maintained  (maintainer: $producer, website: http://www.u2.com/ =)

Controlling Variable Bindings

As we have seen above, variables can be bound to values. Used naively, this can lead to incorrect queries. As an example let us find all two albums which share the same producer. In a first attempt we write:

select $album1, $album2
where
   is-produced-by ($album1: production, $producer: producer) &
   is-produced-by ($album2: production, $producer: producer)

This does not work for us if we want different albums. The usual escape hatch is to have something like this:

select $album1, $album2
where
   is-produced-by ($album1: production, $producer: producer) &
   is-produced-by ($album2: production, $producer: producer) &
   $album1 != $album2

TMQL has a rather eccentric way to fine-control when variables are allowed to match anything or when they must be bound to something different:

select $album, $album'
where
   is-produced-by ($album : production, $producer: producer) &
   is-produced-by ($album': production, $producer: producer)

There is no limit to the number of primes, so should we―by a bizarre twist of fate or customer requirements― need three different albums, this can be achieved nicely:

select $album, $album', $album''
where
   is-produced-by ($album  : production, $producer: producer) &
   is-produced-by ($album' : production, $producer: producer) &
   is-produced-by ($album'': production, $producer: producer)

Association Templates

In the queries so far we made use of association templates. Writing inside a query

   is-produced-by (production : $album, producer: $producer)

To allow for such associations with additional roles to match, TMQL allows to append an ellipsis:

select $album
where
   is-produced-by (production : $album, producer: $whoever, ...)

Association templates actually have more implicit meaning than is obvious at first sight. If, for example, the map contained an association of type is-remastered-by which also connects an album with a producer and is-remastered-by is a subtype of is-produced-by, then also such associations would match the template.

Honoring subclassing also applies to roles. Had we in our queried map an association of type is-remastered-by, but the role (type) for the album is not production, but the subclass remastering, such association would also match the association template.

If you would not care about the role, you can use a wildcard there:

select $album
where
   is-produced-by (production : $album, * : $whoever, ...)

Query Context

You may have wondered how the query processor knows which map to query. One way is that the map itself is hard-coded into the query expression itself:

select $album
from file:mymap.xtm ~ >>
where
   ...

While hard-coding such information into a query may make sense in some cases (query processors may want to analyse the map before query time in order to optimize), generally one would like to pass this map information at query time to the TMQL processor. This is done via a mechanism we already have seen: variable bindings where a value (string, integer, but also whole maps) are bound to variables. Applications will build these binding sets and pass them somehow to the processor.

For our example, our application could have already loaded the map into memory, possibly having done some modifications already to it. When that map is bound to, say, %m, then inside the query we can refer to this variable, blindly trusting that it contains a map:

select $album
from %m
where
   ...

The query context can contain other variables as well. Slightly paraphrasing an earlier query, we could parameterize it with a URL which we additionally pass in bound to $url:

select $album
where
   is-produced-by (production: $album, producer: $url ~)

Path Expressions

The textual overhead of the SQLish style which we have used so far may not be convenient if queries are trivial. Especially for web applications where pages have to be filled with lots of content from a TM backend a much shorter notation is more adequate.

To return all albums from the map bound to %_ we can simply write

// album

// album / name [ @ en ]

We can also, for instance, reformulate an earlier query which returned only the english names of Tom Waits albums:

// album [ . <- production [ * is-produced-by ] -> producer = tom-waits ] / name [ @ en ]

Only these albums where the condition is satisfied will be postprocessed in that the english name is selected from them. Implicitly, the string values are taken.

Chocolate, Vanilla, Caramel

The different language flavours, select and path expressions, can―as we have already seen― be mixed. Not so obvious is the fact that both styles are (almost) equivalent in terms of expressitivity; every select query expression can in fact be transformed into an equivalent path expression.

As path expressions cannot introduce new variables, they can become quite contrived when they get more complex.It is up to the developer to choose the most appropriate combination on a case by case basis.

Both these styles allow to return sequences of tuples of things into the application. This may be exactly along your line of thinking in most cases, but it does not help tremendously if you want to comfortably embed the query results into one of these shiny XML applications servers. To avoid that developers have to write their own template engines, TMQL allows to create XML content using a third flavour, which―you may have guessed it―is otherwise equivalent to the other styles. This flavour, FLWR, is inspired by XQuery and uses RETURN clauses to specify the output:

return
   <albums>{
     for $a in // album return
         <album>{$a / name [@ en ]}</album>
   }
   </albums>

Such body is defined by a nested RETURN clause. It contains an element <album> which wraps only text content which we specify with an embedded TMQL path expression $a / name [@ en ]. Like in XQuery, XML content and query text is separated using {} brackets. Since the processor knows that here text has to be, it will implicitly take the string value of that basename.

Using Exist and All Quantification

On some occasions you will have to test whether particular things exists in a map or whether all things in certain set have a particular property. For illustration, let us ask for all music groups in our map which have at least one female group member

for $group in // group
where
    some $person in $group <- whole -> member
         satisfy
             $person isa female
return
    ($group)

Conversely, we might be interested to find all boy groups, well, at least those groups where all members are male:

for $group in // group
where
    every $person in $group <- whole -> member
         satisfies
             $person isa male
return
    ($group)

Predefined Datatypes and Functions

Since the latests TMDM drafts include features to store arbitrary data (and not just text as in the original XTM-based standard drafts), TMQL has provisions for data. The obvious first thing to support is to denote constants; here TMQL borrows (stealing is such an ugly word) from RDF(S) notations:

select $person / name
where
   $person / age > "18"^^xsd:integer

Now, writing integers, floats, or even dates and strings in this explicit form is somewhat clumsy. Therefore TMQL has adopted a few of these primitive types, specifically integers, decimals, dates, URIs and strings, all imported from the XSD (XML schema data types) namespace. Constants of these types can be written without the explicit typing fluff:

select $person / name
where
       $person / age  >  1           &
       $person / born >= 1962-06-06  &
   not ( $person / name     = "Bill Gates" |
         $person / homepage = http://he.is.so.good.to.us/ )

       

As a consequence, also necessary functions for these data types have been imported into TMQL and are part of the predefined environment. This includes also the comparison functions which we have used above. Actually, when writing a comparison like $person / age > 18, several things happen: first we extract the age occurrence from a person. This characteristic will then silently atomified to its value. If that value is already an integer, then it will be used as-is. Otherwise TMQL will try to convert it, which may only work if the value is a string with only digits in it. Then the two integers will be compared using the predefined function op:numeric-greater-than.

There is also another twist here: You may have noted that a path expression like $person / born may return more than one characteristic; or none, depending how many born occurrences actually exist for the given topic. The interpretation TMQL chooses here is exists semantics. TMQL will translate this to a more explicit

some $_16523 in $person / born 
    satisfy 
        op:dayTimeDuration-greater-than ($_16523, 1962-06-06)

Prefixes

Using whole URIs to identify topics (actually the subjects they stand for) certainly clutters queries. Using prefixes for namespaces is a comfortable way to shorten queries considerably, so TMQL also uses this mechanism; albeit, not in a syntactic way, like in XML, or RDF, but more appropriately here, in a semantic sense.

While the xtm prefix is actually one of the predefined ones, it could be redefined like this:

"""
xtm isa tmql:ontology ~ http://www.topicmaps.org/xtm/1.0/#
"""
select $thing
   $thing isa xtm:topic

For the TMQL processor this means two things: First, it will try to understand the contents of that ontology (or vocabulary). This can be done by either downloading the file, parsing it and remembering the concepts defined therein. It can also mean that the processor already knows about that URI and therefore also vocabulary it stands for. Here it does not really make a difference whether the vocabulary is defined as topic map or via, say, an OWL ontology. The second thing the processor does, is that the topic source identifier (xtm) can be used as prefix within the query which directly follows the map. When the processor encounters xtm:topic, it will find the prefix xtm and will expand the QName to http://www.topicmaps.org/xtm/1.0/#topic; that URI is then interpreted as subject identifier in the queried map. That, of course, only makes sense to be used if such a subject is covered there.

If, in contrast, we would import an ontology which has no relation to the map to be queried, then such a prefix is useless at first. But let us look at the following query fragment:

"""
opera isa tmql:ontology ~ http://www.ope-r-us.org/opera.ltm
"""
select $opera
from %_ + opera >>
   ...

Inside the query body, the opera prefix can the be naturally used:

....
   $opera isa opera:opera &
   written-by (opus: $opera, author: tom-waits)

Finally, what if an ontology exists, but is not in Topic Map format but, say, in OWL? TMQL processors are free to covert from other notations and paradigms into a topic map:

"""
wine isa tmql:ontology ~ http://www.w3.org/2001/sw/WebOnt/guide-src/wine.rdf
"""
select $winery
from %_ + wine >>
where
    $winery is-a wine:Winery

User-defined Functions

Once queries get more complex, they have to be organized and engineered. One vehicle in this process is to compose functions. These can compute values from provided parameters and―semantically speaking―define a functional relationship between values and other things.

We can declare a function in the meta map; there it is just a topic of a certain type:

"""
nr-albums isa tmql:function
description: this function computes the number of albums a certain person has produced
return: fn:length (// albums [ . <- opus -> author = $person ])

"""

Inside the query we can make use of the function:

select $p / name, nr-albums (person: $p)
where
   $p isa person

Any number of parameters can be introduced this way. What can also be done is to provide default values for function parameters, if the caller does not provide one. The following function top-albums finds the most popular albums for a given producer:

"""
top-albums isa tmql:function
description: 
return: """
   select $album
   where
       $album isa album &
       is-produced-by (producer: $person, opus: $album)
   order by $album / popularity desc
   limit $limit || 5
"""

"""

If the function is invoked with all parameters, these then will be used:

select $person / name, top-albums (person: $person, limit: 3) / name
where
   $person isa person
      

The fact that functions are topics, allows a further variation of the theme. A processor may allow a developer to use a language other than TMQL, say, Python:

"""
ctime isa tmql:function
return @ python: """
    from datetime import ctime
    return ctime()
"""

"""

"""
math isa tmql:ontology ~ file:/usr/local/math/api.atm
"""

...
pow  isa tmql:function
...
sqrt isa tmql:function
...

What if the functionality should or cannot be encoded in source code as part of a topic map? Also this case is covered as ontological import:

"""
math isa tmql:ontology ~ http://maths.is.great/
"""

import math

proc = TMQL.processor()
proc.registerNS ('http://maths.is.great/', math)

And there is a last benefit for registering functions in the meta map: introspection. With it, it is possible to actually query the meta map itself to learn about the functions:

select $f / description
from %__
where
    $f isa tmql:function

Association Predicates

Predicates are tests, so they either return true or false depending on whether the claim they represent is verifyable or not. In this sense, predicates are specialized functions. And since functions can be declared in the meta map, so can predicates.

As example let us consider that our queried map contains associations of type has-created where we have recorded which artist (single or group) has created which album, such as "U2 as creator has created an opus 'Rattle and Hum'", or "'Leonard Cohen as creator has created an opus 'Various Positions'".

It lies in the nature of associations of type has-created that if a group has created something, then we might also deduce that every member of the group also has created that thing. Naturally, we do not want to change our map and add all this redundant information to it. Instead we would like add a rule to the purpose of "if an individual belongs to a group and the group has created something, so has the individual".

"""
has-created-indirect isa tmql:predicate
where: """
     has-created (creator: $creator, opus: $opus)
     |
     is-part-of  (member : $creator, whole: $group) &
     has-created (creator: $group,   opus : $opus)
"""

select $person
where
     $person isa person &
     $thing  isa *      &
     has-created-indirect (creator: $person, opus: $thing)

The obvious benefit of such rules is to keep redundancy in maps itself low, while not burdening the query itself; it would have been completely equivalent to hard-code it there:

select $person
where
     $person isa person &
     $thing  isa *      &
     (
     	has-created (creator: $person, opus: $thing)
     	|
     	is-part-of  (member : $person, whole: $group) &
     	has-created (creator: $group,  opus : $thing)
     )

Nested Queries

Nesting of query expressions is quite natural and could potentially be used in quite a few places. To demonstrate how much the language can be stretched, here an slightly obstruse example to use nesting within a SELECT clause.

To lists all groups and their members in the map, the SELECT clause may contain a whole subquery:

select $group / name,
       " members are: " + fn:string-join (
                               fn:tuple ({
                                   select $person / name
                                   where
                                       is-part-of  (member: $person, whole: $group)
                                   }), ",")
where $group isa group

For a given group we will there find all members and extract their name(s). In any case, the subquery will return a table with a single column, that with the names. To convert this into a list of things, we use the function tmql:tuple; only then we can concatenate the individual strings with commas separated using fn:string-join.

Generating Topic Maps

We have already seen that TMQL can generate lists (of tuples) and XML content. But with TMQL we can also generate topic maps, using information from the queried map, provided we use the FLWR style.

In the following example we iterate over all albums in our map and create a new map, one which only contains the albums and one of their names. All other album information (characteristics or association involvements) is ignored. Instead we add one homepage characteristic and one―admittedly absurd―new assocation where we claim that Jessica Simpson has created that album:

for $a in // album
return """

    {id ($a) + "-new"}
    {$a/name}
    homepage: http://music-r-us.com/{id ($a)}

    (has-created)
    creator: jessica-simpson
    opus   : {$a}

"""
      

The first block follows the syntax to create a topic, so the very first thing to appear here is the internal identifier of the topic. That is computed by looking up the internal identifier of the current album and by appending it with -new.

The next line again contains a TMQL expression, this time one which computes all name characteristics. Since at this position in the notation characteristics can be declared, all these names become part of the newly generated topic. The last line of the topic declaration will take care that an occurrence of type homepage will be added to the topic. The value is a URI which is static except from the final part which is provided by the albums internal identifier.

The second block creates one association per album, claiming the unbelievable.

As this template is expanded for every album, the individual expansions are all merged into one, possibly big, topic map. As we did not say much about Jessica Simpson we might want to do so, and combine this information with our result map:

return """

   jessica-simpson isa person
   bn: Jessica Simpson
   quote: "I am an artist."

{
 # here our original query goes
}
"""