XML to CUE Encoding Proposal

[matthew-sladescu]

XML to CUE Encoding Proposal

Problem

Many users would benefit from using CUE with their XML files, however CUE does not currently have an encoding that supports XML.

Purpose of this document

This document puts forward a proposal for an XML to CUE mapping called koala that can be used to add an XML encoding to CUE.

Given XML has constructs like attributes and namespaces that don't have identical analogues in CUE, there are many approaches for mapping from XML to CUE, with other future XML encodings possible. Examples of future options could include encodings that use a schema-guided approach (similar to the textproto encoding), and raw low-level AST style encodings that model each XML construct as a node that associates to abstractions like attributes, tags, and other content.

Objectives

The mapping in this proposal aims to:

• allow users to write CUE constraints against XML files (XML Support)
• provide a mapping that makes it easy to understand what a CUE constraint refers to relative to the XML it is describing (Readability)
• allow the conversion of an XML file into CUE, and to also go back from CUE to a semantically equivalent XML file.

Non-Objective

• This mapping does not aim to represent a lossless mapping. It is known for instance that unless otherwise specified in the rules below, order and XML comments are not preserved.

Proposed Mapping

The proposed mapping follows a convention that is inspired by the Badgerfish convention, with deviations for compatibility with CUE and increased readability.

This new mapping will be called koala and follows the rules below:

1. Each XML element maps to a CUE struct, with the struct key being the element name.
2. Each nested XML element becomes a nested CUE struct.
3. Each XML attribute maps to a CUE struct property, where:
   • the property key is the attribute name prefixed with $, and
   • the property belongs to the struct that is mapped from the XML attribute's parent element.
4. The text content of an XML element maps to a CUE property keyed as $$, where that property belongs to the struct that is mapped from the XML content's parent element.
5. Multiple XML elements at the same level with the same tag text map to multiple CUE structs forming part of a CUE list at that level. When this is the case:
   • the tag text is used as the property key of the list
   • the order of structs in the list corresponds to the order of elements at the same level
6. Each XML attribute that defines a namespace maps to a CUE struct property in the same way that other XML attributes are mapped.
7. When an XML element name includes a namespace label as a prefix, the corresponding CUE struct property will be keyed by the same name and include the same prefix.
8. XML element and attribute values are mapped to strings. Strings are taken as they appear in the XML including any whitespace, except for the cases below:
   • For a given element value, if the decoder detects mixed text/sub-element content and the text is anything other than whitespace, then the encoding returns an error. If the text in mixed text/sub-element content is just whitespace, then this whitespace is discarded.
   • In maintaining consistency with the CUE specification here, carriage return characters (\r) found in a string are discarded from that string.

Sample CUE constraints for XML using koala

Using the rules above, one would be able to write CUE constraints for XML as shown below:

Given an XML file with a note element and a book element, we could write a CUE schema to define types as shown below:

XML

<notes>
	<note alpha="abcd">hello</note>
	<quantity>5</quantity>
</notes>

CUE constraints

//the alpha attribute of notes:note should be a string
notes: note: $alpha: string

//the text within the notes:quantity element should represent an integer
notes: quantity: $$: =~"^[0-9]+$"

Mapping examples

The examples below illustrate each of the mapping rules defined above:

1. Elements

The XML note element below maps to the note struct in CUE.

XML

<note>
</note>

CUE

{
	note: { }
}

2. Nested Elements

Nesting an XML to element to the note element from the first example results in a nested CUE to struct inside the note struct.

XML

<note>
	<to></to>
</note>

CUE

{
	note: {
		to: {}
	}
}

3. Attributes

The alpha attribute of the note element in XML below maps to the $ prefixed $alpha property of the note struct in CUE.

XML

<note alpha="abcd">
</note>

CUE

{
	note: {
		$alpha: "abcd"
	}
}

4. Content

The content of the note XML element below maps to the value of the $$ property of the note struct in CUE.

XML

<note alpha="abcd">
	hello
</note>

CUE

{
	note: {
		$alpha: "abcd"
		$$:     "hello"
	}
}

5. XML Lists

The multiple XML note elements at the same level map to a list of note structs in CUE.

XML

<notes>
	<note alpha="abcd">hello</note>
	<note alpha="abcdef">goodbye</note>
</notes>

CUE

{
	notes: {
		note: [{
			$alpha: "abcd"
			$$:     "hello"
		}, {
			$alpha: "abcdef"
			$$:     "goodbye"
		}]
	}
}

6 and 7. Namespace Definitions and References

The h and r XML namespace definitions declared in the table XML element are declared as properties of the h:table struct in CUE.
Note how the namespace prefixed XML element names like h:table, h:tr, h:td and r:blah carry across to the key names of their corresponding CUE structs.

XML

<h:table xmlns:h="http://www.w3.org/TR/html4/" xmlns:r="d">
  <h:tr>
    <h:td>Apples</h:td>
    <h:td>Bananas</h:td>
    <r:blah>e3r</r:blah>
  </h:tr>
</h:table>

CUE

{
	"h:table": {
		"$xmlns:h": "http://www.w3.org/TR/html4/"
		"$xmlns:r": "d"
		"h:tr": {
			"h:td": [{
				$$: "Apples"
			}, {
				$$: "Bananas"
			}]
			"r:blah": {
				$$: "e3r"
			}
		}
	}
}

8. Element and attribute values

XML element and attribute values map to strings, as shown in the example below.

XML

<notes>
	<note alpha="true">5</note>
	<note alpha="abcdef    ">5.14</note>
</notes>

CUE

{
	notes: {
		note: [{
			$alpha: "true"
			$$:     "5"
		}, {
			$alpha: "abcdef    "
			$$:     "5.14"
		}]
	}
}

Alternative Conventions Considered

Although no known conventions exist to map from XML to CUE, there are a number of known mappings that take XML to JSON, which we can take inspiration from.

Parker and Spark Conventions

The Parker and Spark conventions use a very simplistic model where XML elements are mapped to object properties, and attributes are ignored.

We wish to maintain attribute information so we cannot use these mapping conventions as a whole.

Badgerfish

The Badgerfish convention maps elements, attributes, and content from XML to JSON. We follow the many of the rules in the Badgerfish convention, described here. Notable differences are listed below to allow for mapping to CUE and for increased readability:

• XML attributes map to CUE properties starting with a $ prefix instead of an @ prefix, given @ is already reserved in CUE for CUE attributes. Although we could still use the @ prefix using quotes in CUE, we do not want to overload the usage of @ for two concepts (ie: for XML attribute prefixes and for CUE attributes). Using the $ prefix will also provide a less verbose notation given quotes do not need to be used with this prefix.

• Given a single $ is not a valid identifier in CUE, we use $$ as the property to model element text content instead of $. Although we could use a quoted "$" as the key, we avoid this to prevent ambiguity with usage of $ in other contexts, (such as "root element" in JSONPath), and to minimize usage of quotes.

• For namespaces, we do not recursively define namespaces in nested objects as this would un-necessarily increase verbosity in the mapped CUE. Instead we align more closely to how namespaces are defined in the XML, and only define namespaces in the CUE at the same level as they are declared in the XML. To illustrate how koala simplifies the mapping, we provide the example below (Badgerfish mapping taken from here):

XML

<alice xmlns="http://some-namespace" xmlns:charlie="http://some-other-namespace"> 
    <bob>david</bob> 
    <charlie:edgar>frank</charlie:edgar> 
</alice>

Badgerfish

{ 
	alice: {
		bob: {
			$: "david"
			"@xmlns": {
				charlie: "http://some-other-namespace"
				$:       "http://some-namespace"
			}
		}
		"charlie:edgar": {
			$: "frank"
			"@xmlns": {
				charlie: "http://some-other-namespace"
				$:       "http://some-namespace"
			}
		}
		"@xmlns": {
			charlie: "http://some-other-namespace"
			$:       "http://some-namespace"
		}
	}
}

koala

{ 
	alice: {
		"$xmlns:charlie": "http://some-other-namespace"
		$xmlns:           "http://some-namespace"
		bob: {
			$$: "david"
		}
		"charlie:edgar": {
			$$: "frank"
		}
	}
}

GData

The GData convention is similar to Badgerfish, however makes no distinction between identifiers used for elements and those used for attributes.

Unlike the Badgerfish convention, if one were to use this convention to map from XML to CUE, it would mean that it becomes ambiguous whether you are referring to an attribute or to an element when writing a CUE constraint. Further, it is not clear from the rules specified here what happens when there is a collision between an element name and an attribute name.

Abdera

This convention is similar to the GData convention, however, it uses separate children and attribute abstractions when both nested XML elements and attributes are mentioned. Having to mention children and/or attributes in CUE constraints, as well as integer indexes for children arrays increases verbosity and complexity, which goes against the readability objective of this paper. To illustrate this with an example for Abdera:

XML

<note myAttr="attrVal" attrTwo="two">
	<point>example</point>
	<sample>other</other>
</note>

would map to:

CUE

{
	note: {
		attributes:  {
			myAttr:  "attrVal"
			attrTwo: "two"
		}
		children: [ {
			point:  "example"
			sample: "other"
		}]
	}		
}

JsonML

Short for JSON Markup Language, this convention makes heavy use of arrays to ensure an order-preserving mapping, where each element maps to an array entry, and each attribute also maps to an array entry. An example mapping is shown here.

Having to work out (count) integer indexes when writing a CUE constraint rather than just simply using the element and attribute identifiers found in the XML makes this mapping too unwieldy to use for the purposes of our mapping.

Testing Plan

The XML to CUE mapping scenarios required are covered by the examples described here. We will consider the solution complete once it can both decode and encode the examples shown there, along with any other test cases requested by the CUE maintainer team.

Deployment Plan

The new koala encoding will not be the default XML encoding, but rather an opt-in encoding. Users will be able to use this from the command line using a command similar to:

cue vet schema.cue xml+koala: data.xml

Given this is not the default encoding, the command below would not work:

cue vet schema.cue data.xml

This will initially be an experimental encoding, which will be specified in the documentation, however given that it requires opt-in when the xml encoding to be used is specified, it does not need to be toggled using the CUE_EXPERIMENT variable as other experimental features are.

We also note that embedded XML within CUE will not be supported on day 1.

[mvdan]

Thank you for writing up this proposal! I'll write some minor high level thoughts we've had about it in this comment here, and split up some other thoughts into separate comments so they can spawn their own sub-threads without being mixed together.

1. Using $ like badgerfish would be possible in quotes, like "$": - please clarify that it's a possibility but that it's discarded due to the ambiguity with the meaning of $ in other contexts, such as "root element" in JSONPath.
2. Please clarify the rule for mapping XML text to CUE literals: for example: "if the text, with whitespace removed, is either true, false, null or a plain decimal number, the result is that CUE literal; otherwise it's a string containing the text including surrounding whitespace".
3. Multiple XML elements at the same level map to multiple CUE structs forming part of a CUE list at that level - presumably this should say "with the same tag name"? Otherwise it almost sounds like example 8 should end up with one list of five elements.

[matthew-sladescu]

Thanks for the review Daniel and team. I'll address the points below:

1. Using $ like badgerfish would be possible in quotes, like "$": - please clarify that it's a possibility but that it's discarded due to the ambiguity with the meaning of $ in other contexts, such as "root element" in JSONPath.

We've now added clarification for this under the Badgerfish comparison section.

2. Please clarify the rule for mapping XML text to CUE literals: for example: "if the text, with whitespace removed, is either true, false, null or a plain decimal number, the result is that CUE literal; otherwise it's a string containing the text including surrounding whitespace".

We will add more information on the type mapping to the proposal soon.

3. Multiple XML elements at the same level map to multiple CUE structs forming part of a CUE list at that level - presumably this should say "with the same tag name"? Otherwise it almost sounds like example 8 should end up with one list of five elements.

We've updated rule 5 with this information, alongside additional information on ordering and the list key.

[matthew-sladescu]

RE the type mapping above: we've updated the proposal to map XML element and attribute values to strings rather than trying to infer a type from the XML text.

Using a prescriptive set of rules to map from a string to a type may not suit everyone's needs, so we have made this simplification to avoid forcing a given element/attribute value to be a certain type for this encoding.

[mvdan]

Thanks - can you clarify that the decoded CUE ends up as a string, and whether all whitespace is kept or not? e.g. tab/space indentation, newlines, possible CRLF line endings from Windows, trailing whitespace in lines.

[matthew-sladescu]

Sure, we've updated Rule 8 to clarify this.

[mvdan]

Following my point 3 above, what would happen for repeated XML elements with the same tag name but different namespaces? For example:

<doc>
    <A xmlns="http://foo.example">true</A>
    <A xmlns="http://bar.example">1</A>
</doc>

Interestingly enough, badgerfish seems to not be clear on this edge case.

[matthew-sladescu]

This would map to:

doc: {
  A: [{     
        $xmlns: "http://foo.example/"      
        $$: "true"    
    },    
    {      
        $xmlns: "http://bar.example/"      
        $$: "1"    
    }]
}

(updated to match updates for Rule 8)

[mvdan]

It's worth pointing out that your example

<!-- Blob 1 -->

<h:table xmlns:h="http://www.w3.org/TR/html4/" xmlns:r="d">
  <h:tr>
    <h:td>Apples</h:td>
    <h:td>Bananas</h:td>
    <r:blah>e3r</r:blah>
  </h:tr>
</h:table>

and a variant with the same namespace URIs but with different prefixes, like

<!-- Blob 2 -->

<z:table xmlns:z="http://www.w3.org/TR/html4/" xmlns:r="d">
  <z:tr>
    <z:td>Apples</z:td>
    <z:td>Bananas</z:td>
    <r:blah>e3r</r:blah>
  </z:tr>
</z:table>

are effectively representing the same data, but would result in different decoded CUE. This affects badgerfish as well, so it's presumably not a significant problem.

Update: light edit by @myitcv to name the blobs of XML

[infogulch]

I guess it depends on what granularity at which you want to emulate xml. Imagine a slider that goes from "syntax" to "semantics". Treating both prefixes as equivalent is closer to the "semantics" side, and treating them as different is closer to the "syntax" side. (Or maybe you're greedy and you want to be able to choose the evaluation level arbitrarily.)

[matthew-sladescu]

The first XML snippet would map to:

"h:table" : {
    "$xmlns:h": "http://www.w3.org/TR/html4/",
    "$xmlns:r": "d",

    "h:tr": {
        "h:td": [{
            $$: "Apples"
        }, {
            $$: "Bananas"
        }]
        "r:blah": {
            $$: "e3r
        }
    }
}

and the second snippet would map to:

"z:table" : {
    "$xmlns:z": "http://www.w3.org/TR/html4/",
    "$xmlns:r": "d",

    "z:tr": {
        "z:td": [{
            $$: "Apples"
        }, {
            $$: "Bananas"
        }]
        "r:blah": {
            $$: "e3r
        }
    }
}

Here, even though both z and h refer to the same namespace, we still include the prefix for the namespace in the mapping so that we don't loose this prefix name information, but also so that it's more natural for the constraint author to write constraints against the same tag text they see in the XML.

[infogulch]

I would like to take a second to acknowledge that this discussion is about emulating xml semantics in cue, and that that is insane and impressive at the same time.

[matthew-sladescu]

Thanks Joe. We're hoping this will help a lot of folks out there who would love to use CUE with XML. There were conventions out there for mapping from XML to JSON that we've listed in the proposal which inspired us to build on them for a mapping from XML to CUE.

[mvdan]

As of 096a114, the xml+koala encoding is available in CUE at master. For now, we have only added the decoder; the encoder will come separately.

Please note that this new encoding is experimental for now, given that it's a bespoke design that might need to be tweaked as we gain experience with it.

There is a Go package available here: https://pkg.go.dev/cuelang.org/go/encoding/xml/koala

And the CLI supports it as follows:

$ cat hello.xml 
<root attr="foo">bar baz</root>
$ cue export xml+koala: hello.xml 
{
    "root": {
        "$attr": "foo",
        "$$": "bar baz"
    }
}