Introducing Xtract, a new XML deserialization library for Scala

Thayne Mc Combs

Reading time: about 9 min

Topics:

Last summer, our team added support for importing Adobe InDesign files into Lucidpress, a cloud-based publishing app. Specifically, the app now supports IDML files, which consist of XML files packaged up in a .zip file. The process of conversion involves:

Parsing the XML from the IDML file into a model representing the InDesign document
Converting that to a model representing a Lucidpress document
Serializing that model into a Lucidpress file

Since we use the Play framework for our servers, we do all of this in Scala. To deserialize the XML to Scala classes, we wanted a tool with the following features:

Easily define how XML should be parsed into a class, ideally with declarative syntax
Perform validation and transformations on the input XML before storing in the class
Easily recover from errors, but keep information about the errors so we can inform users what went wrong and investigate it

I looked around but couldn't find anything that met each of these needs. However, we were pretty familiar with the Play! JSON library, which has functionality similar to what we wanted in the form of Reads combinators. We also knew that the magic of the Reads combinators was implemented in a more generic library at play.api.libs.functional, so we decided to build an xml library similar to the Play! JSON library for XML.

Best of all, we are happy to announce that Xtract is now open source and available on Github!

Description of the Xtract Library

The Xtract library has three main components:

ParseResult: The result of deserializing a chunk of XML. It can be a success containing the deserialized value, a failure containing a list of errors encountered, or a partial success which contains both a deserialized value and a list of errors that were recovered from. This is the equivalent of JsResult in the Play! JSON library.
XmlReader: An object that transforms an XML NodeSeq into a Scala object. This is the equivalent of a Reads in the Play! JSON library.
XPath: An object that represents a path in XML using a subset of xpath. This is used to select children and descendents of the root node when constructing XmlReaders. It is the equivalent of JsPath in the Play! JSON library.

Xtract provides several basic XmlReaders for primitive types and some basic Scala types such as Seq and Option. Using these you can create increasingly complex XmlReaders by composing simpler XmlReaders together using combinator syntax (see the example below). Once you have built an XmlReader for the root node of your XML, you can pass in a NodeSeq (from Scala's xml library) to convert it to a ParseResult containing your desired type. You can then get the resulting object and/or handle any errors that occurred.

A Blog Example

As an example let's suppose we have information about blog posts (meta right?) stored as XML and want to process it somehow, maybe render it as html, publish it to an RSS feed, etc. The first step would be deserializing it from XML. We will show how this can be done using Xtract. The full source code for the example project is available on Github.

Let's start by looking at the XML:

<blog type="technical">
    <head>
        <title>Introducing Xtract</title>
        <subtitle>A new xml deserialization library for scala</subtitle>
        <author first="Thayne"
                last="McCombs"
                department="engineering"
                canContact="false" />
    </head>
    <body>
        <!-- first section is intro, so no title -->
        <section>
            <p>This is the introduction to the blog.</p>
        </section>
        <section title="Section 1">
            <p>First Paragraph</p>
            <p>Second Paragraph</p>
            <p>Third Paragraph</p>
        </section>
        <ignoredtag>
            This isn't in an expected tag, so it is ignored.
        </ignoredtag>
    </body>
</blog>

Then let's define the Blog class to contain the data:

case class Blog(
  title: String,
  subtitle: Option[String],
  author: AuthorInfo,
  blogType: BlogType.Value,
  content: Content
)

Nothing special there, just a case class with the data for the blog. AuthorInfo, BlogType, and Content are defined elsewhere (and can be seen in the xtract-example project).

Defining how the Blog should be deserialized from XML is simply a matter of defining an XmlReader for the Blog type. This can be done as follows:

import com.lucidchart.open.xtract.{XmlReader, __}
import com.lucidchart.open.xtract.XmlReader._

import play.api.libs.functional.syntax._

// definition of Blog case class elided

object Blog {
  implicit val reader: XmlReader[Blog] = (
    (__ \ "head" \ "title").read[String] and
      (__ \ "head"\ "subtitle").read[String].optional and
      (__ \ "head" \ "author").read[AuthorInfo] and
      attribute("type")(enum(BlogType)).orElse(pure(BlogType.tech)) and
      (__ \ "body").read[Content]
  )(apply _)
}

The first thing you may notice is the imports. XmlReader is imported so we can use the type, and __ (two underscores) since it represents the root of the XML tree when using paths. Then, com.lucidchart.open.xtract.XmlReader._ is imported to get access to all the default readers, and play.api.libs.functional.syntax._ is imported to get access to the combinatorial syntax for XmlReaders.

Secondly, we define an implicit val inside the companion object of Blog so that the Xmlreader is implicitly available wherever the Blog is available. This makes it so that you just need to supply the type when deserializing from XML. It is possible to define the XmlReader by defining a subclass, or by creating an XmlReader from a function that takes a NodSeq and returns a ParseResult. However, it is usually easier to compose other XmlReaders together using the combinatorial syntax as seen in this definition.

The combinatorial syntax is simple. You combine two or more XmlReaders with the and operator. The result is a FunctionalBuilder which has an apply method that takes a function that takes one parameter for each of the XmlReaders of the reader's type and returns an object of the desired type (in this case a Blog). For a case class the function passed to the FunctionalBuilder is typically the constructor function for the case class.

Now let's look at the components of the reader. (__ \ "head" \ "title") create an XPath object equivalent to the xpath /head/title. That is it selects all title tags that are children of head tags that are children of the root element. In addition to \ there are a few other ways to specify an XML path:

// Select all descendant  tags of path (equivalent of "<path>//tag-name" in xpath)
path \\ "tag-name"
// Select the 4th element at path (equivalent of "<path>[4]" in xpath)
path(4)
// Select the attribute(s) at path with name "attr-name" (equivalent of "<path>@attr-name" in xpath)
path \@ "attr-name"

The .read[String] part gets an XmlReader[String] that reads a string from the node at the specified path. The read method takes an implicit XmlReader. In the case of String, that reader is provided by the XmlReader companion object, or it can be provided by a user-defined implicit reader, or explicitly supplied. The example above also illustrates the use of optional, which converts an XmlReader[T] to a XmlReader[Option[T]] that succeeds with None even if the original reader fails, and orElse combines one reader with another to use if the first one fails. In this case orElse is used to provide a default value using pure -- a function that turns a value into an XmlReader that always succeeds with that value.

Once the reader is defined, using it is extremely simple:

val parsedBlog = XmlReader.of[Blog].read(xml)

The of method simply uses the implicit reader for Blog to get an XmlReader and calling the read method does the actual deserialization, returning a ParseResult, from which the deserialized Blog or any parsing errors can be extracted.

This example only shows some of the features available. Please take a look at the full documentation here.

Conclusion

Xtract is an easy-to-use XML deserialization library. It uses combinatorial syntax to compose simple parsers together to create more complex parsers. It allows you to perform validation and transformation of data, and recover from errors using using easy-to-understand syntax. Although it was originally designed for parsing Indesign files, it is general enough to use for any XML parsing problem.

About Lucid

Lucid Software is a pioneer and leader in visual collaboration dedicated to helping teams build the future. With its products—Lucidchart, Lucidspark, and Lucidscale—teams are supported from ideation to execution and are empowered to align around a shared vision, clarify complexity, and collaborate visually, no matter where they are. Lucid is proud to serve top businesses around the world, including customers such as Google, GE, and NBC Universal, and 99% of the Fortune 500. Lucid partners with industry leaders, including Google, Atlassian, and Microsoft. Since its founding, Lucid has received numerous awards for its products, business, and workplace culture. For more information, visit lucid.co.