extension elements in sitemaps

[pbiron]

extension elements in sitemaps

Describe the bug

#88 added filters that allow extension elements to be added to sitemaps (e.g., core_sitemaps_posts_url_list).

However, there are a couple of problems with the solution:

1. Core_Sitemaps_Renderer::get_sitemap_xml() outputs all elements in the sitemaps namespace (i.e., http://www.sitemaps.org/schemas/sitemap/0.9) and the sitemaps XML Schema specifies that any element children of sitemap:url other than sitemap:loc, sitemap:lastmod, sitemap:changefreq and sitemap:priority must be in another namespace (where sitemap:xyz is a QName for an element whose local-name() is xyz and whose namespace-uri() is the sitemaps namespace URI). Therefore, if a plugin hooks into core_sitemaps_posts_url_list and adds a foo property to each URL in the list, the generated sitemap XML will be invalid against the XML Schema. There is currently no way for a plugin to tell the renderer what namespace to use for these extension elements.
2. The sitemaps XML Schema also specifies that the child elements of sitemap:url must be in the order above, with all elements in a foreign namespace coming at the end. So, if a plugin hooks into core_sitemaps_posts_url_list and does something like foreach ( $url_list as $url ) { $url = array_merge( array( 'priority' => 0.9 ), $url ); } then Core_Sitemaps_Renderer::get_sitemap_xml() will output a sitemap that is invalid against the XML Schema.

I do not know whether sitemap consumers (e.g., Google, Bing, Yandex, etc) would fail to process a sitemap that was invalid against the XML Schema, but do we want to try our best to ensure that generated sitemaps are valid?

[pbiron]

For example,

add_filter( 'core_sitemaps_posts_url_list', function( $url_list ) {
   foreach ( $url_list as &$url ) {
      $url['foo'] = 'bar';
   }
   return $url_list;
} );

Will produce something like:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <url>
      <loc>https://example.com/</loc>
      <lastmod>2020-03-01T18:27:27-07:00</lastmod>
      <foo>bar</loc>
   </url>
</urlset>

which is invalid against the sitemaps XML Schema, since the foo element must be in a namespace other than http://www.sitemaps.org/schemas/sitemap/0.9. The sitemap is well-formed XML, but not valid against the XML Schema.

Similarly,

add_filter( 'core_sitemaps_posts_url_list', function( $url_list ) {
   foreach ( $url_list as &$url ) {
      $url = array( 'priority' => 0.9 ) + $url;
   }
   return $url_list;
} );

Will produce something like:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <url>
      <priority>0.9</priority>
      <loc>https://example.com/</loc>
      <lastmod>2020-03-01T18:27:27-07:00</lastmod>
   </url>
</urlset>

which, again, is not valid against the XML Schema because the <loc> child element must come before the <priority> child element.

[pbiron]

I do not know whether sitemap consumers (e.g., Google, Bing, Yandex, etc) would fail to process a sitemap that was invalid against the XML Schema, but do we want to try our best to ensure that generated sitemaps are valid?

I had a closer look at the XML Schema while I was working on #153, and I sure hope that none of the major sitemap consumers require that sitemaps be valid against the schema...because they way the schema is written ensures that the average WP plugin author won't be able to produce sitemap instances containing extension elements that validate against the schema!

Why? Because the content model for Q{http://www.sitemaps.org/schemas/sitemap/0.9}url requires strict validation for elements in extension namespaces (which is a bad decision on the part of the schema author(s), IMHO):

<xsd:complexType name="tUrl">
   <xsd:annotation>
      <xsd:documentation>
         Container for the data needed to describe a document to crawl.
      </xsd:documentation>
   </xsd:annotation>
   <xsd:sequence>
      <xsd:element name="loc" type="tLoc"/>
      <xsd:element name="lastmod" type="tLastmod" minOccurs="0"/>
      <xsd:element name="changefreq" type="tChangeFreq" minOccurs="0"/>
      <xsd:element name="priority" type="tPriority" minOccurs="0"/>
      <xsd:any namespace="##other" minOccurs="0" maxOccurs="unbounded" processContents="strict"/>
   </xsd:sequence>
</xsd:complexType>

What that means is that a schema validator needs to be able to find a schema for elements in any extension namespaces. I think the chances are pretty low that the average WP plugin author is going to write a schema document for their extensions and make that schema document available on the web for validators to download.

Even if plugin authors did make schema documents available for their extension namespace(s), we'd have to add a filter for them to hook into to specify the URL for the schema document for their extension namespace and then output the appropriate markup in the sitemap instance so that validators knew where to find the schema for the extension namespace, e.g.

<?xml version="1.0" encoding="UTF-8"?>
<urlset
      xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" 
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
      xsi:schemaLocation="urn:sparrowhawk.com/core-sitemaps
                          https://sparrowhawkcomputing.com/schema/core-sitemaps.xsd"
   >
...
</urlset>

What the schema author(s) should have (IMHO) written is:

<xsd:any namespace="##other" minOccurs="0" maxOccurs="unbounded" processContents="lax"/>

which would allow validators to validate extension elements if they could find a schema definition for them, or silently ignore them for validation purposes if a schema definition for them couldn't be found.

[pbiron]

That said, I still think we should try to find a way for plugin authors to specify a namespace URI for extensions they add via the various core_sitemaps_{$object_type}_url_list filters.

I haven't come up a great way yet, but I'll throw the following good way out just for comment:

1. add a filter, such as core_site_maps_namespace_bindings, that plugins would use to add prefix to namespace URI bindings (see below for an example)
   a. we'd have to find a way to resolve conflicts if 2 different plugins tried to register the same prefix
   b. document that plugins that hook into the various core_sitemaps_{$object_type}_url_list filters should use PrefixedNames (i.e., Prefix . ':' . LocalName ) when they add keys to the $url_list
2. modify Core_Sitemap_Renderer::get_sitemap_xml() to:
   a. add namespace declarations for current bindings when creating the SimpleXMLElement (see below)
   b. when iterating over the $url_item keys, check whether the key is a PrefixedName for which there is a namespace binding and call addChild() with the proper arguments for SimpleXML to add the element in the correct namespace

So, a plugin would do:

add_filter( 'core_site_maps_namespace_bindings', function( $namespace_bindings ) {
	$namespace_bindings[ 'pvb' ] => 'urn:sparrowhawkcomputing.com';
} );
add_filter( 'core_sitemaps_posts_url_list', function( $url_list ) {
	foreach ( $url_list as &$url ) {
		$url['pvb:foo'] = 'bar';
	}
	
	return $url_list;
} );

Then, Core_Sitemap_Renderer::get_sitemap_xml() would become something like:

public function get_sitemap_xml( $url_list ) {
	$namespace_bindings = apply_filters( 'core_site_maps_namespace_bindings', array() );

	$namespace_decls = 'xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"';
	foreach ( $namespace_bindings as $prefix => $namespace_uri ) {
		$namespace_decls .= sprintf( ' xmlns:%s="%s"', $prefix, $namespace_uri );
	}
	$urlset = new SimpleXMLElement( '<?xml version="1.0" encoding="UTF-8" ?>' . $this->stylesheet . '<urlset ' . $namespace_decls . '/>' );
	
	foreach ( $url_list as $url_item ) {
		$url = $urlset->addChild( 'url', null, 'http://www.sitemaps.org/schemas/sitemap/0.9' );

 		foreach ( $url_item as $name => $value ) {
 			if ( ! is_scalar( $value ) ) {
 				// @todo add a _doing_it_wrong() since esc_attr() would generate a run-time PHP error.
 				continue;
 			} elseif ( in_array( $name, array( 'loc', 'lastmod', 'changefreq', 'priority' ) ) ) {
 				$url->addChild( $name, esc_attr( $value ), 'http://www.sitemaps.org/schemas/sitemap/0.9' );
 			} elseif ( self::is_xml_PrefixedName( $name ) ) {
 				list( $prefix, $local_name ) = explode( ':', $name );
 				if ( ! empty( $namespace_bindings[ $prefix ] ) ) {
	 				$url->addChild( $local_name, esc_attr( $value ), $namespace_bindings[ $prefix ] );
 				}
 			}
 			else {
 				// @todo add a _doing_it_wrong():
 				//       a. any other NCName wouldn't be valid against the sitemaps XML Schema
 				//       b. to avoid a bug in `SimpleXMLElement::addChild()`, which doesn't
 				//          check that the first param is a legal XML element name:-(  See below.
				continue;
			}
 		}
 	}

	return $urlset->asXML();
 }

and the new is_xml_PrefixedName() method is:

/**
 * Does a string match the PrefixedName production from the Namespaces in XML spec?
 *
 * @link https://www.w3.org/TR/REC-xml-names/#NT-PrefixedName
 *
 * The regexes used by this method are based on the corresponding
 * productions in the {@link https://www.w3.org/TR/REC-xml/ Extensible Markup Language (XML) 1.0 (Fifth Edition)}
 * and {@link https://www.w3.org/TR/REC-xml-names/ Namespaces in XML} W3C specifications.
 *
 * @param string $str The string to check.
 * @return bool True is `$str` matches the PrefixedName production, false otherwise.
 *
 * @todo Strech goal: create a fillblown core XML API, that added an abstraction layer
 *       around SimpleXMLElement (and/or XMLReader/XMLWriter) that a number of useful
 *       functions/methods like this, e.g., is_xml_NCName(), is_xml_QName(), etc.
 *       Of course, such an API is beyond the scope of this project (which is why
 *       it's a "strech goal" :-), but would be really helpful for feeds, WXR import/export, etc!
 */
static function is_xml_PrefixedName( $str ) {
	$NameStartChar = 'A-Za-z_\x{C0}-\x{D6}\x{D8}-\x{F6}\x{F8}-\x{2FF}\x{370}-\x{37D}\x{37F}-\x{1FFF}\x{200C}-\x{200D}\x{2070}-\x{218F}\x{2C00}-\x{2FEF}\x{3001}-\x{D7FF}\x{F900}-\x{FDCF}\x{FDF0}-\x{FFFD}\x{10000}-\x{EFFFF}';
	$NameChar      = "{$NameStartChar}\-\.0-9\x{B7}\x{0300}-\x{036F}\x{203F}-\x{2040}";
	$NCName        = "[{$NameStartChar}][{$NameChar}]*";
	$PrefixedName  = "{$NCName}:{$NCName}";

	return 1 === preg_match( "/^{$PrefixedName}\$/u", $str );
}

To illustrate the SimpleXMLElement bug mentioned in the @todo above, try the following:

add_filter( 'core_sitemaps_posts_url_list', function( $url_list ) {
	foreach ( $url_list as &$url ) {
		$url['this is not well-formed'] = '';
	}
	
	return $url_list;
} );

and you'll see that what SimpleXMLElement generates is:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <url>
      <loc>https://example.com/</loc>
      <this is not well-formed/>
   </url>
</urlset>

Comments?

[pbiron]

And while I'm being pedantic (I'm in one of those moods today...I'll blame it on Covid-ID isolation :-), another bad thing about the sitemaps XML Schema is the type definition for the Q{http://www.sitemaps.org/schemas/sitemap/0.9}loc element is:

<xsd:simpleType name="tLoc">
  <xsd:restriction base="xsd:anyURI">
    <xsd:minLength value="12"/>
    <xsd:maxLength value="2048"/>
  </xsd:restriction>
</xsd:simpleType>

which means that:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <url>
      <loc>http://a.tv</loc>
   </url>
</urlset>

is not a valid sitemap (since the URL is shorter than 12 characters).

Note that http://a.tv is a real URL that resolves!

OK, end of pedantic ranting (for today :-)

[pbiron]

So, a plugin would do:

I just realized that I forgot to show the sitemap that would be generated by the suggested mods above:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:pvb="urn:sparrowhawkcomputing.com">
   <url>
      <loc>https://example.com</loc>
      <pvb:foo>bar</pvb:foo>
   </url>
<urlset>

[joemcgill]

Thanks for clearly explaining the name spacing issue with the XML sitemaps spec and illustrating with examples. I wonder if instead of a simple filter approach for adding custom values to the array, we need to provide a registry for URL properties which requires you to add a namespace in order to register the property. Something like this pseudocode:

register_sitemap_property( $namespace, $name, $callback );

We can pass some data to any registered callback and if it returns a truthy value, then a property is returned that looks like <$namespace:$name>$value</$namespace:$name>. Thoughts?

[swissspidy]

@felixarntz @adamsilverstein would be interested to hear your thoughts on this

[adamsilverstein]

I'm not familiar enough with how sitemaps are typically extended to answer. I like the registry suggestion from @joemcgill which might provider a cleaner API than filters.

What do current popular sitemap plugins provide for extensibility?

[swissspidy]

In #184 I propose a rather simple solution for this. It does not cover some edge cases like the order of elements, which I think could be handled in a future release if necessary.