Remove duplicate XML elements using XSLT with perl

Question

I have this XML file:

<?xml version="1.0" encoding="UTF-8"?>
<d:dictionary xmlns="http://www.w3.org/1999/xhtml" xmlns:d="http://www.apple.com/DTDs/DictionaryService-1.0.rng">
    <d:entry id="a" d:title="a">
        <d:index d:value="a" d:title="a"/>
        <d:index d:value="b" d:title="b"/>
        <d:index d:value="a" d:title="a"/>
        <d:index d:value="c" d:title="c"/>
        <d:index d:value="b" d:title="b"/>
        <d:index d:value="a" d:title="a"/>
        <d:index d:value="b" d:title="b"/>
        <div>This is the content for entry.</div>
    </d:entry>
    <d:entry id="b" d:title="b">
        <d:index d:value="a" d:title="a"/>
        <d:index d:value="b" d:title="b"/>
        <div>This is the content for entry.</div>
    </d:entry>
</d:dictionary>

I'm trying to remove the duplicate <d:index of the entries using XSLT following this posting: https://stackoverflow.com/a/56898207/589924

Note: Every entry have its own independent <d:index, i.e. same index in different entries should not count as a duplicate. And the resulting xml should honor the original xml format.

The xsl file is like this:

<xsl:stylesheet version="2.0"
            xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
            xmlns:d="http://www.apple.com/DTDs/DictionaryService-1.0.rng">
<xsl:template>
    <xsl:copy>
        <xsl:for-each-group select="d:index"
                            group-by="concat(@d:value, '~', @d:title)">
            <xsl:copy-of select="current-group()[1]"/>
        </xsl:for-each-group>
        <xsl:copy-of select="div"/>
    </xsl:copy>
</xsl:template>
</xsl:stylesheet>

But the result is not expected, it removes all tags except for the content of div.

<?xml version="1.0"?>









This is the content for entry.




This is the content for entry.

Answer 1

Sometimes using directly programming libraries may be easier. Following a Perl script using XML::DT

#!/usr/bin/perl
use XML::DT;
my $filename = shift;
my %seen=();

my %handler=(
    'd:entry' => sub{ %seen=(); toxml },                     ## reset seen
    'd:index' => sub{ if ($seen{$v{"d:value"}}++){""    }    ## $v{id} -- attribute id 
                      else                       {toxml}},
);
print dt($filename, %handler);

as usual, sudo cpan XML::DT if not installed.

Answer 2

Using the Muenchian method for grouping:

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
 xmlns:d="http://www.apple.com/DTDs/DictionaryService-1.0.rng" exclude-result-prefixes="d">
 <xsl:output omit-xml-declaration="yes" indent="yes"/>
 <xsl:strip-space elements="*"/>
 <xsl:key name="kIndexByValueTitle" match="d:index"
          use="concat(generate-id(..), '+', @d:value, '+', @d:title)"/>

  <xsl:template match="node()|@*">
    <xsl:copy>
      <xsl:apply-templates select="node()|@*"/>
    </xsl:copy>
  </xsl:template>

  <xsl:template match=
   "d:index[not(generate-id()
               = generate-id(key('kIndexByValueTitle',
                                  concat(generate-id(..), '+', @d:value, '+', @d:title)
                                 )
                                  [1]))]" />
</xsl:stylesheet>

When this transformation is applied against the provided XML document:

<d:dictionary xmlns="http://www.w3.org/1999/xhtml"
 xmlns:d="http://www.apple.com/DTDs/DictionaryService-1.0.rng">
    <d:entry id="a" d:title="a">
        <d:index d:value="a" d:title="a"/>
        <d:index d:value="b" d:title="b"/>
        <d:index d:value="a" d:title="a"/>
        <d:index d:value="c" d:title="c"/>
        <d:index d:value="b" d:title="b"/>
        <d:index d:value="a" d:title="a"/>
        <d:index d:value="b" d:title="b"/>
        <div>This is the content for entry.</div>
    </d:entry>
    <d:entry id="b" d:title="b">
        <d:index d:value="a" d:title="a"/>
        <d:index d:value="b" d:title="b"/>
        <div>This is the content for entry.</div>
    </d:entry>
</d:dictionary>

the wanted, correct result is produced:

<d:dictionary xmlns:d="http://www.apple.com/DTDs/DictionaryService-1.0.rng" xmlns="http://www.w3.org/1999/xhtml">
   <d:entry id="a" d:title="a">
      <d:index d:value="a" d:title="a"/>
      <d:index d:value="b" d:title="b"/>
      <d:index d:value="c" d:title="c"/>
      <div>This is the content for entry.</div>
   </d:entry>
   <d:entry id="b" d:title="b">
      <d:index d:value="a" d:title="a"/>
      <d:index d:value="b" d:title="b"/>
      <div>This is the content for entry.</div>
   </d:entry>
</d:dictionary>