Shred SQL Server XML with Unicode characters

问题:

In my previous question SQL Server XML String Manipluation

I was given the answer below (Thanks Mikael Eriksson) to shred an XML document, and strip out unwanted words out characters. I now need to take it a step further, and strip out Unicode characters over 255. When I have those characters in my XML, they get stored in the @T table variable (in the code below) as question marks. How can I get those characters to come through as the actual Unicode characters, so I can strip them out?

I have a function that works nicely to remove the unwanted characters, but since the Unicode comes in as question marks, it doesn't touch them

 -- A table to hold the bad words
declare @BadWords table
(
  ID int identity,
  Value nvarchar(10)
)

-- These are the bad ones.
insert into @BadWords values
('one'),
('three'),
('five'),
('hold')

-- XML that needs cleaning
declare @XML xml = '
<root>
  <itemone ID="1one1">1one1</itemone>
  <itemtwo>2two2</itemtwo>
  <items>
    <item>1one1</item>
    <item>2two2</item>
    <item>onetwothreefourfive</item>
  </items>
  <hold>We hold these truths to be self evident</hold>
</root>
'

-- A helper table to hold the values to modify
declare @T table
(
  ID int identity,
  Pos int,
  OldValue nvarchar(max),
  NewValue nvarchar(max),
  Attribute bit
)

-- Get all attributes from the XML
insert into @T(Pos, OldValue, NewValue, Attribute)
select row_number() over(order by T.N),
       T.N.value('.', 'nvarchar(max)'),
       T.N.value('.', 'nvarchar(max)'),
       1
from @XML.nodes('//@*') as T(N)

-- Get all values from the XML
insert into @T(Pos, OldValue, NewValue, Attribute)
select row_number() over(order by T.N),
       T.N.value('text()[1]', 'nvarchar(max)'),
       T.N.value('text()[1]', 'nvarchar(max)'),
       0
from @XML.nodes('//*') as T(N)

declare @ID int
declare @Pos int
declare @Value nvarchar(max)
declare @Attribute bit

-- Remove the bad words from @T, one bad word at a time
select @ID = max(ID) from @BadWords
while @ID > 0
begin
  select @Value = Value
  from @BadWords
  where ID = @ID

  update @T
  set NewValue = replace(NewValue, @Value, '')

  set @ID -= 1
end

-- Write the cleaned values back to the XML
select @ID = max(ID) from @T
while @ID > 0
begin
  select @Value = nullif(NewValue, OldValue),
         @Attribute = Attribute,
         @Pos = Pos
  from @T
  where ID = @ID

  print @Attribute

  if @Value is not null
    if @Attribute = 1  
      set @XML.modify('replace value of ((//@*)[sql:variable("@Pos")])[1] 
                       with sql:variable("@Value")')
    else
      set @XML.modify('replace value of ((//*)[sql:variable("@Pos")]/text())[1] 
                           with sql:variable("@Value")')
  set @ID -= 1
end

select @XML

回答1:

This part looks off:

insert into @BadWords values
('one'),
('three'),
('five'),
('hold')

You need the N prefix for Unicode string literals. Without the N, your code is treating them as VARCHAR, and you get question marks for multibyte characters. There are other places you have to be using Unicode friendly strings as well. XML is usually UTF-8 and so should be able to handle Unicode characters, though the standard discourages these. Your code should look like:

insert into @BadWords values
(N'one'),
(N'three'),
(N'five'),
(N'hold')