PHP output showing little black diamonds with a qu

Question

I'm writing a php program that pulls from a database source. Some of the varchars have quotes that are displaying as black diamonds with a question mark in them (�, REPLACEMENT CHARACTER, I assume from Microsoft Word text).

How can I use php to strip these characters out?

Answer 1

This happened to work in my case:

$text = utf8_decode($text)

I turns the black diamond character into a question mark so you can:

$text = str_replace('?', '', utf8_decode($text));

Answer 2

Based on your description of the problem, the data in your database is almost certainly encoded as Windows-1252, and your page is almost certainly being served as ISO-8859-1. These two character sets are equivalent except that Windows-1252 has 16 extra characters which are not present in ISO-8859-1, including left and right curly quotes.

Assuming my analysis is correct, the simplest solution is to serve your page as Windows-1252. This will work because all characters that are in ISO-8859-1 are also in Windows-1252. In PHP you can change the encoding as follows:

header('Content-Type: text/html; charset=Windows-1252');

However, you really should check what character encoding you are using in your HTML files and the contents of your database, and take care to be consistent, or convert properly where this is not possible.

Answer 3

I ran the "detect encoding" code after my collation change in phpmyadmin and now it comes up as Latin_1.

but here is something I came across looking a different data anomaly in my application and how I fixed it:

I just imported a table that has mixed encoding (with diamond question marks in some lines, and all were in the same column.) so here is my fix code. I used utf8_decode process that takes the undefined placeholder and assigns a plain question mark in the place of the "diamond question mark " then I used str_replace to replace the question mark with a space between quotes. here is the [code]

    include 'dbconnectfile.php';

  //// the variable $db comes from my db connect file
   /// inx is my auto increment column
   /// broke_column is the column I need to fix

      $qwy = "select inx,broke_column from Table ";
      $res = $db->query($qwy); 

      while ($data = $res->fetch_row()) {
      for ($m=0; $m<$res->field_count; $m++) {
           if ($m==0){ 
           $id=0;
           $id=$data[$m];
       echo $id;
           }else if ($m==1){ 
             $fix=0;
             $fix=$data[$m];


             $fix = utf8_decode($fix);
             $fixx =str_replace("?"," ",$fix);

        echo $fixx;

        ////I echoed the data to the screen because I like to see something as I execute it :)
            }
            }
         $insert= "UPDATE Table SET broke_column='".$fixx."'  where inx='".$id."'";
          $insresult= $db->query($insert);
      echo"<br>";
        }

        ?>