Why Duplicate posts are being added while fetching

Question

I am trying to add posts in wordpress from rets server using PHRETS. Unfortunately duplicate posts are being added. I have used the WP Query to check the existing post using meta key and value. This code is running well when I am trying to add 10 or 50 posts but when I set the limit to 4000 it start adding duplicate posts. I have run this code so many time and flushing the database so many time. Here is a code sample:

$query = "(922=MIAMI), (246=A)";
$search = $rets->SearchQuery("Property", $class, $query, array("SystemName" => 1, 'Limit' =>4550));

if ($rets->NumRows($search) > 0) {
    $fields_order = $rets->SearchGetFields($search);

    while ($record = $rets->FetchRow($search)) {
        foreach ($fields_order as $fo) { 
            if ($fo == 'sysid') { $systemid = $record[$fo] ; }
            if ($fo == '881') { $saddress = isset($record[$fo]) ? $record[$fo] : ""; }
            if ($fo == '214') { $sremarks = isset($record[$fo]) ? $record[$fo] : ""; }
        }

        $porpertytitle = $saddress;

        $args = array(
            'numberposts' => -1,
            'post_type' => 'property',
            'post_status' => 'publish',
            'meta_key' =>'sysid',
            'meta_value' => $systemid
        );

        $the_query = new WP_Query($args);

        if($the_query->have_posts()) {
            while ($the_query->have_posts()) {
                $the_query->the_post();
                unset($systemid);
                unset($args);
            }
        } else {
            $my_listing = array(
                'post_title' => $porpertytitle,
                'post_type' => 'property',
                'post_content' => $sremarks,
                'post_status' => 'publish',
            );

            $listing_post_id = wp_insert_post($my_listing);

            if($listing_post_id > 0) {
                update_post_meta($listing_post_id, 'sysid', $systemid);
            }

            unset($systemid);
            unset($args);
            unset($listing_post_id);
        }
        wp_reset_postdata();
    }
}

Sorry for this long code. But I have also tried unset the variables after every loop but not working.

Answer 1

I had a similar seemingly random duplicate problem myself. I finally determined the problem is that RETS Server offsets are 1 based, not zero based. Also, if you don't pass an offset with your request the server will sort the results differently than if you do.

If you can inspect the request being sent to the server, see if it includes offset=1 on the first request. If it is zero or missing you'll get repeats of the first page's results scattered in subsequent pages, which can be a lot with a 4000 page size