Transferring Millions of rows from teradata to myS

2019-09-16 03:28发布

问题:

I have to transfer around 5 million rows of data from Teradata to MySQL. Can anyone please suggest me the fastest way to do this over the network, without using the filesystem. I am new to Teradata and MySQL. I want to run this transfer as a batch job on weekly basis, so I am looking for the solution which can be fully automated. Any suggestions or hints will be greatly appreciated.

I have already written the code using JDBC to get the records from the Teradata and insert them into the MySQL. But it is very slow, so I am looking to make that code more efficient. I kept in generic because I didn't have the solution to be constrained by my implementation, as along with making existing code more efficient I am open to other alternatives also. But I don't want to use the file system since it's not easier to maintain or update the scripts.

My implementation:

Getting records from teradata:

connection  =   DBConnectionFactory.getDBConnection(SOURCE_DB);

    statement = connection.createStatement();
    rs = statement.executeQuery(QUERY_SELECT);
    while (rs.next()) {

        Offer offer = new Offer();
        offer.setExternalSourceId(rs.getString("EXT_SOURCE_ID"));
        offer.setClientOfferId(rs.getString("CLIENT_OFFER_ID"));
        offer.setUpcId(rs.getString("UPC_ID"));

        offers.add(offer);
    }

Inserting the records in mySQL:

int count = 0;
    if (isUpdated) {
        for (Offer offer : offers) {

            count++;

            stringBuilderUpdate = new StringBuilder();
            stringBuilderUpdate = stringBuilderUpdate
                    .append(QUERY_INSERT);

            stringBuilderUpdate = stringBuilderUpdate.append("'"
                    + offer.getExternalSourceId() + "'");

            statement.addBatch(stringBuilderUpdate.toString());

            queryBuilder = queryBuilder.append(stringBuilderUpdate
                    .toString() + SEMI_COLON);

            if (count > LIMIT) {
                countUpdate = statement.executeBatch();
                LOG.info("DB update count : " + countUpdate.length);
                count = 0;
            }

        }
        if (count > 0) {
            // Execute batch
            countUpdate = statement.executeBatch();
        }

Can anybody please tell me if we can make this code more efficient ???

Thanks

PS: Please ignore the syntax error in above code as this code is working fine. Some info might be missing because of copy and paste.

回答1:

The fastest method of importing data to MySQL is by using LOAD DATA INFILE or mysqlimport, which is a command line interface to LOAD DATA INFILE and it involves loading data from a file, preferably residing on a local filesystem.

When loading a table from a text file, use LOAD DATA INFILE. This is usually 20 times faster than using INSERT statements.

Therefore despite the fact that you don't want to use the filesystem I'd suggest to consider creating a dump to a file, transfer it to a MySQL server and use above mentioned means to load the data.

All these tasks can be fully automated via scripting.