I've build a job that copy data from a mysql db table to b mysql table.
The table columns are the same except sometimes a new column can be added in table a db.
i want to retrieve all the columns from a to b but only those that exists in table b. i was able to put in the query specific select colume statment that exists in table b like:
select coulmn1,column2,columns3... from table a
the issue is if i add a new column in b that matches a the talend job schema in Mysqlinput should be changed as well cause i work with build in type.
Is there a way to force the schema columns during the job running?
If you are using a subscription version of Talend, you can use the dynamic column type. You can define a single column for your input of type "Dynamic" and map it to a column of the same type in your output component. This will dynamically get columns from table a and map them to the same columns in table b. Here's an example.
If you are using Talend Open Studio, things get a little trickier as Talend expects a list of columns for the input and output components that need to be defined at design time.
Here's a solution I put together to work around this limitation.
The idea is to list all table a's columns that are present in table b. Then convert it to a comma separated list of columns, in my example id,Theme,name
and store it in a global variable COLUMN_LIST
. A second output of the tMap builds the same list of columns, but this time putting single quotes between columns (so as they can be used as parameters to the CONCAT
function later), then add single quotes to the beginning and end, like so: "'", id,"','",Theme,"','",name,"'"
and store it in a global variable CONCAT_LIST
.
On the next subjob, I query table a
using the CONCAT
function, giving it the list of columns to be concatenated CONCAT_LIST
, thus retrieving each record in a single column like so 'value1', 'value2',..etc
Then at last I execute an INSERT
query against table b
, by specifying the list of columns given by the global variable COLUMN_LIST
, and the values to be inserted as a single string resulting from the CONCAT
function (row6.values
).
This solution is generic, if you replace your table names by context variables, you can use it to copy data from any MySQL table to another table.