Is it possible to stream an XML file to a MS SQL Stored Procedure as a parameter from a C# application without having to load it into memory on the c# end?
The XML files i have are quite large, ~600MB with 2mil entries. Which as you can imagine, takes a bucket load of memory.
Once the SP on SQL inserts/updates values in the database from these files it has no further use for them and can be disposed.
SQL Example:
CREATE PROCEDURE [dbo].[AddAllFromList]
(
@XmlData XML
)
BEGIN
Do something with @XmlData;
END;
C# Example
using (Stream XMLFileStream = new someFileInfo.OpenRead())
using (SqlConnection Connection = new SqlConnection(SQLConnectionString))
{
var opensql = Connection.OpenAsync();
SqlCommand sqlcommand = new SqlCommand("AddAllFromList", Connection)
{
CommandType = System.Data.CommandType.StoredProcedure,
CommandTimeout = 360
};
sqlcommand.Parameters.Add("@XmlData", SqlDbType.Xml).Value = XMLFileStream;
try
{
await sqlcommand.ExecuteNonQueryAsync()
}
catch
{
Console.WriteLine("Failed");
}
Would something like this work? Does anyone have any success stories?
With such a big load I'd suggest to do this in two steps anyway. First read this into a staging table, then do the rest from there.
If the file is located where SQL-Server has access you can load the XML from T-SQL directly.
This question asks for the upload of multiple XMLs (using a CURSOR
), but you will see how to go.
This question addresses the export to XML, but there are several important things to know about the file's encoding.
This question covers some traps one might get into.
In general
C# uses unicode (2-byte-encoding) in any case for strings. Such a string can be passed over to SQL Server's NVARCHAR(MAX)
type. If the XML is well-formed you can pass it directly to an XML
typed variable.
SQL-Server is very limited in encodings. Very often such big files are stored with utf-8
. The provided links give information about that.
SQL-Server is not running as you. If it is running on a different machine it will see other drives and paths as you see. And the rights may be different too.
Whether you load the XML with C# and pass it over to SQL-Server or if you use some command from T-SQL will be quite the same amout of data, which must be shipped from A to B.
If memory matters you might split this in parts.