Unit testing for usql applier and scripts

Question

I have a custom USql applier which extends the IApplier class.

[SqlUserDefinedApplier]
public class CsvApplier : IApplier
{
    public CsvApplier()
    {
        //totalcount = count;
    }
    public override IEnumerable<IRow> Apply(IRow input, IUpdatableRow output)
    {
        //....custom logic
        //yield return or yield break
    }
}

This applier is then used from Usql script as

@log =
SELECT t.ultimateID,
t.siteID,
.
.
.
t.eTime,
t.hours
FROM @logWithCount
CROSS APPLY
new BSWBigData.USQLApplier.CsvApplier() AS t(ultimateID string, siteID string, .... , eTime string, hours double, count long?);

I have been able to write unit tests/ATPs for decoupled parts of applier.

How can i write tests for C# code of Apply method and the custom logic dependent on input/output? How can i automate testing of usql scripts with defined inputs and outputs such that no data lake account is needed?

Answer 1

Before we ship the standalone U-SQL Localrun SDK package, please get the needed files from where you installed the ADLA Tool for VS.

In VS2015, all the dependencies needed for local run are in “C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\Microsoft\Microsoft Azure Data Lake DRI Tools for Visual Studio 2015\2.0.XXXX.0\LocalRunSDK. You will also need “C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\Microsoft\Microsoft Azure Data Lake DRI Tools for Visual Studio 2015\2.0.XXXX.0\CppSDK”

Just copy files from these two folders and place them somewhere like:

C:\USQLLocalRunSDK

Then you can use "LocalRunHelper.exe" to compile and run your u-sql scripts locally. On the compilation commandline, you need two options: -DataRoot "Where your local metadata and data are stored", the same as that setup option in ADLA Tool. -CppSDK "Where you copied the CppSDK files" That tool will return -1 upon error (compile or run) and 0 on success.

Just run the tool to see the list of command line options.

Answer 2

As @Michael Rys mentioned, currently it can be done by using the U-SQL local run, but not necessarily in Visual Studio.

We are trying to address this by providing you a "U-SQL Local Run SDK", which you can use as a way to run your scripts locally. Then you can utilize it to test the U-SQL scripts by writing some C# unit test wrappers and then run those unit tests in your CI system/build servers.

We are in the process of publishing this SDK via NuGet, but if you are interested in it, you can drop me an email at xiaoyzhu at Microsoft dot com and we can send you the bits we are trying to release.

Thanks

Xiaoyong Zhu from Azure Data Lake Team

Answer 3

The ADL Tools for VisualStudio have a local mode so you can do local execution also of UDOs. Although you may not get the full parallel processing locally, you should be able to test your code in the local run.