Keep NULL rows last on Dynamic Linq Order By

Question

I am using this snippet below for Ordering my Linq queries dynamically and works great. I am not great at reflection or complex linq queries but I need a way that when ascending order is used, that NULL values are last and vice versa.

So if my property name was an integer and the column values were 1, 3, 5, all NULL rows would be at the end, not at the beginning by default. What can I add to this expression to make that happen?

This code works with entity framework and still needs to for the NULL comparison.

Example

list.OrderBy("NAME DESC").ToList()

Class

   public static class OrderByHelper
    {
        public static IOrderedQueryable<T> ThenBy<T>(this IEnumerable<T> enumerable, string orderBy)
        {
            return enumerable.AsQueryable().ThenBy(orderBy);
        }

        public static IOrderedQueryable<T> ThenBy<T>(this IQueryable<T> collection, string orderBy)
        {
            if (string.IsNullOrWhiteSpace(orderBy))
                orderBy = "ID DESC";

            IOrderedQueryable<T> orderedQueryable = null;

            foreach (OrderByInfo orderByInfo in ParseOrderBy(orderBy, false))
                orderedQueryable = ApplyOrderBy<T>(collection, orderByInfo);

            return orderedQueryable;
        }

        public static IOrderedQueryable<T> OrderBy<T>(this IEnumerable<T> enumerable, string orderBy)
        {
            return enumerable.AsQueryable().OrderBy(orderBy);
        }

        public static IOrderedQueryable<T> OrderBy<T>(this IQueryable<T> collection, string orderBy)
        {
            if (string.IsNullOrWhiteSpace(orderBy))
                orderBy = "ID DESC";

            IOrderedQueryable<T> orderedQueryable = null;

            foreach (OrderByInfo orderByInfo in ParseOrderBy(orderBy, true))
                orderedQueryable = ApplyOrderBy<T>(collection, orderByInfo);

            return orderedQueryable;
        }

        private static IOrderedQueryable<T> ApplyOrderBy<T>(IQueryable<T> collection, OrderByInfo orderByInfo)
        {
            string[] props = orderByInfo.PropertyName.Split('.');
            Type type = typeof(T);

            ParameterExpression arg = Expression.Parameter(type, "x");
            Expression expr = arg;
            foreach (string prop in props)
            {
                // use reflection (not ComponentModel) to mirror LINQ
                PropertyInfo pi = type.GetProperty(prop, BindingFlags.IgnoreCase | BindingFlags.Public | BindingFlags.Instance);
                expr = Expression.Property(expr, pi);
                type = pi.PropertyType;
            }
            Type delegateType = typeof(Func<,>).MakeGenericType(typeof(T), type);
            LambdaExpression lambda = Expression.Lambda(delegateType, expr, arg);
            string methodName = String.Empty;



            if (!orderByInfo.Initial && collection is IOrderedQueryable<T>)
            {
                if (orderByInfo.Direction == SortDirection.Ascending)
                    methodName = "ThenBy";
                else
                    methodName = "ThenByDescending";
            }
            else
            {
                if (orderByInfo.Direction == SortDirection.Ascending)
                    methodName = "OrderBy";
                else
                    methodName = "OrderByDescending";
            }

            return (IOrderedQueryable<T>)typeof(Queryable).GetMethods().Single(
                method => method.Name == methodName
                        && method.IsGenericMethodDefinition
                        && method.GetGenericArguments().Length == 2
                        && method.GetParameters().Length == 2)
                .MakeGenericMethod(typeof(T), type)
                .Invoke(null, new object[] { collection, lambda });
        }

        private static IEnumerable<OrderByInfo> ParseOrderBy(string orderBy, bool initial)
        {
            if (String.IsNullOrEmpty(orderBy))
                yield break;

            string[] items = orderBy.Split(',');

            foreach (string item in items)
            {
                string[] pair = item.Trim().Split(' ');

                if (pair.Length > 2)
                    throw new ArgumentException(String.Format("Invalid OrderBy string '{0}'. Order By Format: Property, Property2 ASC, Property2 DESC", item));

                string prop = pair[0].Trim();

                if (String.IsNullOrEmpty(prop))
                    throw new ArgumentException("Invalid Property. Order By Format: Property, Property2 ASC, Property2 DESC");

                SortDirection dir = SortDirection.Ascending;

                if (pair.Length == 2)
                    dir = ("desc".Equals(pair[1].Trim(), StringComparison.OrdinalIgnoreCase) ? SortDirection.Descending : SortDirection.Ascending);

                yield return new OrderByInfo() { PropertyName = prop, Direction = dir, Initial = initial };

                initial = false;
            }

        }

        private class OrderByInfo
        {
            public string PropertyName { get; set; }
            public SortDirection Direction { get; set; }
            public bool Initial { get; set; }
        }

        private enum SortDirection
        {
            Ascending = 0,
            Descending = 1
        }

Answer 1

It's relatively simple. For each passed sort selector, the method executes one of the following:

.OrderBy(x => x.Member)
.ThenBy(x => x.Member)
.OrderByDescending(x => x.Member)
.ThenByDescendiong(x => x.Member)

When the x.Member type is reference type or nullable value type, the desired behavior can be achieved by pre ordering with the same direction by the following expression

x => x.Member == null ? 1 : 0

Some people use ordering by bool, but I prefer to be explicit and use conditional operator with specific integer values. So the corresponding calls for the above calls would be:

.OrderBy(x => x.Member == null ? 1 : 0).ThenBy(x => x.Member)
.ThenBy(x => x.Member == null ? 1 : 0).ThenBy(x => x.Member)
.OrderByDescending(x => x.Member == null ? 1 : 0).ThenByDescending(x => x.Member)
.ThenByDescending(x => x.Member == null ? 1 : 0).ThenByDescending(x => x.Member)

i.e. the original method on the pre order expression followed by the ThenBy(Descending) with the original expression.

Here is the implementation:

public static class OrderByHelper
{
    public static IOrderedQueryable<T> ThenBy<T>(this IEnumerable<T> source, string orderBy)
    {
        return source.AsQueryable().ThenBy(orderBy);
    }

    public static IOrderedQueryable<T> ThenBy<T>(this IQueryable<T> source, string orderBy)
    {
        return OrderBy(source, orderBy, false);
    }

    public static IOrderedQueryable<T> OrderBy<T>(this IEnumerable<T> source, string orderBy)
    {
        return source.AsQueryable().OrderBy(orderBy);
    }

    public static IOrderedQueryable<T> OrderBy<T>(this IQueryable<T> source, string orderBy)
    {
        return OrderBy(source, orderBy, true);
    }

    private static IOrderedQueryable<T> OrderBy<T>(IQueryable<T> source, string orderBy, bool initial)
    {
        if (string.IsNullOrWhiteSpace(orderBy))
            orderBy = "ID DESC";
        var parameter = Expression.Parameter(typeof(T), "x");
        var expression = source.Expression;
        foreach (var item in ParseOrderBy(orderBy, initial))
        {
            var order = item.PropertyName.Split('.')
                .Aggregate((Expression)parameter, Expression.PropertyOrField);
            if (!order.Type.IsValueType || Nullable.GetUnderlyingType(order.Type) != null)
            {
                var preOrder = Expression.Condition(
                        Expression.Equal(order, Expression.Constant(null, order.Type)),
                        Expression.Constant(1), Expression.Constant(0));
                expression = CallOrderBy(expression, Expression.Lambda(preOrder, parameter), item.Direction, initial);
                initial = false;
            }
            expression = CallOrderBy(expression, Expression.Lambda(order, parameter), item.Direction, initial);
            initial = false;
        }
        return (IOrderedQueryable<T>)source.Provider.CreateQuery(expression);
    }

    private static Expression CallOrderBy(Expression source, LambdaExpression selector, SortDirection direction, bool initial)
    {
        return Expression.Call(
            typeof(Queryable), GetMethodName(direction, initial),
            new Type[] { selector.Parameters[0].Type, selector.Body.Type },
            source, Expression.Quote(selector));
    }

    private static string GetMethodName(SortDirection direction, bool initial)
    {
        return direction == SortDirection.Ascending ?
            (initial ? "OrderBy" : "ThenBy") :
            (initial ? "OrderByDescending" : "ThenByDescending");
    }

    private static IEnumerable<OrderByInfo> ParseOrderBy(string orderBy, bool initial)
    {
        if (String.IsNullOrEmpty(orderBy))
            yield break;

        string[] items = orderBy.Split(',');

        foreach (string item in items)
        {
            string[] pair = item.Trim().Split(' ');

            if (pair.Length > 2)
                throw new ArgumentException(String.Format("Invalid OrderBy string '{0}'. Order By Format: Property, Property2 ASC, Property2 DESC", item));

            string prop = pair[0].Trim();

            if (String.IsNullOrEmpty(prop))
                throw new ArgumentException("Invalid Property. Order By Format: Property, Property2 ASC, Property2 DESC");

            SortDirection dir = SortDirection.Ascending;

            if (pair.Length == 2)
                dir = ("desc".Equals(pair[1].Trim(), StringComparison.OrdinalIgnoreCase) ? SortDirection.Descending : SortDirection.Ascending);

            yield return new OrderByInfo() { PropertyName = prop, Direction = dir, Initial = initial };

            initial = false;
        }

    }

    private class OrderByInfo
    {
        public string PropertyName { get; set; }
        public SortDirection Direction { get; set; }
        public bool Initial { get; set; }
    }

    private enum SortDirection
    {
        Ascending = 0,
        Descending = 1
    }
}

Answer 2

For dynamically constructed Order By expression like this list.OrderBy("NAME DESC").ToList(), you can use the following query helper extension method.

Usage

First of all, we check to make sure property name exists in the given Class. If we do not check, it'll throw run-time exception.

Then we use use either OrderByProperty or OrderByPropertyDescending.

string orderBy = "Name";
if (QueryHelper.PropertyExists<User>(orderBy))
{
   list = list.OrderByProperty(orderBy);
   - OR - 
   list = list.OrderByPropertyDescending(orderBy);
}

Here is the real world usage in my project at GitHub.

Query Helper

public static class QueryHelper
{
    private static readonly MethodInfo OrderByMethod =
        typeof (Queryable).GetMethods().Single(method => 
        method.Name == "OrderBy" && method.GetParameters().Length == 2);

    private static readonly MethodInfo OrderByDescendingMethod =
        typeof (Queryable).GetMethods().Single(method => 
        method.Name == "OrderByDescending" && method.GetParameters().Length == 2);

    public static bool PropertyExists<T>(string propertyName)
    {
        return typeof(T).GetProperty(propertyName, BindingFlags.IgnoreCase | 
            BindingFlags.Public | BindingFlags.Instance) != null;
    }

    public static IQueryable<T> OrderByProperty<T>(
       this IQueryable<T> source, string propertyName)
    {
        if (typeof (T).GetProperty(propertyName, BindingFlags.IgnoreCase | 
            BindingFlags.Public | BindingFlags.Instance) == null)
        {
            return null;
        }
        ParameterExpression paramterExpression = Expression.Parameter(typeof (T));
        Expression orderByProperty = Expression.Property(paramterExpression, propertyName);
        LambdaExpression lambda = Expression.Lambda(orderByProperty, paramterExpression);
        MethodInfo genericMethod = 
          OrderByMethod.MakeGenericMethod(typeof (T), orderByProperty.Type);
        object ret = genericMethod.Invoke(null, new object[] {source, lambda});
        return (IQueryable<T>) ret;
    }

    public static IQueryable<T> OrderByPropertyDescending<T>(
        this IQueryable<T> source, string propertyName)
    {
        if (typeof (T).GetProperty(propertyName, BindingFlags.IgnoreCase | 
            BindingFlags.Public | BindingFlags.Instance) == null)
        {
            return null;
        }
        ParameterExpression paramterExpression = Expression.Parameter(typeof (T));
        Expression orderByProperty = Expression.Property(paramterExpression, propertyName);
        LambdaExpression lambda = Expression.Lambda(orderByProperty, paramterExpression);
        MethodInfo genericMethod = 
          OrderByDescendingMethod.MakeGenericMethod(typeof (T), orderByProperty.Type);
        object ret = genericMethod.Invoke(null, new object[] {source, lambda});
        return (IQueryable<T>) ret;
    }
}

Answer 3

One approach is to pass an additional expression for testing for null into the method, and use it in an additional OrderBy/ThenBy clause.

Two OrderBy clauses would be produced - the first one will be on nullOrder, while the second one will be on the actual property.

private static IOrderedQueryable<T> ApplyOrderBy<T>(IQueryable<T> collection, OrderByInfo orderByInfo, Expression<Func<T,int>> nullOrder) {
    ...
    if (!orderByInfo.Initial && collection is IOrderedQueryable<T>) {
        if (orderByInfo.Direction == SortDirection.Ascending)
            methodName = "ThenBy";
        else
            methodName = "ThenByDescending";
    } else {
        if (orderByInfo.Direction == SortDirection.Ascending)
            methodName = "OrderBy";
        else
            methodName = "OrderByDescending";
    }
    if (nullOrder != null) {
        collection = (IQueryable<T>)typeof(Queryable).GetMethods().Single(
                method => method.Name == methodName
                        && method.IsGenericMethodDefinition
                        && method.GetGenericArguments().Length == 2
                        && method.GetParameters().Length == 2)
                .MakeGenericMethod(typeof(T), type)
                .Invoke(null, new object[] { collection, nullOrder });
        // We've inserted the initial order by on nullOrder,
        // so OrderBy on the property becomes a "ThenBy"
        if (orderByInfo.Direction == SortDirection.Ascending)
            methodName = "ThenBy";
        else
            methodName = "ThenByDescending";
    }
    // The rest of the method remains the same
    return (IOrderedQueryable<T>)typeof(Queryable).GetMethods().Single(
                method => method.Name == methodName
                        && method.IsGenericMethodDefinition
                        && method.GetGenericArguments().Length == 2
                        && method.GetParameters().Length == 2)
                .MakeGenericMethod(typeof(T), type)
                .Invoke(null, new object[] { collection, lambda });
}

The caller would need to pass a null checker explicitly. Passing null for non-nullable fields should work. You can construct them once, and pass as needed:

static readonly Expression<Func<string,int>> NullStringOrder = s => s == null ? 1 : 0;
static readonly Expression<Func<int?,int>> NullIntOrder = i => !i.HasValue ? 1 : 0;
static readonly Expression<Func<long?,int>> NullLongOrder = i => !i.HasValue ? 1 : 0;

Answer 4

My approach is to create a generic class that implements IComparer<TClass>. This way you can use your class in all LINQ statements with a non-default comparer. The advantage is that you will have full type checking at compile time. You can't name properties that can't be compared or that can't be null

class NullValueLastComparer<TClass, TKey> : IComparer<TClass>
    where TClass : class
    where TKey : IComparable<TKey>
{

This generic class has two Type parameters: the class that you want to compare, and the type of the property you want to compare with. The where clauses assert that TClass is a reference type, so you can access Properties, and TKey is something that implements normal comparison.

To create objects for the class we have two Factory functions. Both functions need a KeySelector, similar to lots of Key Selectors you can find in LINQ. The KeySelector function is the function that will tell you which property must be used in your comparisons. It is similar to the KeySelector in function Enumerable.Where.

The second Create function gives you the possibility to provide a non-default comparer, again similar to a lot of functions in the Enumerable class:

    public static IComparer<TClass> Create(Func<TClass, TKey> keySelector)
    {   // call the other Create function, with the default TKey comparer
        return Create(keySelector, Comparer<TKey>.Default);
    }

    public static IComparer<TClass> Create(Func<TClass, TKey> keySelector, IComparer<TKey> comparer)
    {   // construct a null value last comparer object
        // initialize with the key selector and the key comparer
        return new NullValueLastComparer<TClass, TKey>()
        {
            KeySelector = keySelector,
            KeyComparer = comparer,
        };
    }

I use a private constructor. Only the static create classes can construct the null value last comparer

    private NullValueLastComparer() { }

Two properties: the key selector and the comparer:

    private Func<TClass, TKey> KeySelector { get; set; }
    private IComparer<TKey> KeyComparer { get; set; }

The actual compare function. It will use the KeySelector to get the values that must be compared, and compares them such that a null value will be last.

    public int Compare(TClass x, TClass y)
    {   
        if (Object.ReferenceEquals(x, null))
            throw new ArgumentNullException(nameof(x));
        if (Object.ReferenceEquals(y, null)
            throw new ArgumentNullException(nameof(y));

        // get the values to compare
        TKey keyX = KeySelector(x);
        TKey keyY = KeySelector(y);
        return this.Compare(keyX, keyY);
    }

The private function that compares the Keys such that null values will be last

    private int Compare(TKey x, TKey y)
    {   // compare such that null values last, or if both not null, use IComparable
        if (Object.ReferenceEquals(x, null))
        {
            if (Object.ReferenceEquals(y, null))
            {   // both null
                return 0;
            }
            else
            {   // x null, y not null => x follows y
                return +1;
            }
        }
        else
        {   // x not null
            if (Object.ReferenceEquals(y, null))
            {   // x not null; y null: x precedes y
                return -1;
            }
            else
            {
                return this.KeyComparer.Compare(x, y);
            }
        }
    }
}

Usage:

class Person
{
    public string FirstName {get; set;}
    public string FamilyName {get; set;}
}

// create a comparer that will put Persons without firstName last:
IComparer<Person> myComparer =
    NullValueLastComparer<Person, string>.Create(person => person.FirstName);
Person person1 = ...;
Person person2 = ...;

int compareResult = myComparer.Compare(person1, person2);

This compare will compare Persons. When two Persons are compared, it will take person.FirstName for both persons, and will put the one without FirstName as last.

Usage in a complicated LINQ statement. Note that there is full type checking at compile time.

IEnumerable<Person> myPersonCollection = ...
var sortedPersons = myPersonCollection
    .OrderBy(person => person, myComparer)
    .ThenBy(person => person.LastName)
    .Select(person => ...)
    .ToDictonary(...)