NHibernate second level cache performance issue

I am developing a MVC application with NHibernate using the session per request pattern. Most of the time users just read data and therefore i am trying to make use of NHibernate's second level cache in the following manner: I've setup SysCache and have made all my persistant entities cacheable (cache usage="nonstrict-read-write") and on application start I make calls to load all commonly used entities from the database like so:

var all = Session.QueryOver<T>().Cacheable().List<T>()

For evaluation purposes I trace the execution time for the above call and for approximately 50000 results it's ~5 seconds the first time and ~2.5 seconds for any subsequent calls to the cached query.... NHibernate Profiler says the query to the database takes less than 100ms, what takes so much time then? I tried switching providers but got similar (if not worse) results with Velocity and Memcached... I read virtually all I could find regarding NHibernate and its use of the second level cache and I think, though I may not be entirely correct, that in the above statement what happens is: 50000 objects are constructed and their data is stored in the entity and query caches and the session timestamp saved in the timestamp cache. How could that take 5 seconds on an i5 machine? Even if that is normal how just reading the cached data take 2.5 seconds in the subsequent calls with no changes inbetween? Since I am relatively new to NHibernate, can any of you help me figure out what I am doing wrong? Any help would be greatly appreciated...I've been banging my head against the wall for a week now...

You shouldn't put 50000 entities in your cache, that negates the point of having a database, especially if your data comes from a SELECT * FROM TABLE. If it comes from a REALLY expensive query, then caching cost is less than query cost, and it should be put in the cache.

Use queries to get specific data. Then, look for the pages that make most queries, and optimize them, using a cache if necessary.