A good rule of thumb by is that I intelligently refactor any method over 50 lines.
The count does not include comments and white space but actual code. The reason I also say intelligently is there are plenty of times where a class over 50 lines is acceptable and cannot or should not be changed.
I do not have a rule of thumb for classes. Generally I don't check classes to see if they should be refactored.
On my currently project I have just about complete a class that is almost 4000 lines long. However no methods over 50, and most of the lines and methods are private and not acting on any data outside of the class.
What is the rule of thumb for refactoring classes?
My rule of thumb is for methods, more than classes - If I can't see the whole method on screen it needs to be refactored. Of course, the traditional smells apply.
I tend to look at the cyclomatic complexity of each member instead of number of lines of code for the entire class.
Don't let LOC be your primary metric. 50 lines seems really small to me. With 50 line files, you will end up having an unfriendly number of class files in the solution. Your productivity will be dampened by all the navigation you will be doing between files and your IDE will always be littered with too many tabs. I personally try to organize classes into logical groups by namespace first. On a class by class basis, I try to make the code smaller and easier to read. Sometimes, class files do get large. I start to get a sick feeling when the class file is 2000+ lines. Anything less than that, I deal with on a case by case basis.
Refactoring is not so much about reducing the number of LOC, though that is a side effect, but improving the quality. Particularly, you want to try to follow the DRY (Don't Repeat Yourself) principle as much as you can, reasonably.
If your classes have broken one of following "rules", you should consider to refactor.
You are looking for SOLID, more detailed screencasts can be found here.
SRP: single responsibility principle, there should never be more than one reason for a class to change.
OCP: open closed principle, software entities (classes, modules, functions, etc.) should be open for extension but closed for modification.
LSP: liskov substitution principle, functions that use references to base classes must be able to use objects of derived classes without knowing it.
ISP: interface segregation principle, clients should not be forced to depend upon interfaces that they do not use.
DIP: dependency inversion principle:
high level modules should not depend upon low level modules. Both Should depend upon abstractions.
abstractions should not depend upon details. Details should depend upon abstractions.
Refactor whenever you have the opportunity to:
etc, etc.
EDIT: Obviously, the decision to refactor should take into account time, potential pay-off, other responsibilities, etc.