A recent article on Ars Technica discusses a recent study performed by the Psychology Department of North Carolina State University, that showed users have a tendency to do whatever it takes to get rid of a dialog box to get back to their task at hand. Most of them would click OK or yes, minimize the dialog, or close the dialog, regardless of the message being displayed. Some of the dialog boxes displayed were real, and some of them were fake (like those popups displayed by webpages posing as an antivirus warning). The response times would indicate that those users aren't really reading those dialog boxes.
So, knowing this, how would this effect your design, and what would you try to do about it (if anything)?
Lots of good advice above. I just want to add to the book recommendations - Joel Splosky's "User Interface Design for Programmers" book is worth reading:
http://www.amazon.com/User-Interface-Design-Programmers-Spolsky/dp/1893115941/ref=pd_bbs_sr_4?ie=UTF8&s=books&qid=1222233643&sr=8-4
One thing you can do is have the OK button disabled for 3 seconds.
Firefox does this when you install an extension.
Edit: All right, some people find this annoying. I still think that about 1 second would be all right. It would suppress the instant-OK-clicking instinct that people (myself included) have, and force a double-take. Of course, even this will annoy people if your dialog is not something that they actually need to read.
A .NET Rocks episode comes to mind (I believe episode 338, "Mark Miller on the Science of Good UI") discusses this very topic. I think what is key to this whole discussion is that this is basic UI design taken too far. Where the modal was once an acceptable means of communication we now find that it has become programming faux pas. Users understand that 6 times out of 10 the information is not pertinent enough for them to worry about. As a result they treat all modals the same way -- learned helplessness. If a modal comes up and tells me that Application Error X occurred and all I can click is "OK" --even when I don't think it is "OK" I learn a particular behavior. I assosciate modals with the idea that I probably can't do much about them, but if I click OK/Yes then I can get back to what I need.
So, why is it still used? Perhaps developers have tried avoiding the fact that application development is becoming more than just a basic interface, and users require a fluid UI design -- old standbys are hard to give up...
I think the key here is to understand that good UI design now indicates that interruptions (to even the most novice computer user) are annoyances and we need to strive to have a seamless user experience where the focus of the application is the user -- not the needs of the application via prompting and error reporting -- don't allow a user to get into situations where they don't care.
I think you probably want to read this paper: "Impact of High-Intensity Negotiated-Style Interruptions on End-User Debugging", T. J. Robertson, Joseph Lawrance, and Margaret Burnett, Journal of Visual Languages and Computing 17(2), 187-202, April 2006.
It asked a similar question. The result was, let the user know you want their attention and then sit back and wait for the user to respond. Don't interrupt the user though, that's not what he or she wants.
If you must use a dialog, put descriptive captions on the buttons within the dialog.
For example, instead of OK and Cancel buttons, have them say "Send Invoice" and "Go Back", or whatever is appropriate in the context of your dialog.
That way, the text is right under their cursor and they have a good chance of understanding.
Mac OS X does this most of the time. Here's an example image.
Edit:
This is a better image, and I found it at the Apple Human Interface Guideline site, which is a great reference, and very readable. This document on that site is all about Dialogs.
One suggestion:
Sometimes that is hard... how do you handle opening a file? Sometimes it's easy... do you really need to warn the user that they are about to overwrite a file? Chances are, if I'm blindly clicking "OK", I'm not going heed any warnings whatsoever.