I'm just now diving into responsive CSS and design, and I'm wondering how I'm supposed to figure out various device widths out. I don't want to spend all day testing every single mobile device possible, I just want to get the responsive layout enough to where it works.

I saw some sites using @media only screen and (max-device-width: xx) but it seems that limits it to very specific resolutions, not actual browser window sizes.

Any advice at all would be great. I'm not new to web development or CSS, but totally new to responsive design.

It would seem that you're trying to figure out where and when to apply your breakpoints.

Rather than testing resolution on a bunch of devices why not build a responsive design that is acceptable at every resolution? There are no magic breakpoints that fix responsive websites for every device because every responsive website is going to differ depending on layout, content, etc. and there are tons of different devices with varying resolutions.

You might be thinking, "what the hell, I am not going to go pixel by pixel and check my website," but that's not really what I mean.

Finding your website's breakpoints:

Jump into a browser, navigate to your website, and open the console
Resize the viewport to a very low resolution. 320px is a good starting point.
(Note: To get the size of the viewport type window.innerWidth in console. See Resources for more ways to enhance debugging your design)
Analyze your layout. How does it look at this resolution? If you need to change the layout at this resolution then its time to add a breakpoint!
Slowly stretch the browser window until something breaks or looks horrible. At this point you'll need to insert another breakpoint.
Repeat step 4 to your heart's content

Keep in mind:

The point of responsive design isn't to make your site look good on all devices, its to make your content look good anywhere - Sam Richards

Resources:

Responsive Web Design - Programming with Anthony
Responsive Typography
Logical Breakpoints For Your Responsive Design
As mentioned in the above video, Modernizr is an awesome JS library that helps in detecting device-specific features
Responsive Design View Feature in Firefox 15+
Responsive Design View Tutorial for Chrome