How does Google App Engine sandbox work?
What would I have to do to create my own such sandbox (to safely allow my clients to run their apps on my engine without giving them the ability to format my disk drive)? Is it just class loader magic, byte manipulation or something?
You would probably need a combination of a restrictive classloader and a thorough understanding of the Java Security Architecture. You would probably run your JVM with a very strict SecurityManager specified.
In the Java case, I think it's mostly done by restricting the available libraries. Since Java doesn't have pointer concept, and you can't upload natively compiled code (only JVM bytecode), you can't break out of the sandbox. Add some tight process scheduling, and you're done!
I guess The hardest part is to pick the libraries, to make it useful while staying safe.
In the Python case, they had to modify the VM itself, because it wasn't designed with safety in mind. Fortunately, they have Guido himself to do it.
This can be easily achieved using the Java Security Manager. Refer this answer for an example.