I'd like to use the UIView "sucking" transition (view is "sucked" into a point) referenced here:
http://iphonedevwiki.net/index.php/UIViewAnimationState#Using_suckEffect
The method above, however, uses private APIs, which makes it next to useless for an App Store app. How can I recreate a similar effect without relying on private APIs? I'm guessing the answer has to do with OpenGL, although I have zero experience there.
have you checkked out the cocos2d framework. it's an objective c wrapper for openGL and it's done really well. you can check out their demo video which shows all the transition types that the library supports.
Please have a look at my project at git://github.com/rmd6502/Genie.git - it's very slow right now, but improvements are forthcoming.
As long as Apple doesn't publicly document the effect, and you want to avoid the risk of being rejected for using private API's you'll have to roll your own.
A suggestion to start working on this would be using OpenGL ES shaders.
The following blog post includes a demo Quartz Composer patch to create the Genie effect as seen in OS X and iOS: http://www.aderstedtsoftware.com/users/erik/weblog/c7cb9/
A good start for your OpenGL ES shader would be to use the code in the Genie Filter in this QuartzComposer file. The code can almost be copied to a shader for OpenGL as is. The most work would probably be integrating OpenGL in your application: rendering the right view to a OpenGL texture and overlaying OpenGL over your application to perform the animation based on your shader.