I'm trying to protect my videos the most possible way, so I know putting them up on a red5 media server will make them streaming so the flv file won't be downloaded to users cache but the main concern I have is that possible to protect the video from being embeded? so it will run only from domains that I specify?
and also, is it possible to somehow encrypt a streaming video file? and encrypt it down in my flash player when it's downloading?
To prevent the video from being embedded, don't use an embeddable player (i.e. if you are using a commercial player, disable the embedding option). If you don't want them to find the stream then embed the video in their own player, you have to either hide the URL of the movie. There are a number of ways to do this. Most popular is by using a php file to output the video via readfile. you could also use symbolic links or some similar solution.
re: encryption, there are some DRM solutions like iTunes uses, but I assume that if you are asking here that may be overkill for you.
To secure your streams you can edit the file: red5-web.properties (in $red5/webapps/$your-app/WEB-INF/)
webapp.virtualHosts defines the IP/domains that are allowed to connect to your Red5 application.
So if your application is hosted on myserver.com, and you want only SWFs that are loaded from myserver.com to be able to connect to your streams then you can configure red5-web.properties.
AFAIK: The "security" here is that every SWF sends automatically to the RTMP server a IP/Domain where it has been loaded from. So with some motivation you could build a RTMP client in Java or C++ that simulates connecting from another domain.
However, for simple scenarios, for example for protecting your stream from being embeded with some FLV player in another website, this should already help.
Sebastian