I'm talking about an action game with no upper score limit and no way to verify the score on the server by replaying moves etc.
What I really need is the strongest encryption possible in Flash/PHP, and a way to prevent people calling the PHP page other than through my Flash file. I have tried some simple methods in the past of making multiple calls for a single score and completing a checksum / fibonacci sequence etc, and also obfuscating the SWF with Amayeta SWF Encrypt, but they were all hacked eventually.
Thanks to StackOverflow responses I have now found some more info from Adobe - http://www.adobe.com/devnet/flashplayer/articles/secure_swf_apps_12.html and https://github.com/mikechambers/as3corelib - which I think I can use for the encryption. Not sure this will get me around CheatEngine though.
I need to know the best solutions for both AS2 and AS3, if they are different.
The main problems seem to be things like TamperData and LiveHTTP headers, but I understand there are more advanced hacking tools as well - like CheatEngine (thanks Mark Webster)
In my experience, this is best approached as a social engineering problem rather than a programming problem. Rather than focusing on making it impossible to cheat, focus on making it boring by removing the incentives to cheat. For example, if the main incentive is publicly visible high scores, simply putting a delay on when high scores are shown can significantly reduce cheating by removing the positive feedback loop for cheaters.
You may be asking the wrong question. You seem focused on the methods people are using to game their way up the high score list, but blocking specific methods only goes so far. I have no experience with TamperData, so I can't speak to that.
The question you should be asking is: "How can I verify that submitted scores are valid and authentic?" The specific way to do that is game-dependent. For very simple puzzle games, you might send over the score along with the specific starting state and the sequence of moves that resulted in the end state, and then re-run the game on the server side using the same moves. Confirm that the stated score is the same as the computed score and only accept the score if they match.
Whenever your highscore system is based on the fact that the Flash application sends unencrpyted/unsigned highscore data via the network, that can be intercepted and manipulated/replayed. The answer follows from that: encrypt (decently!) or cryptographically sign highscore data. This, at least, makes it harder for people to crack your highscore system because they'll need to extract the secret key from your SWF file. Many people will probably give up right there. On the other hand, all it takes is a singly person to extract the key and post it somewhere.
Real solutions involve more communication between the Flash application and the highscore database so that the latter can verify that a given score is somewhat realistic. This is probably complicated depending on what kind of game you've got.
I usually include "ghost data" of the game session with the highscore entry. So if I'm making a racing game, I include the replay data. You often have the replay data already for replay functionality or ghost racing functionality (playing against your last race, or playing against the ghost of dude #14 on the leaderboard).
Checking these is very manual labour, but if the goal is to verify if the top 10 entries in a contest are legit, this can be a useful addition to the arsenal of security measures others have pointed out already.
If the goal is to keep highscores list online untill the end of time without anybody having to look at them, this won't bring you much.
You cannot trust any data the client returns. Validation needs to be performed on the server side. I'm not a game developer, but I do make business software. In both instances money can be involved and people will break client side obfuscation techniques.
Maybe send data back to server periodically and do some validation. Don't focus on client code, even if that is where your applicaiton lives.
You are talking about what is called the "client trust" problem. Because the client (in this cash, a SWF running in a browser) is doing something it's designed to do. Save a high score.
The problem is that you want to make sure that "save score" requests are coming from your flash movie, and not some arbitrary HTTP request. A possible solution for this is to encode a token generated by the server into the SWF at the time of request (using flasm) that must accompany the request to save a high score. Once the server saves that score, the token is expired and can no longer be used for requests.
The downside of this is that a user will only be able to submit one high score per load of the flash movie - you've have to force them to refresh/reload the SWF before they can play again for a new score.