How to prevent Wix from installing an older versio

I have an application that we are switching over to a WiX installer. So far almost everything seems to be going well. The one problem I'm having is that if an older version is downloaded and attempted to install, it does so.

And that's a bit of a problem. If there is a newer version installed I don't want it to install the older version. I thought the problem was with the "Upgrade" component but I must admit I've hit a wall. How can I change it so that the older versions see there is a newer version already installed and not install it?

My test product is now in version 2.4 (the newest version we're trying to push out). It upgrades correctly to 2.4.1 or 2.5 or 3.0. But if I make a 3.0 version, and then run the msi for 2.4 it still adds it.

My upgrade component:

<Upgrade Id="PUT-GUID-HERE">
  <UpgradeVersion Maximum="2.4" Property="PREVIOUSVERSIONSINSTALLED" />
  <UpgradeVersion Minimum="1.0" Property="NEWERPRODUCTFOUND" OnlyDetect="yes" IncludeMinimum="no" />
</Upgrade>

Follow Up:
After following Steins suggestion I got an error like this
"Duplicate symbol 'WixAction:InstallExecuteSequence/RemoveExistingProducts' found"
After looking around in the Product.wxs file under the <InstallExecuteSequence>I had to delete <RemoveExistingProducts Sequence="6550" /> because that was the duplicate it was referring to. After doing that the installer worked and old versions can no longer be installed on top of new versions.

Packages of Futures Past: You can not change older versions of your package to detect newer ones. You need to build protection into your packages from the start. Packages need to be pre-cognitive. It's an industry problem.

Modern Times: The WiX elements you show above are "old-style". There is a new "convenience feature" described here: How do you detect installed product versions at each startup? It involves the "new" MajorUpgrade element. This new MajorUpgrade element features some auto-magic and I believe it adds the protection you describe by default (downgrade protection). Hence you can switch to using it. I would try that first. Let me inline the basic markup:

<MajorUpgrade Schedule="afterInstallInitialize" 
              DowngradeErrorMessage="A later version of [ProductName] is already installed. Setup will now exit." 
              AllowDowngrades="no" AllowSameVersionUpgrades="no" />

In addition to removing the old-style upgrade elements, please also remember to delete any hard coded scheduling of the RemoveExistingProducts standard action. For example, remove this line (sequence number will likely be different, but same name):

<RemoveExistingProducts Sequence="6550" />

Decoupling: If you find that there is a high risk that people will run the older versions and mess with your newest application, you could set a new installation location and a new upgrade code for your latest version and install side-by-side to decouple your old and new products.

Side-By-Side: For this to work your product(s) must be capable of co-existing peacefully and not fight over file associations, per-machine registered COM servers, or other globally shared data that make the products interfere with each other. Whether this is possible or not depends on your application. A globally shared COM server can not be registered from two different locations - if you use normal registry registration (you can use manifest based reg-free COM though - though this is involved at times). There can be many challenges to overcome before your application supports side-by-side installation, or it could be rather trivial if your package is simple with no registry involvement.

Component GUIDs: You need to set new component GUIDs as well - for all components - in addition to the mentioned change of upgrade code in order to really shield the products from each other. If you use WiX auto-GUIDs this will happen auto-magically. The reason you need new component GUIDs is attempted explained here: Change my component GUID in wix? Essentially a GUID reference counts an absolute installation location, not a file per-se. You install to a new location, you need a new component GUID.