I wrote an application that encrypts text in this way:
Get the input text
Reverse the text
Convert to hexadecimal
XOR with a key
Base64 encode
Now, I didn't do a lot of encryption/encoding myself, so my question might sound stupid, but, say I get a file which has a content from the above algorithm and I didn't know about this algorithm. How would one start "breaking" the text, are there any guidelines, principals, rules to follow?
My question is not tied to those 5 steps, that was a pure example.
As a different example, take the text: A751CD9E1F99. How would I start investigating what this might mean?
In order to break a cipher, cryptanalysts use all the information they can gather. Attacks fall into a number of categories, depending on what is known. Some of the main attacks, from hardest to easiest, are
Nowadays, likely ways to break a code are through flaws in the system. For example, poor key management might be used, allowing the key to be stolen or guessed. In other cases, a "side-channel" attack might be used. For example, by carefully measuring the time it takes for certain cryptographic operations, an attack might be able to guess that certain bits or bytes of a key are zero, causing a fast path through some algorithm.
Up near the "tinfoil hat" end of the spectrum are methods to intercept radio emissions from computing equipment. This allows a remote agent to "see" what is displayed on a monitor. There are even specially designed fonts to try and disrupt this sort of eavesdropping.
That is kind of impossible, you'd fail at the XOR decryption if you don't have any knowledge about what key was used.
In a general case, it is even more impossible (if that is possible :)) to gauge what an encrypted string might mean.
If you have access to a black box which does the encryption, you can get a lot of information by feeding it particular input values.
As a simple example, if the black box does "one time pad" style encryption, if you feed it all zeroes you get the one time pad. (In fact, feeding it any input value will get you the one time pad with an additional xor.)
Note that good cryptosystems are resistant to such attacks, even if the cryptosystem is already known (but the key is not).
I think you should start by reading The Code Book. What you are asking is how to crack encryption methods and that will give you a start as to how they work.
You would need a larger text base than that and some understanding that the crypt is coming from a particular language/domain. Then based on the frequency of words in that language/domain, one could potentially decipher certain attributes form the text.
Of course, good ciphers work around this. Only poorly implemented ciphers can be broken easily with this method.