I'm by no means a cryptography expert, but a few difficult bits from what I've gathered, observing from a different area of CS that also cares about hardness (AI):
1. At the base, we need a one-way function [1], something that's easy to compute in one direction but hard to reverse. For example, it is believed to be much easier to multiply two large primes than to factor the result back into the original two primes. However so far nobody has come up with a way of proving that a function is one-way for any plausible definition. This is often, in the popular press, thought to be the P=NP question, but that would only prove the negative: if P=NP, then there are no one-way functions (for a certain definition). But even if P!=NP were proven, it is still not necessarily the case that there are one-way functions. In particular, NP-complete problems are not automatically one-way functions, because that is only a worst-case hardness notion, and cryptosystems that are only unbreakable in the worst case for the attacker are not so useful [2] (and indeed none of the major cryptosystems currently used rely on NP-complete problems).
In short, the seemingly hard problems that have had practical cryptosystems built out of them, such as integer factorization (closed related to the RSA problem) and the discrete logarithm problem on elliptic curves (ECC) only seem hard, but are not proven hard in any rigorous sense. This is an interesting outstanding question mathematically, because it would be nice if we understood either why these are hard and be able to prove them hard for some definition, or else to discover why they in fact aren't.
2. At the practical level, it's hard to produce a security proof that takes into account an attack completely outside the conception of the original framework. For example, early security proofs were completely oblivious to the problem of side-channel attacks, since they were proving the security of certain mathematical procedures, in a framework that did not countenance things like "your mathematical procedure might be run on an EC2 instance where your attacker can also get a VM and collect some (noisy) data about what it's doing". This is arguably a quite different question from whether the fundamental cryptosystem's approach is flawed, but this kind of consideration is the source of many practical attacks.
1. At the base, we need a one-way function [1], something that's easy to compute in one direction but hard to reverse. For example, it is believed to be much easier to multiply two large primes than to factor the result back into the original two primes. However so far nobody has come up with a way of proving that a function is one-way for any plausible definition. This is often, in the popular press, thought to be the P=NP question, but that would only prove the negative: if P=NP, then there are no one-way functions (for a certain definition). But even if P!=NP were proven, it is still not necessarily the case that there are one-way functions. In particular, NP-complete problems are not automatically one-way functions, because that is only a worst-case hardness notion, and cryptosystems that are only unbreakable in the worst case for the attacker are not so useful [2] (and indeed none of the major cryptosystems currently used rely on NP-complete problems).
In short, the seemingly hard problems that have had practical cryptosystems built out of them, such as integer factorization (closed related to the RSA problem) and the discrete logarithm problem on elliptic curves (ECC) only seem hard, but are not proven hard in any rigorous sense. This is an interesting outstanding question mathematically, because it would be nice if we understood either why these are hard and be able to prove them hard for some definition, or else to discover why they in fact aren't.
2. At the practical level, it's hard to produce a security proof that takes into account an attack completely outside the conception of the original framework. For example, early security proofs were completely oblivious to the problem of side-channel attacks, since they were proving the security of certain mathematical procedures, in a framework that did not countenance things like "your mathematical procedure might be run on an EC2 instance where your attacker can also get a VM and collect some (noisy) data about what it's doing". This is arguably a quite different question from whether the fundamental cryptosystem's approach is flawed, but this kind of consideration is the source of many practical attacks.
[1] http://en.wikipedia.org/wiki/One-way_function
[2] http://www.kmjn.org/notes/nphard_not_always_hard.html