First of all, it is important to remember that "letting go" does not mean "getting rid of". The act of letting go doesn't suddenly give us unlimited control over what we don't like! But another reason letting go is so hard has to do with our beliefs about our identity. Here's a brief description of how these beliefs come about:
You have been conditioned since you were very young to relate everything to yourself. As soon as you learn to recognise people and things, you're taught how to relate these to the "I" and "mine"-- my mom, my dad, my toy, etc. As you grow up you're taught how to relate ideas and concepts to yourself. You have to learn that so that you can function properly in society. But at the same time, this process slowly and unconsciously creates a concept of selfhood, and you build up your ego. This build up is strengthened by the values of society. You learn to compete, to achieve, to accumulate knowledge, wealth and power. In other words, you are trained to possess and to cling.
By the time you are grown up, the concept of ego-self has become so real that it is difficult to tell what is illusion and what is reality. It is difficult to realise that "I" and "mine" are temporary, relative and changeable. The same is true of all that is related to "I" and "mine." Not understanding that "I" and "mine" are temporary, you struggle to keep them permanent; you cling to them. This desire to try to keep everything permanent is what makes it so difficult to learn to let go.