It has to end... No matter how painful it is, that time is now. Breaking up is never easy. Deciding to end a relationship, no matter how imperfect it is, will always be emotionally and mentally traumatic. In spite of the imperfections, you did share wonderful and intimate times together.
Like what they say, memories are easy to make but are hard to forget and remembering makes it harder to move on. That person you are saying goodbye to has been a part of you. Detaching yourself from him can be very painful and unimaginable. But like everything else in life, think that this too, shall pass and no matter how hard it is to move on, in the end, things will fall into its place.