The fastest way to fix it is not using displacement map at all. This video shows you how.

More on displacement map: