I have the exact same problem as well as 3 of my friends. WIth the sound on, we have problems with the lag and the sound is very distorted. When I'm near my watermill and can hear it there is no lag. As soon as I move away from it the lag is terrible. Muting the sound within the game resolves it.
My PC:
i5 2500k @ 5ghz
8gb Ram
GTX 570