in the configs, its not possible. but as an administrator of the server, you should be using world border to limit your players from going to unloaded chunks.
Turning off gravi speed only partially fixes the problem you are having. The player will still load your world, with its flight and will still lag the server by scouting a new part of the world.
Regardless of what CPU you have, your server will be slow when its loading. MC only uses one core out of the 6 you have.