[WIP|1.7.10] FastCraft 1.25 (Note: All Posts of new Members need to be approved first, so if you press the submit button but no new post appears here, it is in the folder of posts to be approved first, and Player will receive them a few hours later)

    Quote from Player

    I also added a set of kcauldron fixes for ajthemacboy ;)


    It works! Thanks a bunch :) I did experience one crash with Dimensional Doors remake for 1.7.10 that I've never had before, but I hope it was a one time thing. I'll report if it happens again.

    Hey Player, thanks so much for creating and actively developing this mod! It's helped SO much with a private server I run.


    I'm curious as to whether you'll look into threading some things? I understand Nallar had a go at it with TickThreading, but as we both know, trying to thread that many things is beyond hard. Is there any chance you may try running chunk generation or something else in it's own thread? Even something small would be awesome. I'd love to help but my knowledge of Java is low to nil, currently all I can offer is a willingness to test (with logs) and maybe some insight.


    Please have a think about it, at first it might seem crazy to even try it, but you might think of some easy way to implement it. Even if it only helps a tiny bit, every bit counts!

    • Official Post

    ajthemacboy: This is a craftbukkit/spigot incompatibility, which I fixed for the next one.


    stardustrider: Yours is only slightly related to ajthemacboy's since you obviously don't use bukkit, which was conflicting with Fastcraft in a very specific way. The underlying cause in your case is different and I don't see what it could be without extra data. Can you reproduce/repeat the crash sufficiently reliably? If yes I'd be very interested in a copy of your installation (mods, configs, saves all via pm ofc) and instructions on how to make it crash.


    Feiya: The world generation bits are unfortunately not easier to offload to threads than entity/tile entity ticks, there are many hooks and detours into arbitrary mod code I can't control to ensure thread safety. The data flow is way too rigid for efficient partial offloading and with my "conventional" optimizations already in place it's very hard to gain much extra. I've seen several attempts by others to do threading based optimizations, but they were all too naive and broken in various ways. It's easy to make something fast by moving it to a thread, but ensuring proper thread safety while maintaining API compatibility is somewhere between very hard and impossible, depending on the specific task. I keep exploring opportunities for using threads, but so far other methods were more applicable.
    If you want to help just give the preview builds I keep posting a try (with backups ofc) and/or send me Sampler reports created during high lag phases by using "/sampler report <filename>" and I'll see what can be done. The worst cases usually result in bug reports to mods though, not Fastcraft changes ;)

    Thanks for responding :) It's a shame that's how it is, I do hope either you or someone else finds a good way to offload some things to threads, it sucks having a powerful server with only one core at max use ;-;

    Quote from Player

    The world generation bits are unfortunately not easier to offload to threads than entity/tile entity ticks, there are many hooks and detours into arbitrary mod code I can't control to ensure thread safety.

    Excuse the ignorance, but why is this the case? Is there not some way to get in either before or after mods do their injection/modifications? Is it really that hard to run say... mob path-finding on another thread? If all else fails with that, why not try and do asynchronous things? Chunk generation absolutely is the number one killer for a lot of these big packs, being able to control chunk gen speed or even have it limited by the server would be amazing.


    I'm really not trying to sound rude here, I'm legitimately curious as to why this situation is like it is, I've never had a good discussion about it.

    Some thoughts about the threading:


    1. OptiFine has had some success, from what I understand, spreading chunk [something] across multiple ticks, and multiple cores. However, I'm not sure if this is chunk generation, or chunk rendering. If it's the former, then that might be something to look at.


    2. Even with all the complicated mod injections into the chunk generation code, I still believe it should be possible to assign world generation to a different thread, instead of splitting it into multiple. This could also be done with other things like mob path finding, machines from mods, etc.


    3. Finally, this is probably the most complicated, but there are commercial and server programs that can take a task that is normally single-threaded and divide it into multiple workloads, then assign each to a core/thread. Instead of intercepting the original chunk generation code, it might be possible to take the math and calculations and assign them to multiple cores.


    Please, correct me, I'm sure I'm mostly babbling and wrong, and I don't want to remain the idiot ?( These are some ideas though.


    Edit: Thought I'd share some results with the last FastCraft build on KCauldron: The world took 36 seconds to generate before FastCraft, and 24 after, in a low ram environment on a modpack with ~150 mods. While KCauldron may improve chunk generation, FastCraft is certainly NOT useless with KCauldron!

    • Official Post
    Quote from ajthemacboy

    Thought I'd share some results with the last FastCraft build on KCauldron: The world took 36 seconds to generate before FastCraft, and 24 after, in a low ram environment on a modpack with ~150 mods. While KCauldron may improve chunk generation, FastCraft is certainly NOT useless with KCauldron!


    As expected really. The KCauldron people only blocked Fastcraft from loading before because "there was nothing it could help with on a server that KCauldron doesn't already do", which seemed an overly bold statement.

    145 Mods isn't too many. 9 types of copper and 8 types of tin aren't too many. 3 types of coffee though?

    I know that you believe that you understood what you think I said, but I am not sure you realise that what you read was not what I meant.


    ---- Minecraft Crash Report ----
    // I just don't know what went wrong :(


    I see this too much.

    • Official Post

    I did a quick measurement on where the time is being spent in world loading with mostly worldgen, using the BOP world type and the FC2 mods:
    - 42% biome decoration - almost exclusively in bop's decorator code ofc
    - 31% mod generation (ores, beehives etc.)
    - 11% deflate and nbt construction
    - 1% disk i/o
    - 6% chunk load event + hooks
    - 6% entity and tile entity handling
    - 3% misc


    All numbers reflect only CPU time contribution on the server thread.


    That's at least 79% in mod code, deflate+i/o are already threaded if chunks aren't being requested as quickly (I teleported around in a narrow grid). The entity handling requires strong consistency.


    Overall that doesn't leave much to optimize with everything else being insignificant and riddled with hooks/events. The major lag spikes from world generation are actually being caused by broken decorators that place large features, although the code only allows to touch at most 4 chunks (x/z, x-1/z, x/z-1 and x-1/z-1). Accessing others quickly leads to chained generation - thus lag spikes - and sporadic "already decorating" crashes. Fixing world gen related lag shouldn't involve more than getting mods to respect this limitation and being smarter about large feature generation.


    Regarding ajthemacboy's points:


    @1. This is chunk rendering. Optifine stops the client thread, renders chunk sections concurrently in multiple threads and then results the client thread again. It's quite broken, LWJGL 2 doesn't properly support multi threaded OpenGL accesses and the mod render handlers/hooks involved in the process aren't necessarily thread safe.


    @2. There's no guarantee that the code executed by thread won't share state with another thread in a non-safe way, even if no other thread does anything strongly related/similar. Just like Optifine's multithreaded rendering it may mostly work, but break randomly in obscure ways that are difficult to track and hardly fixable. I believe this is only marginally easier than TickThreading's approach. There's a few cases where some gains may be achievable with a lot of trickery, that's been on my TODO-list for quite a while due to high risk/questionable benefit/lack of time though.


    @3. For the most part this refers to forking off an isolated task, processing it concurrently and then collecting the results to resume normal operation. The obvious limitation is that the tasks have to be well parallelizable and sufficiently large, otherwise the associated substantial management overhead will dominate. Minecraft doesn't have much in this area as large working sets are not desirable in a realtime application.


    Decent multithreaded processing requires cooperation from mods, only Mojang can require this effectively by simply distributing everything over multiple threads. Apparently they tried doing so with a thread per world during 1.8 development, but presumably their code didn't like it with all the global state involved, so it has been scrapped. Doing it with mods that aren't aware of it or on a more granular level, i.e. a thread per world and task like worldgen, entities, tile entities, block ticks etc., is a lot more problematic.



    Noire: This issue is already fixed for the next build.

    That's actually really interesting, thanks Player.


    Would I be right to assume that async chunk gen/loading would be easier? Or would it be just as hard if not harder?


    Also have you looked into mob-spawning algorithms? I have yet to find a mod that efficiently handles spawning without it getting to 2000+ entities randomly.


    Are you able to check CPU time for entities? I feel like it's the entities that cause the most TPS lag from my server. Even with only ~150, they sure love eating my servers resources. (And I don't exactly want to just set spawning to off).

    Quote from Feiya

    Are you able to check CPU time for entities? I feel like it's the entities that cause the most TPS lag from my server. Even with only ~150, they sure love eating my servers resources. (And I don't exactly want to just set spawning to off).


    On my server I feel like it's machines that use the most resources, for example we have the Ancient Warfare mod which has a treefarm that I've found to be super laggy. Quarries and farms use a lot too. I'm not sure there is any way to optimize machines or quarries, other than possibly changing the way light updates happen.


    Player, when can we expect that next beta build?