You are not logged in.

IC2 Experimental builds (jenkins):
v2.0/2.1/2.2 / 2.3 / 2.5 / 2.6 (For Minecraft 1.6.4/1.7.2/1.7.10 / 1.8.9 / 1.9.4 / 1.10)
IndustrialCraft² recent version: v1.117! (For Minecraft 1.5.2 → topic)

Dear visitor, welcome to IC² Forum. If this is your first visit here, please read the Help. It explains in detail how this page works. To use all features of this page, you should consider registering. Please use the registration form, to register here or read more information about the registration process. If you are already registered, please login here.

ajthemacboy

Tree Cutter

Posts: 24

Location: Tennessee, US

  • Send private message

1,021

Wednesday, September 30th 2015, 9:32pm

I also added a set of kcauldron fixes for ajthemacboy ;)


It works! Thanks a bunch :) I did experience one crash with Dimensional Doors remake for 1.7.10 that I've never had before, but I hope it was a one time thing. I'll report if it happens again.

ajthemacboy

Tree Cutter

Posts: 24

Location: Tennessee, US

  • Send private message

1,022

Wednesday, September 30th 2015, 10:21pm

Whoops, spoke too soon, just got this crash on a server with a preexisting world: http://pastebin.com/Wn6YA8vj

1,023

Wednesday, September 30th 2015, 11:31pm

Hmm I need more information, http://files.player.to/tmp/fastcraft-1.22-ltest3.jar should be a little more verbose about the issue.

ajthemacboy

Tree Cutter

Posts: 24

Location: Tennessee, US

  • Send private message

1,024

Thursday, October 1st 2015, 12:15am


1,025

Thursday, October 1st 2015, 12:18am

It made 1.7.10 slower and corrupted my world! So many chunks...... gone! ;( I should have made a backup like it said. But why so laggy? Isn't supposed to increase fps?

This post has been edited 1 times, last edit by "ZephaniahNoah" (Oct 1st 2015, 2:21am)


1,026

Thursday, October 1st 2015, 3:02am

That's the first time someone reported this, I need more information than that. Corruption usually involves a crash or at least log entries.

1,027

Thursday, October 1st 2015, 9:28pm

Similar crash to @ajthemacboy:

http://pastebin.com/TmaTc5VL

1,028

Friday, October 2nd 2015, 1:45am

Hey @Player, thanks so much for creating and actively developing this mod! It's helped SO much with a private server I run.

I'm curious as to whether you'll look into threading some things? I understand Nallar had a go at it with TickThreading, but as we both know, trying to thread that many things is beyond hard. Is there any chance you may try running chunk generation or something else in it's own thread? Even something small would be awesome. I'd love to help but my knowledge of Java is low to nil, currently all I can offer is a willingness to test (with logs) and maybe some insight.

Please have a think about it, at first it might seem crazy to even try it, but you might think of some easy way to implement it. Even if it only helps a tiny bit, every bit counts!

1,029

Friday, October 2nd 2015, 7:44am

ajthemacboy: This is a craftbukkit/spigot incompatibility, which I fixed for the next one.

stardustrider: Yours is only slightly related to ajthemacboy's since you obviously don't use bukkit, which was conflicting with Fastcraft in a very specific way. The underlying cause in your case is different and I don't see what it could be without extra data. Can you reproduce/repeat the crash sufficiently reliably? If yes I'd be very interested in a copy of your installation (mods, configs, saves all via pm ofc) and instructions on how to make it crash.

Feiya: The world generation bits are unfortunately not easier to offload to threads than entity/tile entity ticks, there are many hooks and detours into arbitrary mod code I can't control to ensure thread safety. The data flow is way too rigid for efficient partial offloading and with my "conventional" optimizations already in place it's very hard to gain much extra. I've seen several attempts by others to do threading based optimizations, but they were all too naive and broken in various ways. It's easy to make something fast by moving it to a thread, but ensuring proper thread safety while maintaining API compatibility is somewhere between very hard and impossible, depending on the specific task. I keep exploring opportunities for using threads, but so far other methods were more applicable.
If you want to help just give the preview builds I keep posting a try (with backups ofc) and/or send me Sampler reports created during high lag phases by using "/sampler report <filename>" and I'll see what can be done. The worst cases usually result in bug reports to mods though, not Fastcraft changes ;)

1,030

Friday, October 2nd 2015, 12:34pm

Thanks for responding :) It's a shame that's how it is, I do hope either you or someone else finds a good way to offload some things to threads, it sucks having a powerful server with only one core at max use ;-;
The world generation bits are unfortunately not easier to offload to threads than entity/tile entity ticks, there are many hooks and detours into arbitrary mod code I can't control to ensure thread safety.
Excuse the ignorance, but why is this the case? Is there not some way to get in either before or after mods do their injection/modifications? Is it really that hard to run say... mob path-finding on another thread? If all else fails with that, why not try and do asynchronous things? Chunk generation absolutely is the number one killer for a lot of these big packs, being able to control chunk gen speed or even have it limited by the server would be amazing.

I'm really not trying to sound rude here, I'm legitimately curious as to why this situation is like it is, I've never had a good discussion about it.

ajthemacboy

Tree Cutter

Posts: 24

Location: Tennessee, US

  • Send private message

1,031

Friday, October 2nd 2015, 2:20pm

Some thoughts about the threading:

1. OptiFine has had some success, from what I understand, spreading chunk [something] across multiple ticks, and multiple cores. However, I'm not sure if this is chunk generation, or chunk rendering. If it's the former, then that might be something to look at.

2. Even with all the complicated mod injections into the chunk generation code, I still believe it should be possible to assign world generation to a different thread, instead of splitting it into multiple. This could also be done with other things like mob path finding, machines from mods, etc.

3. Finally, this is probably the most complicated, but there are commercial and server programs that can take a task that is normally single-threaded and divide it into multiple workloads, then assign each to a core/thread. Instead of intercepting the original chunk generation code, it might be possible to take the math and calculations and assign them to multiple cores.

Please, correct me, I'm sure I'm mostly babbling and wrong, and I don't want to remain the idiot ?( These are some ideas though.

Edit: Thought I'd share some results with the last FastCraft build on KCauldron: The world took 36 seconds to generate before FastCraft, and 24 after, in a low ram environment on a modpack with ~150 mods. While KCauldron may improve chunk generation, FastCraft is certainly NOT useless with KCauldron!

This post has been edited 1 times, last edit by "ajthemacboy" (Oct 2nd 2015, 3:14pm)


Chocohead

Drowning in mods.

  • "Chocohead" is male
  • Korea, Democratic People's Republic of United Kingdom

Posts: 5,980

Location: Badly generated chunks.

  • Send private message

1,032

Friday, October 2nd 2015, 3:49pm

Thought I'd share some results with the last FastCraft build on KCauldron: The world took 36 seconds to generate before FastCraft, and 24 after, in a low ram environment on a modpack with ~150 mods. While KCauldron may improve chunk generation, FastCraft is certainly NOT useless with KCauldron!

As expected really. The KCauldron people only blocked Fastcraft from loading before because "there was nothing it could help with on a server that KCauldron doesn't already do", which seemed an overly bold statement.
145 Mods isn't too many. 9 types of copper and 8 types of tin aren't too many. 3 types of coffee though?

I know that you believe that you understood what you think I said, but I am not sure you realise that what you read was not what I meant.


---- Minecraft Crash Report ----
// I just don't know what went wrong :(

I see this too much.

1,033

Friday, October 2nd 2015, 8:56pm

Hi Player! tested 1.22 itest-3 and my server crashed with this error:

http://pastebin.com/RWjTp3Th


Please, fix it ^^

1,034

Saturday, October 3rd 2015, 12:26am

I did a quick measurement on where the time is being spent in world loading with mostly worldgen, using the BOP world type and the FC2 mods:
- 42% biome decoration - almost exclusively in bop's decorator code ofc
- 31% mod generation (ores, beehives etc.)
- 11% deflate and nbt construction
- 1% disk i/o
- 6% chunk load event + hooks
- 6% entity and tile entity handling
- 3% misc

All numbers reflect only CPU time contribution on the server thread.

That's at least 79% in mod code, deflate+i/o are already threaded if chunks aren't being requested as quickly (I teleported around in a narrow grid). The entity handling requires strong consistency.

Overall that doesn't leave much to optimize with everything else being insignificant and riddled with hooks/events. The major lag spikes from world generation are actually being caused by broken decorators that place large features, although the code only allows to touch at most 4 chunks (x/z, x-1/z, x/z-1 and x-1/z-1). Accessing others quickly leads to chained generation - thus lag spikes - and sporadic "already decorating" crashes. Fixing world gen related lag shouldn't involve more than getting mods to respect this limitation and being smarter about large feature generation.

Regarding ajthemacboy's points:

@1. This is chunk rendering. Optifine stops the client thread, renders chunk sections concurrently in multiple threads and then results the client thread again. It's quite broken, LWJGL 2 doesn't properly support multi threaded OpenGL accesses and the mod render handlers/hooks involved in the process aren't necessarily thread safe.

@2. There's no guarantee that the code executed by thread won't share state with another thread in a non-safe way, even if no other thread does anything strongly related/similar. Just like Optifine's multithreaded rendering it may mostly work, but break randomly in obscure ways that are difficult to track and hardly fixable. I believe this is only marginally easier than TickThreading's approach. There's a few cases where some gains may be achievable with a lot of trickery, that's been on my TODO-list for quite a while due to high risk/questionable benefit/lack of time though.

@3. For the most part this refers to forking off an isolated task, processing it concurrently and then collecting the results to resume normal operation. The obvious limitation is that the tasks have to be well parallelizable and sufficiently large, otherwise the associated substantial management overhead will dominate. Minecraft doesn't have much in this area as large working sets are not desirable in a realtime application.

Decent multithreaded processing requires cooperation from mods, only Mojang can require this effectively by simply distributing everything over multiple threads. Apparently they tried doing so with a thread per world during 1.8 development, but presumably their code didn't like it with all the global state involved, so it has been scrapped. Doing it with mods that aren't aware of it or on a more granular level, i.e. a thread per world and task like worldgen, entities, tile entities, block ticks etc., is a lot more problematic.


Noire: This issue is already fixed for the next build.

1,035

Saturday, October 3rd 2015, 12:37am

Player: You meant itest-4 version?

1,036

Saturday, October 3rd 2015, 12:55am

Yes, I didn't publish anything yet though.

1,037

Saturday, October 3rd 2015, 1:11am

I will be waiting for this fix :D

1,038

Saturday, October 3rd 2015, 5:22am

That's actually really interesting, thanks Player.

Would I be right to assume that async chunk gen/loading would be easier? Or would it be just as hard if not harder?

Also have you looked into mob-spawning algorithms? I have yet to find a mod that efficiently handles spawning without it getting to 2000+ entities randomly.

Are you able to check CPU time for entities? I feel like it's the entities that cause the most TPS lag from my server. Even with only ~150, they sure love eating my servers resources. (And I don't exactly want to just set spawning to off).

1,039

Saturday, October 3rd 2015, 12:57pm

Modpack usage

Your Mod is so amazing! Its that good I'm adding it to my FTB modpack! It allows some clients to go from 30fps to 50 or even 60fps!

ajthemacboy

Tree Cutter

Posts: 24

Location: Tennessee, US

  • Send private message

1,040

Saturday, October 3rd 2015, 4:40pm

Are you able to check CPU time for entities? I feel like it's the entities that cause the most TPS lag from my server. Even with only ~150, they sure love eating my servers resources. (And I don't exactly want to just set spawning to off).


On my server I feel like it's machines that use the most resources, for example we have the Ancient Warfare mod which has a treefarm that I've found to be super laggy. Quarries and farms use a lot too. I'm not sure there is any way to optimize machines or quarries, other than possibly changing the way light updates happen.

Player, when can we expect that next beta build?

Counter:

Hits today: 1,870 | Hits yesterday: 29,849 | Hits record: 152,331 | Hits total: 65,265,234