Hello Mr."Player" MinecraftChar, I have (unfortunately) encountered a bug when combining with optifine and other mods in the FTB Infinity pack, I'm not sure if you'll be able to track down the problems, but i'll tell you the conditions and what exactly it is. 
I play FTB Infinity with the latest version of Fastcraft and Optifine, and I use sphax's x128 patch for it, when i go to the title screen my computer's calculations slow down and it lags, even though it is an amazing computer, and the title screen FPS is always <2, then when I join the game, My Fps is about 2-10, And optifine's Lagometer shows it is from post-graphic render and other calculations, not rendering, but my graphics card can handle it and I have 12GB RAM allocated. If I don't use optifine, this doesn't happen, but my fps is about 20-30. If I only use optifine, my fps is about 55. If you would like a console log and dxdiag log etc. i could supply you with that. Either way, Don't rush to fix it, because it could also be a fault in optifine, and i can survive without the mod (for now) in the modpack.