You are using 64 bit Java? The working set size is suspiciously close to what a 32 bit program on a 64 bit OS may have available. With 3 GB heap the remaining 1 GB would be insufficient to hold the texture data in the OpenGL client. Where did the remainder of your 8 GB RAM go?
I've calculated a total extra of 500-600 MB with this texture pack when using FastCraft, not taking memory savings elsewhere into account. The block texture atlas is 8192*8192 px with almost 10k individual textures, the items one is 8192*4096 px. It'd be interesting to know what "/csampler memory" reports besides the task manager readings (see 2nd spoiler in the op) to know how heap memory behaves. I unfortunately can't compare against Vanilla myself, since I'm running into a bug when using its stitcher instead of FastCraft's -_-
I've already changed some code to reduce peak memory usage during startup. Additionally I could disable an optimization which is memory intense with such large texture packs, 64x textures use 16 times as much memory as the normal 16x ones. I'll need the extra data from sampler before being able to determine a decent threshold for this. Taking the optimization out is not an option since it's quite beneficial with enough RAM or smaller textures.
Trying to run anything while this issue is happening is nearly impossible. Where did the rest of my memory go? That's a very good question. With FC in something is basically blowing through 6G of physical RAM and rolling over into swap which brings the machine to its knees. Unfortunately it's not being reported well (<sigh> Windows) and not be attributed to the Java process itself, at least with system tools. I don't currently have the hours it would take to iterate through some regression style testing. Given enough spare time I could pull up VisualVM and dig down into things at the class or thread level. I'll just have to pass on things for now. Thank you very much for your time so far.