Alright, so I've done some more tests and I found out that I had to set the fast previews to Adaptive Resolution -- Open GL Off. Now my GPU stores up to exactly 2GB (the value I set my texture memory to). So it looks like if I can set my texture memory higher I'll also be able to take a greater advantage of my GPU. However, After Effects CS5.5 won't let me increase the texture memory any higher than 2000MB. Is there a workaround for this or is After Effects just not capable of using more than 2GB of VRAM?
Edit: Did some more tests. For some reason 2GB seems to be the limit for 1080 video, but bumping up to a 4K comp got me to 4GB very quickly. So it looks like the high quality 3D models I was adding didn't take up as much VRAM as I thought they would. Even better than that is after changing the fast preview to Adaptive Resolution I can now have 1000 low to medium poly count models and move around the scene in real time, no lag at all. Still not sure why having Open GL on causes things to slow down so much though. Maybe because Element 3D uses their own Open GL renderer and the After Effects Open GL renderer starts to conflict? No idea, either way I'm far more than satisfied with my findings.
Edit 2: Ran some tests with a 4k aircraft model in a 1080 comp. It takes about 1.5GB of VRAM to load it, but once I run a RAM preview it takes up 3.5GB. Looks like it all depends on the models you use in your projects. If you aren't using crazy high res/polycount models then you'll be fine with 3GB, or maybe even 2GB of VRAM, but as soon as you start using high res models, especially high res models of very large/complicated models like large planes, people, cars, etc. then more VRAM definitely makes a difference. I hope this helps someone else down the road!