My only experience with the F10 M5 is as a passenger in a hot lap around the BMW Handling Course / skid pad at the BMW Performance Center, most of which was spent going sideways drifting in the wet. It was freaking awesome.
I've also had the pleasure of driving an X6M (not mine, it was BMW's - thanks BMW!) on the same track and she did awesome. What BMW has engineered with the X6M is nothing short of a miracle, it drives and handles like a car it's size should not be able to.
I think the X6M is similar in room for the driver and passengers, with more cargo space. You also sit higher. So if you think the M5 is huge, you're not likely to think the X6M is smaller.
That being said, I daily drive my X6M, and take my 2 young kids out in the DadMobile (tm) all the time. It's awesome. Drives like a car (but not like an M5) and the performance is definitely there for any driving situation. It's not a dedicated track car and as such is perfect for what it is.
It will feel 10x smaller than your GL (I have a GL550 as well), be much faster to the point that it's funny, and handling is not comparable; the GL is a barge when it comes to handling. It won't be as nimble as a C63 or an M5, of course.
Luxury wise, make sure to get one with Full Merino leather and Nappa leather dash, and all the tech gadgets, and chances are you won't even know there's unexposed plastic in the car unless you spend a lot of time staring at the B-pillar or the inside of the trunk.
I considered an M5 before getting my X6M. Part of the decision was that I previously owned an X6 35i, loved everything about it and missed her after I sold her. No regrets here after 11.5k miles since January including a 1200 mile road trip from BMW Performance Center through Tail of the Dragon back to South Florida.