Alan,
I had an '07 E92 335, traded it for an '09 X6 35, and I'm just moving to a '12 X5 50. I *love* the big X models.
I had a few minor glitches with my X6 (which I didn't have with the 335) but the dealer has always fixed them quickly and easily. For such a big car, the performance is not at all bad with the 35 engine (mine was an N54, the N55 is supposed to be even better!).
The trim is good quality, the interior was actually less rattly than my 335.
If you're buying tho - find a car with comfort seats ("multi-contour seating") and try them. I bought off the lot and was "forced" to have the fancier seats I wouldn't have otherwise paid for, as the sport seats in the 335 were so comfortable. I will never buy a BMW again *without* the comfort seats - they're incredible
I'm only going to the 50i "because I can", honestly. It sounds a little more raw than the 35, but it's got nothing to do with the lack of performance on the 35 engine. I think you'd enjoy an X5
-Alistair