For what it's worth, I spoke with someone locally who's looking to make the move from a '13 S4 to an E90 M3. In fact, he might likely buy the one I have for sale if he could first sell his car. He spoke of the S4 having a great drivetrain but lacking in the steering feel/connectedness department. He has decided that the M3 would be more of his ideal car.
Good luck with your decision!