I never saw it as a rude one ... back to our little discussion while ago : it's informal but not impolite ...
example :
(said by Hugh Grant character who is a polite British man )Actually, I can't think of what it is, really. Let me just have a quick look... No, no, beats me.
or:
(said by a character in The Lion King 1½ which is an animation for kids)Pumbaa:Beats me, Timon. What?
and countless examples in conversations