I'd say absolutely not.
With pick 3 you really need to take a stud RB. Adrian Peterson, SJax, Westbrook, Addai, LT should all go ahead of Brady IMO.
If you really want Brady, I'd try and trade down 2-4 spots. I just can't see picking him at #3 overall. Even at the 7 spot if someone takes Brady ahead of you, you can get one of the RB's I listed, Manning (if you are really QB crazy) or Gore.
Conventional wisdom says to load up on RB's and it's good advice. Take a RB with your first round pick *for sure at the 3 spot* then you can start to be a little more flexible.
Generally it's easier to find servicable WR's to fill in roster spots any given week than RB's which is why you don't want to screw yourself with weak RBs.
This all does depend on your league settings, though.