I think I can be one of the very few who went up against a team that had Tom Brady and came away with a 'W'. That was the week when New England went up against Cleveland. Threw for 265 yards and 3 TDs while accumulating a whopping 21.63. So you're probably wondering how did he lose; Boldin was injured, and Chester Taylor and Reggie Brown were on bye weeks. Result: he lost 106.24 - 93.32.
My advice, hope that your opponent has players on byes or are out for that week, then you'll have a chance to win because Brady is white hot right now. No pass he can't complete, no receiver he can't find, and no defense he can't read. It is a matter of luck, just like fantasy football always is.