I can't stand Brady, but if he was my #1 QB, there's no way I'd sit him for Cassel. Sure, NYJ defense is verrrrry good, but they aren't perfect. Plus, NE cheats (yeah, I haven't gotten over that). Besides, the Jets are great at CB against deep threats. I see Brady going underneath a lot to Welker and his two TEs (Gronk and Hernandez). The real downside to Brady is that they may run a lot more than normal because the Jets are a little soft against the run, thus limiting Brady's passing yards. However, with NE, you don't have to worry about them getting a big lead and sitting on it. They seem to enjoy running up the score, so even with a big lead, Brady will continue to throw.
By my league's scoring system (4/-1 for pass TD/Int), Brady got his lowest point total of the year against the Jets so far. That was 15 points. Cassel only scored more than 16 points once all season, against IND. I can see how you might be tempted to start Cassel over Brady this week, but frankly, I don't see how you could seriously consider it...go with Brady.