Are you also starting 2 QB's, 3 RB's, and 4 WR's? Curious, if so, then I'd expect some owners will make trades when needed and will be doing so to cover defense bye weeks also. If all 32 defenses are drafted evenly, 8 teams will have 3 defenses and 4 will have 2. Surely 4 teams can make trades, and if starting 2 QB's, teams probably are trading anyways.
If not starting 2 QB....., then I'd recommend going down to 1 Defense and 1 kicker.
League champion two years running on XpertSports.com (free survivor leagues and mock drafts at www.XpertSports.com)