Here is an older study(pdf) that looks at how diet affects testosterone. Total fat, saturated fat, and monsaturated fat are all strongly correlated with greater testosterone levels. Polyunsaturated fat and cholesterol are both positively correlated with higher testosterone as well. Carbohydrate is negatively correlated with testosterone, and protein is strongly negatively correlated with testosterone.
The carbohydrate connection is easy enough to explain: too many carbs lead to high insulin levels which then reduce testosterone. But it's intriguing that, again, a higher protein intake is linked with lower testosterone levels. Bodybuilding diets usually push protein intake, but the charts in this paper clearly show that beyond a certain point there is a negative impact. Instead, it appears that dietary fat is anabolic, and that fat is a crucial stimulus for muscle growth.
