See the problem here is that all of the symptoms you mention, bar none, can be caused by many other ailments/medical issues, not necessarily specifically low test.
Hell I have most of the symptoms you mention, and I am on a pile of test
But the doc is failing you regardless. You have went to him with a problem, and a possible solution for him to look into, and it seems he has dismissed you out of hand, not just in respect to your suggested first area of looking, but full stop.
Unnacceptable IMO, get another doctor - and keep shaking things up until someone helps you.