That is an odd scenario. If my two cents are worth anything I would offer these as possible answers:
First, and most correctable- they saw you. Be it subtle movement, poor stand/cover, bad camo. This is the easiest fix, but probable one you've already corrected.
Second- Scent- No wind and heavy air= odor everywhere. If there was no wind your odor will permeate where it wants to go to. The other scent issue is if these deer crossed where you came in. They'll smell you on the ground and scrub that you brush against.
Third- It is possible that your spot is not as good as you think, unless you're hunting does. In areas that aren't particulary deer friendly, such as open or low cover funnel areas does will "blow" through them. I feel it is their generally uneasy feeling in these areas and also trying to establish noises they hear as a deer(which they say will blow back).
If I had not witnessed this on multiple occasions in different areas, I would take this third reason as bologna. However, couple this with not seeing bucks(usually a good sign of a poor area), and I would put money on it that this is not only the reason you are hearing all the blowing, but why you aren't seeing the bucks.
Good luck. Hope it improves for you.