Hunting clothing has come a long way over the years, as has our awareness about how to dress for different conditions. Regardless of how many layers you wear, at some point in your layering you MUST stop the wind in order to stay warm -- and the farther from your body you stop it, the better.
The key to staying warm is to build up a barrier of warm air next to your skin. That is what insulation does -- traps your body heat and holds it, creating a buffer zone between you and the outside air temperature.
Keep in mind that insulation weight does not factor into stopping wind, however. Wind will force its way through all your insulation and rob you of your barrier layer of warm air.
So when considering your clothing options, make sure to invest in some type of wind-stopping outer garment. If it's a true wind-blocking fabric, it will have a special tag or logo that tells you so.