Which is a state of affairs I share with the sleeping bag industry. It's all rule of thumb "X" inches of dead air should result in "Y" insulation value, or dummy "C," when warmed and then cooled, should mimic human body type "D" and so on. The variables are endless: Individual metabolism, fat/muscle ratio, whether a person is tired or fresh, what they ate for dinner, and on and on.
Bottom line: For you, the Polarguard bag ain't cutting it. For someone else, it might be fine. Certainly, it sounds like you're doing all the right things. You have a good pad, you're dressing warmly (ignoring the old saw, "you'll sleep warmer if you go to bed nude!"), and wearing a hat to bed. It is true that eating a calorie-rich snack before bedtime can help; a Snickers bar or something like that can make a significant difference. Maybe keep a snack handythe metabolism of most people slows around 3 A.M. or so, which probably is why you're waking up cold around then.
Otherwise, you just need a warmer bag. Or maybe a silk liner ($40 to $60) would add just enough insulation to keep you comfy.