It's a combo of a lot of different pressures.
Producers want to make the money back, but since the industry is so small, they also want their games to be successes (i.e. "I produced this!"). So they end up basically playing middleman between the developers and the actual folks holding the money pouch (operations directors, major shareholders, etc.), and the latter are not especially friendly to flexible deadlines and changing milestone goals.
There's also a big tendency in AAA titles towards feature creep; it's a hit-driven business at that level, which means that every time there is a new hit, related games in that space need to either have key features they have or make a credible argument for why they don't need them. It's like how everything has to have deep multiplayer now. Multiplayer is far from a bad thing, but it may not be the best fit for every game, and trying to shoehorn it (or any other "must-have" competitive feature) in absolutely takes dev time away from other things.
Add in the typical American corporate bloat, and it's not too surprising that we end up with games that really should have gone another month or three in the clean-up/polish phase.
Right now, Valve is the closest thing we have to a publisher/developer house that is relatively streamlined and focused enough to (mostly) overcome these problems, and large part of why a lot of this stuff doesn't happen is also because they have no public shares and Gabe Newell is much more insane game designer than money man (which says quite a lot, seeing as how he's possibly a billionaire).