The morale check is a game mechanic in D&D that is designed to remedy this, but I've always found it a bit of a blunt instrument, contingent on abstractions (25% casualties, half hit points lost, etc.) which may not be appropriate in a given situation.
My general approach these days is not to bother with morale checks at all, but just have monsters flee (or surrender) based on what seems appropriate. I try as best I can to assess, subjectively, what in a given instance would likely happen, depending on what kind of creature is involved and what is at stake. Put your mind into that of an orc, for example: you are mean-spirited, malicious and above all self-centred. While patrolling around the countryside with a band of fellow orcs, you come across a party of adventurers who are heavily armed, and one of them looks like a magician. A fight ensues and one of your comrades is felled by a magic missile. What incentive is there to carry on the skirmish? There is a good chance you'll die, and there does not seem to be a huge benefit to continuing - certainly not outweighing the chance of death. Wouldn't you aim to disengage?
As well as being realistic, of course, this approach is generally also better for the game. Monsters fleeing presents players with interesting choices: to follow or not to follow? It also presents the DM with interesting options: how do the monsters go about seeking revenge or otherwise reacting to the defeat?
What always surprises and interests me about military history is how comparatively few casualties tend to arise from battles and skirmishes when set against what one might expect - especially when it comes to what you might think of as the bloodiest part of a medieval battle, melee. Speaking in general terms, in most medieval battles the large part of the killing tended to take place in routs, when there was a disorderly retreat, and/or subsequent massacres of captives. Taking just one example, the Battle of Poitiers in 1356 was one of the major military engagements of the Hundred Years' War and a shattering blow to the French psyche, yet out of what seems to have been around 20,000 participants on both sides, after a hard day of fighting (at the very least 4-5 hours) there appear to have been something like a total of 3,000 men killed and wounded. Assuming the normal ratio of wounded-to-killed soldiers in a battle, that would be something around 1,000 deaths, or 5% of the total combatants. That would simply be unrealistic if we imagine the men involved going at it hammer-and-tongs like a typical D&D fight. Other examples will produce varying figures, of course, but it was pretty rare for a medieval battle to even produce casualty rates of 25% of total combatants, let alone anywhere near 100%. (The bloodiest battle ever to take place on English soil, the Battle of Towton, during the Wars of the Roses, is said to have produced 28,000 dead, which was approximately half the total number of combatants - but this is believed to have been exaggerated, most were killed during the Lancastrian rout, and the figure includes many murdered captives.)
In any event, this is surely what one would expect when you forget about Hollywood and put yourself into the mindset of somebody in a battle. What's your first priority? Survival. What's your second? Not being captured. What's your third? Winning. I would expect that "killing the enemy" is fairly far down the list. Battle casualty figures bear that out: what happens when 20,000 men gather together with the aim firstly of survival, secondly of not being captured, and third winning? Most of them end up surviving, by hook or by crook.
But that's only half of the issue, because of course - as I've alluded to - a rout will often end up in slaughter of the losing side. Loss of discipline and disorganisation is not a great recipe for survival. In fact it's more or less the worst thing that could happen to an army. The best thing after it becomes clear that victory is not on the cards is orderly retreat.
So I wonder whether it would be an interesting experiment to interpret the morale check more broadly, and it's as follows (assuming a 1 minute round):
When the DM rolls a morale check, a successful result means the monsters either remain in the fight, or make an organised retreat. A failed check which is better than half the monsters' morale results in an organised retreat. A failed check which is less than half the monsters' morale results in a rout.
An organised retreat takes one round, during which time the monsters are open to a bonus attack from any PCs within range, but at a +4 bonus to their AC. They then have a free turn of movement to retreat.
During a rout, the monsters are open to a bonus attack from any PCs within range which hits automatically. After this, they may flee, but initiative applies in the normal way.