Compels are negotiated, and not merely to the extent that they can be refused. They are also subject to a test of 'reasonableness' to be judged by the table.
Personally, and with the groups I have gamed alongside, 'you lose the fight' is very rarely a 'reasonable' compel.
Something to the effect 'you need to offer a Concession' might very well be acceptable, though. For instance, an aspect might be invoked representing a dire and urgent need to address matters elsewhere, forcing an opponent to flee, or an aspect representing a fiery temper might be invoked to force a concession in a duel of words (possibly resulting in the beginning of a new physical conflict scene to replace the now-ended social conflict).
This, though, is typically the extreme end of the power of compels. Handle with care, and remember to negotiate, negotiate, negotiate. Everyone should come out the other side happy with the results. If they don't, then something has gone wrong.