I don't think actually compelling is always necessary. Certain aspects demand certain types of actions, even without a compel. If you are "on fire", you are burning, your immediate reaction should be to do something about it. If you are on the floor, you need to get up before you can do any running around.
That's not the same as being compelled to do something. It's just a logical conclusion of the story. You just can't do everything from any position you are in.

However, if that maneuver forces you down a specific way, and that's to your detriment, that's a compel.

I'm a bit more of a hard-ass for aspects

Maybe you paid off the compel, but you still haven't removed the aspect.  If you pay off the compel and still don't try to remove the aspect, you are still subject to future compels, depending on the situation. 

It makes maneuvers actually have weight and encourages players (and npc's) to try to remove the aspect.

Drawing a sword is not an action in the sense of fate. It can be an action, if you draw it, for example, to intimidate someone. In that case, it would be a maneuver to tag on a subsequent intimidation attempt.

But you don't have to do an action to draw your sword. Unless it has an impact on the story. So if there is a signature move you draw your sword with, that is part of your fighting style, you can absolutely do that as an action. Otherwise, you can just attack with your sword from the sheath. But you won't get a bonus from drawing then, either.

Nope. It can also be invoked or compelled even after it's tagged.

I have to protest. The names of aspects are definitely not just fluff. If you hit someone to make him lie prone, he is lying on the ground with everything that entails. If you want to keep him down, then you'd do a block. Or maybe even a grapple. But if the other guy is "prone" and wants to run away, I would first make him get up again. There is not much sense to let someone run if he is lying on the ground, after all.

A maneuver is not just a mechanical way to introduce an advantage for yourself or a disadvantage for someone else, it changes something about the scene, it introduces a new fact. A character is lying on the ground, the curtain is catching fire, you move a chair between yourself and your opponent, and so on. The fact should be regarded as being true, not just as being there.

If a player refuses a Compel, you should either escalate or move on.  You should not simply present the same Compel again and again until they run out of the means (or the will) to refuse.  That's what's called by the technical term 'a dick move'.