Personally, my hangup is that if you have some kevlar or composite weave or SOMETHING, and you start stitching runes into it, you should be able to reinforce it for less effort than it'd take to create an equivalent defensive effect without that underlying infrastructure. Getting Armor:3 from kevlar should be easier, require less work and personal investment, than from a t-shirt.
On the other hand, the system doesn't really _track_ non-magical items on a character strength/balance level, so there isn't anything currently to keep everyone from going "My armor is this special material plus this extra enchantment"..
And of course the gripping hand is that we haven't seen anyone wearing enchanted body armor yet, have we? Is that because of traditions/closed mindedness on the part of wizards we deal with? Aside from Harry, who hasn't had the resources/funds for actual armor throughout most of the series, it's pretty well established that most of the White Council is kind of stuck in their ways.
I feel like I'm missing a crucial piece of this puzzle.