That sounds like the situational and plot based bonuses that either Stunts or aspect invocations would provide, depending on how often they'd come up.So keep your skill at 0 and buy a bunch of stunts.Your story really only reinforces the idea that the suggested mechanic only works if EVERYONE has one.