Refresh can never drop below 1, or you are considered an NPC.
However, you can easily drop to 0 fate points, simply by spending them all. If you did spend them all, you'll get 1 fate point at a refresh event, since you have to have at least 1 refresh. Other than that, you can have pretty much any number of fate points you can think of. Including negative numbers (aka pre-accepted compels).