Gosh Gilles, that's really dependant on your subject matter and what kind of photography you normally find yourself shooting. It might also depend somewhat on your environment. Here is what I normally find myself carrying:
Clear or Skylight 1A: This I usually keep on my lens always. Simply because I think its good practice to have something as a protection against the lens glass itself. Modern lenses and sensors don't need much UV blocking, but often a clear filter cost more than a Skylight/UV. Some Skylight's add just a tad warming to the image, hardly noticeable. If you don't like that you can look for a clear filter.
Tiffen Soft FX: This is a soft diffusion filter, great for people photography and portraits. It comes in different strengths, I use a medium level. It has the ability to give a nice soft touch to faces, and dimish blemishes, scars, etc. Tiffen has several varieties of these type of effects filters. I like the SoftFX because its very subtle and doesn't look like an "effect".
Warmup 81A or 81B: I love having a warming filter. Especially outdoors but the softer one even indoors sometimes. If you shoot outdoor landscapes, nature or people, a warming filter can simply bring out the richer tones you'd normally see with better light, or as if you were always shooting during that sweet time of day just before dusk. This may be a generalization, but its more or less true, or true enough to have one with you once you know what scenes will most benefit. Sometimes I take the warmup off during that sweet spot of the day or it could be too much. I come from Velvia chrome shooting, so I'm always trying to get better color out of digital and a warmup helps. I love the richer tones.
ND/Neutral Graduated: Comes in different levels. Good to have 2 within a few stops of each other. A must for landscape type work, when you have too large a range of light difference between foreground/background, sky/ground, etc. I don't use it much but its indespensable when you need it, because you can't fix that kind of thing very easily in PS. When you can it's very time consuming and never as good as getting it right in the camera.
Circular Polarizer: A must for landscape and beach scenes. Cuts reflections from water, glass, etc. Can darken or lighten the scene in general. Can focus the photons in variable directions, allowing you to tune in the hues a little bit. Takes an average of 2 stops of light. Can acheive deep color tones combined with a warmup filter, if you have enough light for that.
If you only have one filter, for me its a clear or skylight 1A and I keep it on the lens always. Its good protection for your lens itself, and makes cleaning it easier as well. I don't like anything to touch the actual lens glass any more than necessary. It always keeps it protected from the elements, which could be sand, bugs, atmospheric particles you don't see, etc. And if you bang it into a wall or something otherwise dreadful, usually you can simply replace the filter which took the heat while protecting the lens.
That's it for me.