If you live north of the glaciation line, you might want to check out this pdf key to invasive earthworms.  You can read about the ecological groups of earthworms here, and can also see photos of some of the worst invasives.  We should all be very careful about any earth-moving operations which can introduce invasive worms, and should definitely refrain from dumping excess bait worms in the wild.

The question I really wanted answered, though, was --- should I hunt down my vermicomposting worms and smash them?  Lumbricus rubellus is an invasive species which is occasionally used in worm bins.  Luckily, most vermiculture worms are Eisenia fetida, a species that appears to be safe to use, even though it's not from around here.  You can identify the troublesome L. rubellus by its dark red to maroon color with a yellow underside and no striping between segments.  If you have it, kill it!  Luckily, it looks like our worms are Eisenia.