Consider replacing instances of `Vec` with `HashSet`

[hwittenborn]

From what I'm reading in some places, HashSet can be a lot faster than Vecs in certain scenarios, and we should look into replacing time-sensitive parts of the codebase with uses of HashSet instead.

I'm not entirely sure, but it may solve issues such as #31.