You want to do a good job at refactoring, and it's already hard enough to keep everything straight in your head.
Therefore, if you are refactoring, refactor. Don't be tempted to change the meaning of the code. If you are, put on your RewritingHat first. (See RefactoringAndRewriting.) A good test is probably whether you need to change any UnitTests. If you do, then you're rewriting.
After a refactoring session, it is often easy to enhance the system by putting helper methods and classes into the PublicInterface of the component / package / module they're in. Then is the time to start writing UnitTests again.
SemiOt?: where the "Put On Your <Something> Hat" SnowClone originated? Why do people use it so much? Is it an AmericanCulturalAssumption? Is it a GeekCulturalAssumption? ?