I've been working with people starting their transition to agile. They are all smart people—some of them scary-smart. And some of them are misusing some of the terminology. Some people are using the word “refactoring” to describe significant work, say, weeks or even months of effort of rework. Sorry, I call that redesign.
To me, refactoring is simplification of existing design. It's something you can complete in a few hours, or, at most, a day. Refactoring could take a day if you need to write some tests to verify you haven't broken anything.
But if you're taking several days to a week or more to “refactor,” I don't think you are refactoring anything. You are redesigning. Redesigning is a useful activity, especially if you have technical debt. But call the activity what it is.
If you have enough technical debt to need redesign, say so. Call the activity redesign. Allow for the time. Explain that you are paying down technical debt. If you're like me, you might even chunk the redesign into smaller pieces, so you can iterate on the redesign, paying down your technical debt in pieces.
If you're refactoring, that's something you can do without needing a ton of extra time. Refactoring is something you should be able to do in the time allowed for the story or the task, because most of the time, refactoring is sufficiently short that it is absorbable into the time for the story. If you see technical debt that is not absorbable, make a card and put it on the backlog, because you need to make the technical debt transparent to the rest of the team, including the project manager, and most especially the product owner or the customer. No matter what, you need to make technical debt transparent to management.
Refactoring and redesign are two different beasts. As a project manager, I want to know when team members are performing which activities. I want to make the choice about refactoring a “yes” by default. I want to be able to make the redesign choice with the product owner or sponsor, because it affects the project cost and schedule and the sponsor needs to have that discussion. Redesign is not a default yes decision. It depends on many variables.
But you can't have that discussion if all redesign is called refactoring. It's not. Call the activity what it is. If it's redesign, call it that. If it's refactoring, call it that. But don't call redesign refactoring just because you don't want to call it redesign or because you think the cool new word for redesign is refactoring.
Redesign is still redesign. Refactoring is simplification, and takes very little time. Make sure you don't confuse the two. Once you start messing with significant technical debt, watch out. You are likely redesigning, not refactoring.