They do get set back a bit. If memory serves it's the last time they woke up, got sick, digivolved, etc. So you'll lose a few hours but nothing too intense.
It's tough when trying to get a specific character, you lose where you get training, care misses, etc, since the device doesn't show you...