Imagine this situation: The tower is one or two hits away from being destroyed, but so are you. Suddenly, a group of minions and an enemy player appear. While you might want to just take the tower down, it's better to retreat, recall yourself to base, and recharge your health. The damaged tower isn't going anywhere, and by recalling you can take it down and move to the next one with full health.