It's not selfish if it's causing you distress. In that sense, you're abandoning yourself/your own needs. You can be the most empathetic person in the world, but what good is it if it's affecting your wellbeing? It's possible to be empathetic and also set boundaries as to how much suffering of others you're willing to take on. The suffering of others won't go away, but your health will if you take on too much of the emotional burden.