From what you say, he already has an idea that you're gun-shy about marrying again. But feelings can change. Never say never. If it's not a major source of contention at the moment, I wouldn't say anything and just appreciate the relationship as it is now.