Naysayers will make you question your decisions, and it's okay to consider naysayers' comments, but then you have to make the best decision given your knowledge, instinct, experience, and desires.
I write this because too often in life I've followed the naysayer's advice only to be disappointed later. Now I realize that naysayers will lend their advice for all kinds of reasons and sometimes those reasons have little to no merit.
Yet, if a naysayer is sharing caution because they are afraid of danger or big problems, it may be best to follow their advice or at least use their advice to research the situation more. People usually don't share their worries without due cause. Also, as the old adage goes, it's best err on the side of caution. That way, even if you're wrong, you're safe.
If you give decisions good thought, consult, research, and care, you need to trust yourself. Onward.