You need an appointment for either one. I've renewed my passport there before and attempted to get something notarized another time. For the notary thing, I needed it urgently, so I had gone there without an appointment, hoping they'd attend to me. They didn't let me in and said I needed to make an appointment.
I ended up using an online notary, for $25 (notarize .com). I'd recommend that. It was $50 at the US embassy then (2022), if you have time to wait for an appointment.