Do you have them configured to clear those settings on exit? Is the certificate actually the same when you visit the site again?

Chrome and Firefox remember the acceptance of self-signed certs for a long time on my PC.