Don't log exceptions when failing to fetch server keys

Not being able to resolve or connect to remote servers is an expected
error, so we shouldn't log at ERROR with stacktraces.