Why do people say that Mitt Romney's religion should not be a factor?

I believe his religious beliefs need to be taken into account and that may influence how he does his work and how he reacts to certain ideas. We need to understand he comes from a church that does not have a good history regarding racism.