Does it need to? Does it actually matter if infinity exists if we are getting good enough results?

Imagine a discussion with a mathematician. The mathematician says to you "look at all this calculus! a wonderful theory!"

and you reply, "how do you know it's true?"

and he replies, "because infinity exists."

And then he says "look at algebra! it's a miraculous achievement!" again, you question the truth of algebra, to which he replies it rests on the idea that infinity exists.

And the mathematician continues, "and toplogy! how gorgeous!" and you ask him how he knows it's true. And he replies that infinity exists.

And you and this mathematician continue on and on, discussing topics for hours. And throughout that discussion, the mathematician keeps reaching into his back pocket and pulling out "infinitiy exists!"

When a religious person resorts to "because god" reptitively in reply to my skepticism, I do not accept it and I think him a lunatic. So why would I accept mathematics if mathematicians have to keep claiming "infinity exists!" over and over?

At one point, a god was a good enough explanation for society. And then it wasn't. Are we at the point where "infinity exists" is good enough but later we will need to do better?