What should we say to such people?
I could mention Matt 16:18 ("thou are Peter...etc) and I could mention the Church's own claim to infallibility (Lumen Gentium 25 of Vatican II and Pastor Aeturnus of Vatican I) but I think the point can be answered more simply.
In fact every serious belief system considers itself possessed of a principle of "infallibility" in the sense that it considers its own principal doctrines to be absolutely true. Even Secularists consider it self-evidently true that there is no God or at least that God should have no part in the public affairs of men.
The difference is simply that the Catholic Church defines the scope of its own view of its capacity to teach infallibly rather more scientifically than any other belief system. It thus sets out the boundaries of its own infallibility with some precision, unlike all other belief systems which more or less expect their disciples to "take it as read" that the principal teachings of their belief system are absolutely true.
The circularity involved in saying that doctrine "X" is true because it is true, is so obviously unsatisfactory to any real searcher after truth that the Catholic Church, following the lead of its Founder, took steps to set out clear parameters as to when - and more importantly why - some of its teachings were taught as infallibly true and beyond question.
The advantage that theists have over atheists is that we can appeal to God as a final authority (provided, of course, that He has given His view which, in the case of Christianity, He has, through Revelation).
Atheists are thus only able to say on any matter not clearly knowable by deductive or inductive logic, in effect, "I am right and you better believe me". That is because they do not believe in any higher authority than man and, indeed, often do not much believe in higher authorities even among men. They therefore have no higher authority to which they can appeal in the event of dispute.
Or, as Gilbert and Sullivan put it more succinctly, "when everybody's somebody then no-one's anybody".
In short, one man's view is as good as another's and so, absent respect for logic, truth boils down to having more people on your side than the other guy. In short, might becomes right.
To be fair, many atheists do respect logic. But on any matter that transcends logic alone, such as the existence of God, of spirits, of an after life, of the cause and origin of virtue, vice and free-will, they have no answer other than their own unaided opinions.
Theists have God.
But that is not enough. After all, who knows the mind of God?
We need to know what God says, at least, to us and we need to know with certainty and precision.
Step forward the principle of infallibility.
This principle is a logical extension of the idea that there is right and there is wrong, that there is truth and there is falsehood.
Those who deny that truth exists are self-defeating since the very statement "nothing is true", is, itself, being asserted as true, but, if it is true, then there is such a thing as truth and the statement is false anyway.
It is like Bernard Shaw's self-contradictory rule that "The Golden Rule is that there are no Golden Rules". One merely replies to him "including your own Golden Rule?".
Given that there is truth, then, it follows that some things must be true. We can arrive at truth by the use of logic but, as I said above, some ideas transcend logic and cannot be knowable by the same.
How then do we know whether they are true or not?
The only answer can be that someone with greater knowledge than ourselves - greater than all of humanity - must be able to tell us (and, once told, we may then be able to apply our skills in logic to what we have been told). That must mean either a being superior to men, such as a spirit of some sort, or perhaps a being from another part of the Universe or perhaps even from another Universe.
Of the latter two, scientists have only been able to speculate since no human has ever met such a being.
As to the first, we have more concrete scientific knowledge.
We have various religions which claim to have had metaphysical knowledge communicated to them from spirits of one kind or another. So far as I am aware only two claim that such knowledge can be taught and mediated infallibly - the Mormon Church and the Roman Catholic Church.
Thus, the majority of religions are nearly in the same difficulty as the Secularist in that they do not have any mechanism by which they can, with certainty or even precision, confirm that any of the teachings taught them by their spirit guides are absolutely true. One is more or less driven back to "they are true because I say they are true".
The difficulty with the infallible organ of the Mormon teaching office is that it freely contradicts itself. "X" is true today but tomorrow it isn't. A more obvious self-contradiction cannot be.
For a truth to be absolutely true, it must always, everywhere and forever, be true. That is logically self-evident.
Thus, as Newman reminds us, doctrine cannot be true if it is not consistent with itself here and now, in the past and in the future.
The Catholic Church claims this consistency and not only invites others to put it to the test, scientifically, but puts itself to such test with regularity and rigour. It also applies the other tests that Newman adumbrates in his Development of Christian Doctrine, viz., preservation of type, continuity of principles, logical sequence, conservation of past principle etc.
Thus, the Catholic Church is among the first to admit that truth not knowable by logic requires a higher teacher than humanity, posits such a teacher in God, admits that His teaching is of no use unless communicated to mankind, witnesses to the fact of His having done so, teaches that it happened by means of God assuming human form and so teaching men, His teachings later being partially collected in writings called "The Books" (ta Biblia in the Greek of its time) and the power to interpret "The Books" being imparted by this God-in-assumed-human-form to His formally-appointed successors, particularly the chief of them, St Peter, and his successors, the chief bishops of the Catholic Church.
This arrangement appears to be almost unique in the annals of human history. That gives it a claim upon our attention, if nothing more.
Of course, Catholics will claim, like most other belief-systems, that our system is absolutely true. But we have an added advantage. We can show how we arrive at such absolute truth, with precision and with certainty. They cannot - at least nothing like to the same degree.
The Chief Bishop (Father or "Pope") has been appointed by God, when God visited men, as the authentic interpreter of The Books. In interpreting The Books, he can, logically, also interpret what it means when it records the moment when God appointed him and his successors as its interpreters.
The chief bishops and their advisers (or councils of bishops) have done precisely that.
They have done so in very precise, scientific and legal language so as to provide the very certainty and precision that I mentioned was lacking from most other belief systems.
Not only that but they have done so in a manner that leaves no room for doubt or uncertainty, knowing that men are entitled to know exactly what the God-appointed interpreters mean by particular teachings. They have used legal formulae to "define" and to "exclude", to say "we define" and to say "let the opposite view be anathema". This provides clarity and certainty of the most consummate kind.
Any fool can bumble, guess, procrastinate or prevaricate. A real teacher must be able to teach clearly, precisely and with authority.
Such the God-appointed Catholic "interpreters" have done and continue to do.
We can access and read their clear teachings at our choice and at our leisure.
The more recent include Pastor Aeturnus of the First Vatican Council and Lumen Gentium 25 of the Second Vatican Council.
The first of these even uses the same precise, clear and defining formula to set out the bounds of the same certainty. It teaches thus:
"It is a divinely revealed dogma that the Roman Pontiff when he speaks ex cathedra, that is:
1. when acting in the office of shepherd and teacher of all Christians;
2. he defines;
3. by virtue of his supreme apostolic authority;
4. a doctrine concerning faith and/or morals;
5. to be held by the universal Church
possesses through the divine assistance promised to him in the person of Blessed Peter, the infallibility with which the divine Redeemer willed His Church to be endowed in defining doctrine concerning faith and/or morals, and that such definitions of the Roman Pontiff are therefore irreformable of themselves and not because of the consent of the Church (ex sese, non autem ex consensu ecclesiae). But if anyone presumes to contradict this our definition – which God forbid! - anathema sit."
This is admirably clear and scientifically precise.
No other belief-system has this admirable degree of clarity and scientific precision in setting out the limits of its own teaching authority.
Note that the Council teaches that it is a "divinely revealed dogma" and that those who contradict the definition are to be held anathema.
It is, indeed, a high degree of precision which is here displayed. No-one need be in any doubt about this teaching. Any man can quite readily say to himself that he does or not believe this teaching and thus is - or is not - a believer in the teachings of the Catholic faith.
(the popes wore red until Pope St Pius V, a Dominican, continued to wear his white Dominican soutane, establishing the tradition of popes in white soutane)
Any man can apply this yardstick to any teaching emanating from any Catholic source and can readily see whether or not it is taught infallibly and thus is (or is not) an integral part of the Catholic faith.
For instance, the Catholic Church has infallibly taught that the Virgin Mary, the Jewish and human mother of God-in-assumed-human-form (whom we call Jesus Christ, from the Greek Iesous Christos, and the Hebrew, Y'shua Moshiach, meaning "anointed saviour"), was taken up to heaven, body and soul, upon her death.
Equally, for instance, the Catholic Church has never taught, let alone infallibly, the doctrine that some races are inferior to others.
One can thus readily apply the relevant criteria to any idea, teaching or proposition.
Those who seek to deprive the Catholic Church of its charism of infallibility thus do a tremendous disservice not only to the Church but also to human civilisation as a whole by eliminating the admirable clarity and scientific precision which the Church applies to itself, unlike any other belief system.
Even if the Catholic Church were not the true religion (which it is) it would be a retreat into obscurity to prefer it to deprive itself of such admirable clarity and precision and would set at defiance the efforts of men to reach truth with as much clarity and certainty as we can.
Far from being the approach of a scientist it would rather be the retreat by an obscurantist who preferred muddle to precision, opacity to clarity, something which no real scholar, scientist or philosopher, worthy of the name could possibly prefer.
Yet there are some Catholics who would prefer a retreat into obscurity than an advance into clarity. Not liking the clarity that they see, they retreat from it.
Note also that the criteria provided infallibly by Pastor Aeturnus does not restrict an exercise of the ex cathedra teaching authority to any time, place or method, beyond the 5 criteria.
Thus it is an unwarranted restriction upon the application of Pastor Aeturnus to aver, as some do, that the two Marian dogmas promulgated in the last 200 years are the only examples of infallibly taught doctrine over that time.
Pastor Aeturnus provides no such restriction.
Unsurprisingly, therefore, Cardinal Newman himself believed that the papal encyclical Quanta Cura of Blessed Pope Pius IX taught doctrine infallibly.
Others say that the Pope cannot teach infallibly on his own but only by consent of a General Council.
Such a "conciliarist" view was infallibly condemned by the 4th Lateran Council (1215), the Council of Lyon (1274), the Council of Vienne (1311) and - in the most express terms - by the 5th Lateran Council (1512).
The teaching contained in the papal encyclical Humanae Vitae occasioned much controversy as many thought that it would teach that artificial contraception is morally licit. It taught the opposite, in fact.
Whether it taught so infallibly becomes of lesser importance when we acknowledge that such teaching was already infallibly taught as true by virtue of the ordinary infallible teaching of the bishops dispersed throughout the world.
The infallibility of this so-called "ordinary" infallible teaching office has been taught since the days of the Apostles but was most recently and conveniently re-affirmed in Lumen Gentium 25 of the Second Vatican Council which taught:
"Although the individual bishops do not enjoy the prerogative of infallibility, they nevertheless proclaim Christ's doctrine infallibly whenever, even though dispersed through the world, but still maintaining the bond of communion among themselves and with the successor of Peter, and authentically teaching matters of faith and morals, they are in agreement on one position as definitively to be held".
There can be no doubt that a moral unanimity of bishops throughout the world always taught, consistently, against the licitness of artificial contraception. This was thus the infallible teaching of the ordinary teaching office long, long before the encyclical Humanae Vitae was issued.
Many Catholics, however, would rather throw out the baby of clarity of teaching with the bathwater of the teaching on contraception because they have found it inconvenient in their own personal lives. They love vice more than truth.
I need hardly add that the inconvenience of some individual persons cannot be a basis for over-throwing the teaching office of the Catholic Church established, as we saw above, by God himself when he formally appointed his followers to be authentic interpreters of His own teachings.
If God is anything at all, He must surely be forgiving of creatures whom He knows to be prone to sin. Therefore, there is simply no need to re-write His teachings to suit our vices. It is simpler and better just to say sorry for not reaching the set goals.
As G K Chesterton once memorably put it, "The Christian ideal has not been tried and found wanting; it has been found difficult and left untried".
More importantly, we do not advance into clarity by retreating into obscurity.
Let us, then, thank God for the gift or charism of infallibility, the guarantor of clarity in teaching or doctrine.