Several years ago I got a book by Ted Shifrin on multivariate mathematics. It's a textbook intended for those who have taken calculus and it presents linear algebra and multivariable calculus in a joint course. At the back is suggestions for further (more advanced) reading on topics covered. One recommendation is for a book by Michael Spivak entitled Calculus. It's a single variable calculus textbook. In other words, it's more basic than Shifrin's text. Yet it was suggested for further reading.
Even more startling, unlike most calculus textbooks it doesn't begin with a quick review of precalc or a chapter on functions but with addition.
That's right. Here's a book on calculus more advanced than one intended for those who have completed both single and multivariable calculus and it begins with addition.
Why? Because it's really more about analysis (which calculus serves as the doorway to). Most calculus textbooks quickly get to limits and then derivatives. This one starts with numbers, then more about numbers, then functions, and doesn't get to limits until a few chapters in. Why? Because it is about really understanding what adding, subtracting, etc., is. It deals with proofs. It deals with the structure of mathematics imposed by operations on a set like the integers. It's about not simply learning how to solve problems in calculus but what calculus really is.
Einstein wasn't all that great with math and he was a genius. But it is one thing not to know math well and be smart. It is another to be smart and not know math well but try to redefine it anyway. The route to understanding isn't through trying to reinvent the last several thousand years of mathematical study. You'll never get as far as Euclid and Archimedes, let alone Leibniz and Newton. If you wish to really understand the basics, there are resources that will allow you to do so. Nor is it a waste of time. I used that calculus book which began with addition after I had already been teaching high school math, test prep math, college statistics, and had worked through calculus. It was still useful. Mathematics is about learning to be clear and explicit. It's about syntax and structure. Sure, philosophy enters into it. But one shouldn't try to take a position on the philosophy of mathematics without knowing a decent amount of math.