Division by 0 isn't defined. In order to get to a point where you can do calculus on the real or complex numbers you first have to build up the set theoretic and algebraic structures. But the required axioms don't allow for division by 0, because for ANY number a, if a|b iff there exists a number c s.t. a*c=b
Thus, if a=0, then c=0 and b=0 and the necessary structure doesn't hold. This structure is required for the necessary inverse properties.
Also keep in mind that it was widely believed for a long time that 0 wasn't a number, and that negative numbers were nonsense. What is or isn't a number, practically speaking, is a matter of definitions. More philosophically, but still mathematically, it has more to do with preferences and the philosophical prejudices one has.
Finally, I'm not sure what your point is when you include this: lim x-->infinity (1/x) = 0
After, all, as you know, lim x-->0 (1/x) = infinity. So it isn't just a limiting process but also the limit point itself, which is a bit of a problem (from a rigorous approach) if this point can't even be approached in the reals let alone be fined as the unique limit. These and similar issues are one thing that motivates the extension of R to to R* (the extended reals) and the associated algebraic properties of -infinity and infinity.