To the point about the tone in this passage in particular, to me the author's "confidence that the existence of numerous unstable systems would call into question one of the foundations of science" is made apparent in the last sentence of the passage:
If other such systems do exist, metaphorical examples of riddled basins of attraction may abound in the failed attempts of scientists to replicate previous experimental results--in which case, scientists would be forced to question one of the basic principles that guide their work.
In general, I'd recommend making some sort of notation on the passage whenever you detect the author's tone coming across. This might be the function of an entire paragraph, or it might come across from a single word (like "clearly...") in a sentence. I'd make some sort of "+/-" denotation so that you can quickly reference what the author's overall tone is. If you're unsure about the answers, hopefully that'll at least get you to sorting it into contenders and losers.
In terms of going for answers that are more reserved/less strong, I think you're generally best-advised to stick with the less strong answers. As you rightly note, there usually needs to be some "bold" language to warrant a strong answer choice. Sometimes, though--as here--it seems possible for an author to talk about, say, a scientific discovery, or mathematical theory and endorse it but in a limited sense. Again, the quoted language shows this limited sense: "If other such systems ego exist...scientists would be forced to question one of the basic principles that guide their work.
In sum, if you have to guess on a tone question, you're right to err on the side of caution---because strong language will hopefully be obvious, and because it's less likely that you'll come across a passage with strong tone on the LSAT (though such passages definitely do exist).
Hope those thoughts help!