I’ve heard people say that Rashi on Chumash is a semi-prophetic work. Rashi was Divinely inspired to comment on each verse and his commentary reflects God’s intended interpretation of the verses. This is a disservice to Rashi. It seems to me that Rashi saw inconsistencies or anomalies in the text and he chose his preferred interpretations from his vast knowledge of Chazal’s teachings. Rashi saw a problem and Rashi provided the solution.
I think it’s easier to see where this is taking us by looking at Ibn Ezra and Rambam. Why did the Ibn Ezra say that there are 7 places in the Chumash that are not original to the text? Was it something handed down to him from his teachers? I don’t think so. I am fairly certain that Ibn Ezra saw a problem in the text, there was evidence that contravened the accepted (or maybe it wasn’t accepted at that time) wisdom that every word of the Torah was given to Moses by God and those were the exact same words they had in the 11th century Torah scrolls. His solution was to accept the evidence and change the dogma. Religion cannot contradict truth so Ibn Ezra adjusted the religion accordingly.
Rambam managed to make the Torah consistent with Greek and Arabic philosophy. Do we think that Rambam thought that Greek and Arabic philosophy were handed to Moses from God at Sinai? That’s preposterous. Rather, Rambam held that the maxims he accepted in those non-Jewish sources were the truth. If they are true, then the Torah must work with Greek and Arabic philosophy. Rambam made it work. It’s the same idea. Rambam accepted external evidence and then conformed the Torah to make sense in the context of his truth. Truth cannot be changed. But sometimes religious dogma can be changed. Even more surprising is that Rambam did not say “prove Greek and Arabic philosophy to me and then I will work out the Torah but until then I accept the Torah as truth and all this secular stuff is obviously wrong.” He simply accepted the secular wisdom as true and conformed Torah to that truth.
This is how the rishonim and Chazal dealt with Torah vs. evidence.
Similarly, when both Chazal and rishonim encountered a verse or section that seemed too unlikely to be literal, they interpreted it metaphorically. We don’t circumcise our hearts because that verse was deemed allegorical. Rishonim, like Ralbag and Rambam went even further by turning sections that contradicted their common sense into dream-sequences or metaphors. Many rishonim and even some achronim have done the same thing with difficult passages in the Talmud. If it can’t be explained literally because of evidence then it could be allegorized. (Granted, some achronim felt no pangs of guilt by simply saying that Chazal were incorrect.)
The point to all this is to demonstrate how things have changed. The achronim lived in a time where science and other evidence was used to counter religion, not bolster it. In turn, they began the trend of lifting religion above the fray and arguing that the science or other evidence must be reinterpreted to fit religion. If it could not reinterpreted, it was wrong. This is the paradigm of almost all religions since the Renaissance and Enlightenment. There were exceptions to this rule, but for the most part this rule has held firm for several hundred years. Evolution doesn’t seem to fit in with Biblical narratives? Despite much more evidence than there is for the erroneous astronomy Rambam used in the Mishneh Torah, religious people all over the world are fighting against evolution. The Bible seems to be a composite work? Religious people reply that it’s impossible. Chazal seemed to be mistaken about medicine or science or math? Our modern thinking must be wrong, or alternatively the Talmud meant something else, or alternatively nature has changed.
It’s such a different attitude from the way Torah scholars approached evidence for almost 2000 years.
We don’t learn Chumash the same way the rishonim learned Chumash. Instead we learn what the rishonim said about the Chumash. This handicaps us from being too innovative on the one hand and has a way of maintaining tradition on the other hand.