@ArnoldNeumaier: Without the nonmeasurable sets, there just aren't thousands of little arguments to do regarding any ordinary probability stuff, after the set theoretic headaches are resolved, there's nothing difficult left. The construction of white noise, or of free fields, or models which are defined by a precise Nicolai map, or stochastic equation requiring no renormalization, or continuous limits of Levy variables (this is not done yet AFAIK), or defining Ising model measures, or whatever always excluding renormalization, is as straightforward as in elementary calculus. This simplification of ideas in intuitive probability is precisely why they put nonmeasurable sets on page one of any rigorous probability book, it's there to scare students away from intuitive probability, as mathematicians were scared in the 1920s, so that they think intuitive probability is logically inconsistent (false), just because it is inconsistent within ZFC (true).
Using the intuitive arguments, elementary probability is a piece of cake, which is why physicists were able to relatively easily and correctly develop sophisticated path-integral constructions of interest to pure mathematicians usually without ever taking a single course in measure theory. It's not because the physicists are using "heuristics" or "imprecise constructions", it's because the path integral belongs to intuitive probability, which is perfectly logically consistent, and mathematicians can't handle intuitive probability. The Axiom of Choice was just a gift from pure mathematics to the physics department, it means the mathematicians have to play catch-up for nearly 100 years.
The intuitive argument in probability go like this: for example, to "prove" that all subsets of [0,1] are measurable: consider a set S in [0,1] and sample [0,1] countably many times. For each sample, determine if it is in your set or not, and define the measure to be the limit of the number of generated random picks that land in S over the total number of picks. This argument is obvious, and it works intuitively and one feels that it "should" work as a rigorous argument in some way, but it can't be made rigorous in ZFC. It is circular in ZFC, because the definition of random variables is through measure theory, and the end result of translating the argument just becomes a justification for using the word "measure" as "probability". This non-argument then only works to show that measurable sets have the property that random variables land in them in proportion to their measure, or rather, more precisely (since only measurable sets can be "landed in", so the concept of "landing in" is not precise in ZFC) that the probability of the event of a random variables being an elements of a measurable set S is equal to the measure of S. But this random-picking argument is not circular (or at least not obviously circular) when the definition of random pick is by random forcing, and you have some sensible external framework in which to speak about adjoining new elements to R as you randomly generate them, and a good characterization of how to assign them membership to sets predefined by their generative properties (so that you understand what the "same interval" means in different models--- this is an essential part of Solovay's construction).
Continuing in this manner, if you have any way of taking countably many random variables (in a sensible intuitive probability universe) and forming a convergent sequence of distributions, you define a measure on any set of distributions which includes the image of the random variables with probability 1. The measure of any set X is just the probability that the construction ends up in X, which can be defined by doing it again and again, and asking what the fraction of throws that land in X is. This type of construction obviously doesn't qualify as a rigorous argument when there are nonmeasurable sets around, but it is completely precise when there aren't.
Each separate construction is one simple argument which requires no effort to remember or reproduce. It's like when you develop calculus, you don't need a separate argument with estimates for Riemann sums for the integral of 1/(x^2+1) as opposed to 1/x. You use a unified formalism.
For me, I don't care whether a construction is certified by a community as rigorous. I call a construction rigorous when there is a reasonable formal system which corresponds to some model of a mathematical universe, an equiconsistency proof of that system to a well-accepted one or a reasonable reflection of it, and a sketch of a proof that the formalization would go through in this system. The system I have in mind is not really Solovay's, because Solovay is complicated. But let's take Solovay's to start, because it works for this type of stuff, ZF+(dependent choice)+(Lebesgue measurability of R). Also it exists for sure, and is known to be equiconsistent with a well-established relatively intuitive large-cardinal hypothesis that is in no way controversial (possibly only because Grothendieck Universes are not controversial now, maybe because people get large cardinal hypotheses better, I don't know). I don't say this is the optimal model because Solovay's model is complicated to construct, and complicated to prove stuff about, because of a silly issue in the construction regarding collapsing the inaccessible, which is only required to be consistent with the axiom of power set.
I prefer a completely different system, which I just made up, which replaces the axiom of powerset with a different, to me more intuitive, thing. The basic idea is to make all countable models of ZFC (normal ZFC, including powerset), and speak about the universe as a container set theory for all these countable models, where the powersets are all proper classes containing all the powersets of all the different models. Then the notion of measure is for the class not for the sets (although there is a notion of measure on the sets also, as they are consistent with ZFC). The ambient theory has the axiom of choice, but it doesn't have an axiom of powerset, so it doesn't lead to measure paradoxes. It is just a theory of countable sets, so there is no issue with choice leading to nonmeasurable sets. The uncountabiltiy of R is the uncountability of R as a class, not as a set. I'll explain it in an answer to a self-answered question, but since I know it's not already out there, I want to run it past a logician that hates me first, because they might tear it apart correctly.