I'm just warning you the fluorite is going to require really going into something as I don't think it survives long enough to be something you can find while dicking around by the right river cut. Gypsum doesn't last long either.
Beware that messing around in a quarry even if it's "just a sand quarry" will get you in serious trouble since the petroleum industry relies and certain grain sizes and purity that Illinois has, and the concrete business is not another group of companies to be love around with. Also be careful. Trespassing will get you shot in certain areas of the state. Since Chicago is fiercely anti-gun (for good reason) the rest of the state tries to counter-balance this with what I'll call "excessive use" of pro-guns laws. To give you idea shooting an 8 year picking flowers that wouldn't leave someones property was a "celebration of American freedom" a few years back around my college since they're all nuts.
As far as more "interesting" stuff you're better off poking around in Wisconsin. Illinois, unless you're into paleo, is a contender for the most boring geologic state in the USA and probably high in the running worldwide (seriously).
If you like fossils look up the "Mazon Creek" lagerstatten (sp?). It's really the only non-fossil fuel worthwhile thing in the state if you're interested in the thing. They've found a ton of what I'll call "connector fossils" there since the conditions were right to remove the hard bodied preservation bias commonly found so you can see the soft parts of some animals.
"Careful man, there's a beverage here!"