White Box and a Question of Dice

If I have one reservation about Swords & Wizardry White Box (and, by extension bare bones OD&D), it's the fact that there are fewer opportunities to use dice other than the d6 or the d20. This is a minor misgiving given that some of my favorite role-playing games use no other dice than six-siders (or in some cases percentile dice), but I have a number of players who are new to our beloved hobby, and they have an understandable urge to collect the polyhedral dice that are so intertwined with the popular concept of Dungeons & Dragons. I imagine they would like an excuse to use them for something other than stacking, but maybe I'm wrong.

So, why don't I move on from White Box and use the varied hit dice by class and varied damage dice that all other versions use? If only it were that simple. Reverting to d6 for damage and hit dice, you see, stabilizes a game that otherwise deteriorates into one of power inflation. At the moment, I am enjoying a game in which the players are motivated by what their characters want, and they are not obsessed with optimization or min-maxing like the average power gamer. They choose weapons according to what they imagine their characters would prefer, and the system doesn't penalize them for it. Varied hit dice by class has never made sense to me in a game in which monsters do not enjoy the same variety. Why should all monsters, regardless of size or ferocity, be limited to d8 hit dice? Should not monsters of a frailer, more cerebral nature be limited to d4 or d6 hit dice? Thankfully, White Box puts monsters and player characters on the same d6 playing field. Honestly, the game should go entirely one way or the other. I can imagine a workable system in which wee folk roll d4 hit dice, giants roll d12, and everything in between rolls d6, d8, or d10. In such a system, varied damage dice are completely justified. Perhaps one could even justify linking the hit die type to the damage die type for both character classes and monsters. Maybe this is a valid divergence for a possible variant of White Box (hm...), but shy of this, is there another way to bring the fun of rolling polyhedral dice back to White Box without destroying its delicate balance?

Random tables are always an option for referees, but this still leaves the question open for players (and I certainly don't want an overlay of new rules that require players to roll on lots of random tables). I am open to suggestions, but perhaps in the meantime I'll write up the variant rules and try it with a one-shot adventure.