This is not a bad idea. I tried something like this before. The issue that came up was that some players didn't understand it was a work in progress game even when told so. But if you find player types that understand how beta testing works, the pnp idea could work great for you to test mechanics.