I don't think I'd worry about that from a radiation standpoint. Natural uranium is only around 1000 times more radioactive than natural potassium, it's really not very bad in that sense and it wouldn't have survived from the birth of the solar system to the present day if it was.
On the other hand, it is a toxic heavy metal so please, please don't eat it.
And you'll get a lot more of that potassium into your body than uranium. So long as you don't eat such low-level things they're not a threat. (And, personally, I'm not convinced there's any threat. All of our estimates of the threat of radioactivity are based on high level exposures. IIRC it's about 100 Sieverts = 1 death. We project that linearly into the lose dose realms that are effectively impossible to test, but we have some data points that suggest the linear projection is not accurate and low level exposure is of much lower or possibly even negative threat--that the body's reaction to the low level radioactivity actually is protective. There is no relationship between cancer and background radioactivity. The threat of radon noted in uranium miners only shows up if they're also smokers--huh??)
On the other hand, it is a toxic heavy metal so please, please don't eat it.