Installed for All Users instead of Admin Only

I recently installed for all users in a sandbox but.  My thought was the permission sets would control who ultimately could perform Accounting Seed functions.  But I see that users without AS permission sets can still create Sales Order/Purchase Orders etc.  

How can I remove those permissions without having to uninstall/re-install?  

Is it just removing object permissions?  Seems like it could be a security gap.