Not sure if this is the correct way but I did it in the absence of any answer here.
Just connect to your server and delete the plugin folder (the one that you want to remove) from
and then re-compile using:
RAILS_ENV=production bundle exec rake assets:precompile
After this, pre-compile finished with no errors and plugin was removed.