Dll failed to uninstall need restart Visual Studio to finish the process

I had suffered a lot with this error like Xamarin.Forms2.x failed to uninstall. Restart Visual Studio to finish the process. In this case I have found a solution that helped me a lot:

Instead of deleting all of ~/packages, see if there are any *.deleteme files in ~/packages and delete them. Then restart Visual Studio.

This is a simple solution but potential one.