The deferring should not affect the layout because what is does, as it is said in the link @andresrc provided, is changing when the parsing of JS starts. Maybe this could affect your WP theme in someway, but not in general.
About you question about browser and server-side caching, it is a quite broad subject I recommend you to do some research about it.
As general idea, caching is having prepared whatever it is, instead of having to generate it under demand. So for browser this means to store a copy of, for example, and image, and don't request it again to the server until some conditions are met. This will save time and transferring data.
In the server side, caching is for example, having the html rendered stored. This way, when a new request reach the server, it doesn't have to build it again. In WP, to generate the html returned to the server, many PHP files need to be executed and many queries to the BD server needs to be done. In this case, it saves time and CPU.
So, as you can see, this is a complex topic.
The browser do caching automatically, for the server side you can have a look at varnish. You can use this guide to configure varnish: https://docs.bitnami.com/general/apps/wordpress/administration/configure-use-varnish/ . Have into account that this won't help in your current problem.
Other thing you can do without all that hassle is tuning your page, revisiting the stats from GTmetrix:
I think you can use the wdt-emoji-concat.min.js, the recaptcha__es.js only in the pages you really need them. That would save like 80% of all the js of the page.
Rafael Rios Saavedra