The WP Offload Media plugin replaces the URL to each media file with the one from the S3 bucket. Alternatively it's possible to use a Cloudfront URL if you have that configured, but it's optional, not required.
For cost related questions I recommend you to check specialized forums. I found the following thread regarding this issue:
Please, click on if you think my answer was helpful