For others reading this in the future, the guide referenced for configuring WordPress with AWS storage is based around the WP Offload Media plugin, which by default stores the media for all the multisite sites in one S3 bucket. There is a way to separate different sites into different buckets (which I think is what I would prefer) but it requires editing a plugin php file to implement "filters". I haven't looked into the details of this yet, but I think a search and replace in the database might be more straightforward. The search and replace would not deal with subsequent media uploads. I think my approach to that might be to upload directly to the S3 bucket and then link to the new media with a URL on the WordPress page.
Also, note that links to media in WordPress multisite look like this:
17 in the above URL is the site ID.)
So if you just replace to domain name in the database, then the S3 bucket folder structure has to duplicate the