Wget Direct to S3 with Golang Streams
Downloading a File
By creating an HTTP request, I get a basic request object that I can change other parameters of if I need to. This example sets an Authorization
header too, to give you an idea of how the other configuration works.
req, _ := http.NewRequest("GET", RecordingURL, nil) req.Header.Set("Authorization", "Bearer "+jwt) client := &http.Client{} resp, err := client.Do(req) if err != nil { panic(err) } defer resp.Body.Close()
We haven’t actually done anything with the request yet, but by deferring the close we can access the resp.Body
as a stream when we come to send the file over to S3.
Uploading to S3
It turns out that AWS already has a very nice SDK for working with Golang – it’s not as popular as some of the other SDKs so I didn’t find lots of code examples for working with it, but it was a very easy starting point! The SDK has a few ways to supply credentials but I like to use environment variables:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
With these credentials set, and the resp
from the section above to read from, I can upload the file to S3. This is a .mp3
file so the ContentType
is set accordingly – change this for other file types in your own applications.
format := "audio/mpeg" sess, err := session.NewSession(&aws.Config{ Region: aws.String("us-east-1")}, ) if err != nil { panic(err) } uploader := s3manager.NewUploader(sess) _, err = uploader.Upload(&s3manager.UploadInput{ Bucket: aws.String("my-bucket"), Key: aws.String("uploaded-file.mp3"), Body: resp.Body, ContentType: &format, })
I don’t need to do anything with the response here, but if there’s an error I do want to catch it and respond, so I’ve assigned err
in the example above.
Just a very quick and simple example but it took me a little while to line up all the pieces I needed so I thought I’d share what I did!
Also published on Medium.