Wget Direct to S3 with Golang Streams

One thing I find very elegant about working with golang is that it is very open minded about where data is flowing in from or going out to. I recently built an application that downloaded an audio file from one location and pushed it to s3 – and golang makes it very easy to do that without needing to write an intermediate file and then upload that. Here’s my code, in case I need to do this again some day :)

Downloading a File

By creating an HTTP request, I get a basic request object that I can change other parameters of if I need to. This example sets an Authorization header too, to give you an idea of how the other configuration works.

    req, _ := http.NewRequest("GET", RecordingURL, nil)
    req.Header.Set("Authorization", "Bearer "+jwt)
    client := &http.Client{}
    resp, err := client.Do(req)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

We haven’t actually done anything with the request yet, but by deferring the close we can access the resp.Body as a stream when we come to send the file over to S3.

Uploading to S3

It turns out that AWS already has a very nice SDK for working with Golang – it’s not as popular as some of the other SDKs so I didn’t find lots of code examples for working with it, but it was a very easy starting point! The SDK has a few ways to supply credentials but I like to use environment variables:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

With these credentials set, and the resp from the section above to read from, I can upload the file to S3. This is a .mp3 file so the ContentType is set accordingly – change this for other file types in your own applications.

        format := "audio/mpeg"
        sess, err := session.NewSession(&aws.Config{
            Region: aws.String("us-east-1")},
        )
        if err != nil {
            panic(err)
        }

        uploader := s3manager.NewUploader(sess)
        _, err = uploader.Upload(&s3manager.UploadInput{
            Bucket:      aws.String("my-bucket"),
            Key:         aws.String("uploaded-file.mp3"),
            Body:        resp.Body,
            ContentType: &format,
        })

I don’t need to do anything with the response here, but if there’s an error I do want to catch it and respond, so I’ve assigned err in the example above.

Just a very quick and simple example but it took me a little while to line up all the pieces I needed so I thought I’d share what I did!

Leave a Reply

Please use [code] and [/code] around any source code you wish to share.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

To respond on your own website, enter the URL of your response which should contain a link to this post's permalink URL. Your response will then appear (possibly after moderation) on this page. Want to update or remove your response? Update or delete your post and re-enter your post's URL again. (Find out more about Webmentions.)