Wget Direct to S3 with Golang Streams

One thing I find very elegant about working with golang is that it is very open minded about where data is flowing in from or going out to. I recently built an application that downloaded an audio file from one location and pushed it to s3 - and golang makes it very easy to do that without needing to write an intermediate file and then upload that. Here's my code, in case I need to do this again some day :)

Downloading a File

By creating an HTTP request, I get a basic request object that I can change other parameters of if I need to. This example sets an Authorization header too, to give you an idea of how the other configuration works.

	req, _ := http.NewRequest("GET", RecordingURL, nil)
	req.Header.Set("Authorization", "Bearer "+jwt)
	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()

We haven't actually done anything with the request yet, but by deferring the close we can access the resp.Body as a stream when we come to send the file over to S3.

Uploading to S3

It turns out that AWS already has a very nice SDK for working with Golang - it's not as popular as some of the other SDKs so I didn't find lots of code examples for working with it, but it was a very easy starting point! The SDK has a few ways to supply credentials but I like to use environment variables:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

With these credentials set, and the resp from the section above to read from, I can upload the file to S3. This is a .mp3 file so the ContentType is set accordingly - change this for other file types in your own applications.

		format := "audio/mpeg"
		sess, err := session.NewSession(&aws.Config{
			Region: aws.String("us-east-1")},
		)
		if err != nil {
			panic(err)
		}

		uploader := s3manager.NewUploader(sess)
		_, err = uploader.Upload(&s3manager.UploadInput{
			Bucket:      aws.String("my-bucket"),
			Key:         aws.String("uploaded-file.mp3"),
			Body:        resp.Body,
			ContentType: &format,
		})

I don't need to do anything with the response here, but if there's an error I do want to catch it and respond, so I've assigned err in the example above.

Just a very quick and simple example but it took me a little while to line up all the pieces I needed so I thought I'd share what I did!


Also published on Medium.

Leave a Reply

Please use [code] and [/code] around any source code you wish to share.

This site uses Akismet to reduce spam. Learn how your comment data is processed.