Skip to main content

Copy from a Reader to a Writer

There are many ways of passing data from a Reader to a Writer, and we'll go through many of them in these chapters.

For this chapter I want only to focus on directly copying data from a Reader to Writer with the help of Copy and CopyBuffer functions from the io package.

io.Copy

To copy from a Reader to a Writer we can use io.Copy function:

func Copy(dst Writer, src Reader) (written int64, err error)

Copy copies from src to dst until either EOF is reached on src or an error occurs. It returns the number of bytes copied and the first error encountered while copying, if any as the docs say.

EOF is the error returned by Read when no more input is available meaning that you read everything from the src Reader.

In the bellow example we have a strings.Reader created from which we can read. We will read all the bytes on the os.Stdout which is just an open os.File and thus an io.Reader as well because the os.File implements the io.Reader interface.

package main

import (
"io"
"log"
"os"
"strings"
)

func main() {
r := strings.NewReader("some io.Reader stream to be read\n")

if _, err := io.Copy(os.Stdout, r); err != nil {
log.Fatal(err)
}
}

run in Playground

io.CopyBuffer

Another way to copy from a Reader to a Writer is to use an almost identical function io.CopyBuffer:

func CopyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error)

CopyBuffer is identical to Copy except that it stages through the provided buffer (if one is required) rather than allocating a temporary one. If buf is nil, one is allocated; otherwise if it has zero length, CopyBuffer panics. [from docs]

Let's see what that actually means:

CopyBuffer is identical to Copy

This is the implementation of io.Copy and io.CopyBuffer from standard library:

func Copy(dst Writer, src Reader) (written int64, err error) {
return copyBuffer(dst, src, nil)
}

func CopyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error) {
if buf != nil && len(buf) == 0 {
panic("empty buffer in CopyBuffer")
}
return copyBuffer(dst, src, buf)
}

As you can see both use the copyBuffer function under the hood. For io.Copy the buf is nil while io.CopyBuffer uses the buf provided by the user.

If buf is nil, one is allocated;

If we go inside copyBuffer we see this part:

if buf == nil {
size := 32 * 1024
...
buf = make([]byte, size)

So we can see that in the case of io.Copy, where the buf is nil, an internal one is allocated with a size of 32K bytes.

So both io.Copy and io.CopyBuffer use a buffer. Just that io.Copy uses a default buffer of 32k bytes and io.CopyBuffer uses a buffer provided by the user.

When to use io.CopyBuffer?

You might ask yourself what are the benefits of using io.CopyBuffer?

I think by default you can easily go with io.Copy. There are some benefits of using io.CopyBuffer:

  • choose your buffer size. If you know the data you are copying is larger than 32k bytes, you can allocate a bigger buffer. With a bigger buffer you could get the benefit of less writes. This goes the other way around. If you know you have data less than 32k bytes, you can use a custom buffer with fewer bytes getting the benefit of allocating less memory.
  • re-use buffer. If you have multiple io.CopyBuffer calls, you could use only one buffer for all. An example for that is shown in the docs:
package main

import (
"io"
"log"
"os"
"strings"
)

func main() {
r1 := strings.NewReader("first reader\n")
r2 := strings.NewReader("second reader\n")
buf := make([]byte, 8)

// buf is used here...
if _, err := io.CopyBuffer(os.Stdout, r1, buf); err != nil {
log.Fatal(err)
}

// ... reused here also. No need to allocate an extra buffer.
if _, err := io.CopyBuffer(os.Stdout, r2, buf); err != nil {
log.Fatal(err)
}

}

Conclusion

That's all for today! Now you know how to copy data from a Reader to a Writer using io.Copy or io.CopyBuffer. We're only scratching the surface here. There are many Readers and Writers to try out, so just bear with me.

However, you should give yourself a pat on the back! If you followed along so far, you should know how to read from a Reader, write to a Writer and copy from a Reader to a Writer. Those are not small things, you're getting dangerous, so be careful!