In this post, I outline a simple, easy to understand implementation for two components of a Redis client in Go as a way of understanding how the Redis protocol works and what makes it great.
If you’re looking for a full-featured, production-ready Redis client in Go, we recommend taking a look at Gary Burd’s redigo library.
Before we get started, be sure you read our gentle introduction to the Redis protocol - it covers the basics of the protocol that you’ll need to understand for this guide.
A RESP command writer in Go
For our hypothetical Redis client, there’s only one kind of object that we’ll need to write: an array of bulk strings for sending commands to Redis. Here is a simple implementation of a command-to-RESP writer:
package redis
import (
"bufio"
"io"
"strconv" // for converting integers to strings
)
var (
arrayPrefixSlice = []byte{'*'}
bulkStringPrefixSlice = []byte{'$'}
lineEndingSlice = []byte{'\r', '\n'}
)
type RESPWriter struct {
*bufio.Writer
}
func NewRESPWriter(writer io.Writer) *RESPWriter {
return &RESPWriter{
Writer: bufio.NewWriter(writer),
}
}
func (w *RESPWriter) WriteCommand(args ...string) (err error) {
// Write the array prefix and the number of arguments in the array.
w.Write(arrayPrefixSlice)
w.WriteString(strconv.Itoa(len(args)))
w.Write(lineEndingSlice)
// Write a bulk string for each argument.
for _, arg := range args {
w.Write(bulkStringPrefixSlice)
w.WriteString(strconv.Itoa(len(arg)))
w.Write(lineEndingSlice)
w.WriteString(arg)
w.Write(lineEndingSlice)
}
return w.Flush()
}
Rather than writing to a net.Conn
object, RESPWriter
writes to a io.Writer
object. This allows us to test our parser without tightly coupling to the net
stack. We simply test the network protocol the way we would any other io
.
For example, we can pass it a bytes.Buffer
to inspect the final RESP:
var buf bytes.Buffer
writer := NewRESPWriter(&buf)
writer.WriteCommand("GET", "foo")
buf.Bytes() // *2\r\n$3\r\nGET\r\n$3\r\nfoo\r\n
A simple RESP reader in Go
After sending a command to Redis with RESPWriter
, our client would use
RESPReader
to read from the TCP connection until it has received a full RESP
reply. To start with, we’ll need a few packages to handle buffering and parsing
the incoming data:
package redis
import (
"bufio"
"bytes"
"errors"
"io"
"strconv"
)
And we’ll use a few variables and constants to make our code a little easier to read:
const (
SIMPLE_STRING = '+'
BULK_STRING = '$'
INTEGER = ':'
ARRAY = '*'
ERROR = '-'
)
var (
ErrInvalidSyntax = errors.New("resp: invalid syntax")
)
Like RESPWriter
, RESPReader
doesn’t care about the implementation details of
the object that it’s reading RESP from. All it needs the ability to
read bytes until it has read a full RESP object. In this case, it
needs an io.Reader
, which it wraps with a bufio.Reader
to handle the
buffering of the incoming data.
Our object and initializer are simple:
type RESPReader struct {
*bufio.Reader
}
func NewReader(reader io.Reader) *RESPReader {
return &RESPReader{
Reader: bufio.NewReaderSize(reader, 32*1024),
}
}
The buffer size for bufio.Reader
is just a guess during development. In an
actual client, you’d want to make its size configurable and perhaps test to find
the optimal size. 32KB will work fine for development.
RESPReader
has only one method: ReadObject()
, which returns a
byte slice containing a full RESP object on each call. It will pass back any
errors encountered from io.Reader
, and will also return errors when it
encounters any invalid RESP syntax.
The prefix nature of RESP means we only need to read the first byte to decide
how to handle the following bytes. However, because we’ll always need to read
at least the first full line (i.e. up until the first \r\n
), we can start by
reading the whole first line:
func (r *RESPReader) ReadObject() ([]byte, error) {
line, err := r.readLine()
if err != nil {
return nil, err
}
switch line[0] {
case SIMPLE_STRING, INTEGER, ERROR:
return line, nil
case BULK_STRING:
return r.readBulkString(line)
case ARRAY:
return r.readArray(line)
default:
return nil, ErrInvalidSyntax
}
}
When the line that we read has a simple string, integer, or error prefix, we return the full line as the received RESP object because those object types are contained entirely within one line.
In readLine()
, we read
up until the first occurrence of \n
and then check to make sure that it was
preceded by a \r
before returning the line as a byte slice:
func (r *RESPReader) readLine() (line []byte, err error) {
line, err = r.ReadSlice('\n')
if err != nil {
return nil, err
}
if len(line) > 1 && line[len(line)-2] == '\r' {
return line, nil
} else {
// Line was too short or \n wasn't preceded by \r.
return nil, ErrInvalidSyntax
}
}
In readBulkString()
we parse the length
specification for the bulk string to know how many bytes we need
to read. Once we do, we read that count of bytes and
the \r\n
line terminator:
func (r *RESPReader) readBulkString(line []byte) ([]byte, error) {
count, err := r.getCount(line)
if err != nil {
return nil, err
}
if count == -1 {
return line, nil
}
buf := make([]byte, len(line)+count+2)
copy(buf, line)
_, err = r.Read(buf[len(line):])
if err != nil {
return nil, err
}
return buf, nil
}
I’ve pulled getCount()
out to a separate method because
the length specification is also used for arrays:
func (r *RESPReader) getCount(line []byte) (int, error) {
end := bytes.IndexByte(line, '\r')
return strconv.Atoi(string(line[1:end]))
}
To handle arrays, we get the number of
array elements, and then call ReadObject()
recursively, adding the resulting
objects to our current RESP buffer:
func (r *RESPReader) readArray(line []byte) ([]byte, error) {
// Get number of array elements.
count, err := r.getCount(line)
if err != nil {
return nil, err
}
// Read `count` number of RESP objects in the array.
for i := 0; i < count; i++ {
buf, err := r.ReadObject()
if err != nil {
return nil, err
}
line = append(line, buf...)
}
return line, nil
}
Wrapping up
The above hundred lines are all that’s needed to read any RESP object from Redis. However, there are a number of missing pieces we’d need to implement before using this library in a production environment:
- The ability to extract actual values from the RESP.
RESPReader
currently only returns the full RESP response, it does not, for example, return a string from a bulk string response. However, implementing this would be easy. RESPReader
needs better syntax error handling.
This code is also entirely unoptimized and does more allocations and copies than
it needs to. For example, the readArray()
method: for each
object in the array, we read in the object and then copies it to our local
buffer.
If you’re interested in learning how to implement these pieces, I recommend looking at how popular libraries like hiredis or redigo implement them.
Ready to get started with a production-ready Redis host backed by excellent support? Get started now with your own RedisGreen server.
有疑问加站长微信联系(非本文作者)