这篇blog将给大家介绍在go语言项目使用Redis作为数据持久层。开始之前我将介绍几个基本概念，然后构建一个go web项目来展示如何让Redis实现安全的并发。
这篇blog是假设你是了解Redis的一些东西的，如果你之前没有学习或者了解过Redis。我建议你先去阅读一些Redis的知识再来看这篇blog效果更好。
安装驱动
第一步我们需要安装Redis的go语言驱动程序，大家可以在这个列表中选择一个http://redis.io/clients#go
这篇blog的示例我还是选择了Radix.v2作为驱动程序。因为它维护的很好，API用起来也很舒服，如果你也想用的话 可以在终端执行下面的命令：
|
1 |
$ go get github.com/mediocregopher/radix.v2 |
有一点要注意的是Radix.v2的包被分为了(cluster, pool, pubsub,redis, sentinel and util)6个子包。最开始我们只需要使用其中的redis包。
|
1 2 3 |
import ( "github.com/mediocregopher/radix.v2/redis" ) |
开始使用 Radix.v2
作为示例程序，我们为它设定一个场景也就是所谓的需求。假设我们有一个在线的唱片店，并希望存储有关的专辑的销售信息存储在Redis中。我们可以有很多不同的方式来设计数据模式。我希望我们的model是简单的并且通过hash的方式来存储我们的专辑信息包括标题，艺术家和价格之类的字段。
我们可以通过Redis的CLI 执行HMSET命令：
|
1 2 |
127.0.0.1:6379> HMSET album:1 title "Electric Ladyland" artist "Jimi Hendrix" price 4.95 likes 8 OK |
如果我们在Go项目中来做同样的操作，我们就需要通过Radix.v2 redis包来实现，首先需要Dial()函数，需要它返回一个新的connection。第二个我们需要使用client.Cmd()方法。通过他我们可以发送一个命令到我们的服务器，这里会返回给我们一个Resp对象的指针。
我们来看一个简单的示例：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
File: main.go package main import ( "fmt" // Import the Radix.v2 redis package. "github.com/mediocregopher/radix.v2/redis" "log" ) func main() { // Establish a connection to the Redis server listening on port 6379 of the // local machine. 6379 is the default port, so unless you've already // changed the Redis configuration file this should work. conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } // Importantly, use defer to ensure the connection is always properly // closed before exiting the main() function. defer conn.Close() // Send our command across the connection. The first parameter to Cmd() // is always the name of the Redis command (in this example HMSET), // optionally followed by any necessary arguments (in this example the // key, followed by the various hash fields and values). resp := conn.Cmd("HMSET", "album:1", "title", "Electric Ladyland", "artist", "Jimi Hendrix", "price", 4.95, "likes", 8) // Check the Err field of the *Resp object for any errors. if resp.Err != nil { log.Fatal(resp.Err) } fmt.Println("Electric Ladyland added!") } |
在这个示例中我们所关心的并不是Redis会返回什么。因为所有成功的操作都会返回一个“OK”的字符串。所以我们不需要对*Resp对象做错误检查。
在这样的情况下我们只需要检查err就好了：
|
1 2 3 4 |
err = conn.Cmd("HMSET", "album:1", "title", "Electric Ladyland", "artist", "Jimi Hendrix", "price", 4.95, "likes", 8).Err if err != nil { log.Fatal(err) } |
回复处理
如果我们留意Redis的回复的话，我们发现Resp对象有很多有用的函数让Go的类型转换变得容易：
- Resp.Bytes() – converts a single reply to a byte slice (
[]byte) - Resp.Float64() – converts a single reply to a
Float64 - Resp.Int() – converts a single reply to a
int - Resp.Int64() – converts a single reply to a
int64 - Resp.Str() – converts a single reply to a
string
- Resp.Array() – converts an array reply to an slice of individual
Respobjects([]*Resp) - Resp.List() – converts an array reply to an slice of strings
([]string) - Resp.ListBytes() – converts an array reply to an slice of byte slices
([][]byte) - Resp.Map() – converts an array reply to a map of strings, using each item in the array reply alternately as the keys and values for the map
(map[string]string)
接下来我们使用HGET 命令获取一些专辑的数据：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
package main import ( "fmt" "github.com/mediocregopher/radix.v2/redis" "log" ) func main() { conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } defer conn.Close() // Issue a HGET command to retrieve the title for a specific album, and use // the Str() helper method to convert the reply to a string. title, err := conn.Cmd("HGET", "album:1", "title").Str() if err != nil { log.Fatal(err) } // Similarly, get the artist and convert it to a string. artist, err := conn.Cmd("HGET", "album:1", "artist").Str() if err != nil { log.Fatal(err) } // And the price as a float64... price, err := conn.Cmd("HGET", "album:1", "price").Float64() if err != nil { log.Fatal(err) } // And the number of likes as an integer. likes, err := conn.Cmd("HGET", "album:1", "likes").Int() if err != nil { log.Fatal(err) } fmt.Printf("%s by %s: £%.2f [%d likes]\n", title, artist, price, likes) } |
值得指出的是，当我们使用这些方法的时候，这时候的错误返回会设置一种或者2种错误：一种是任一命令执行失败，或者返回数据转换成了我们需要要的类型。这样的话我们就不知道错误是哪一种了？除非我们检查错误消息的方式。
如果我们执行代码我们将会看到：
|
1 2 |
$ go run main.go Electric Ladyland by Jimi Hendrix: £4.95 [8 likes] |
现在让我们来看一个更完整的例子，我们使用HGETALL命令来获取专辑的信息：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
File: main.go package main import ( "fmt" "github.com/mediocregopher/radix.v2/redis" "log" "strconv" ) // Define a custom struct to hold Album data. type Album struct { Title string Artist string Price float64 Likes int } func main() { conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } defer conn.Close() // Fetch all album fields with the HGETALL command. Because HGETALL // returns an array reply, and because the underlying data structure in // Redis is a hash, it makes sense to use the Map() helper function to // convert the reply to a map[string]string. reply, err := conn.Cmd("HGETALL", "album:1").Map() if err != nil { log.Fatal(err) } // Use the populateAlbum helper function to create a new Album object from // the map[string]string. ab, err := populateAlbum(reply) if err != nil { log.Fatal(err) } fmt.Println(ab) } // Create, populate and return a pointer to a new Album struct, based on data // from a map[string]string. func populateAlbum(reply map[string]string) (*Album, error) { var err error ab := new(Album) ab.Title = reply["title"] ab.Artist = reply["artist"] // We need to use the strconv package to convert the 'price' value from a // string to a float64 before assigning it. ab.Price, err = strconv.ParseFloat(reply["price"], 64) if err != nil { return nil, err } // Similarly, we need to convert the 'likes' value from a string to an // integer. ab.Likes, err = strconv.Atoi(reply["likes"]) if err != nil { return nil, err } return ab, nil } |
我们执行代码将会看到下面的输出：
|
1 2 |
$ go run main.go &{Electric Ladyland Jimi Hendrix 4.95 8} |
在Web项目中使用
一个重要事情就是我们知道什么是并发不安全的使用Radix.v2中的redis包。
如果我们要访问一个Redis服务器而通过多个goroutines，那我我们在web项目中就需要使用pool包来替换redis。这将是我们能够建立连接池，每次使用连接我们只需要从连接池中获取就可以了。
执行命令，然后返回到连接池中。
简单的介绍下我们的Web项目的结构：
| Method | Path | Function |
|---|---|---|
| GET | /album?id=1 | Show details of a specific album (using the id provided in the query string) |
| POST | /like | Add a new like for a specific album (using the id provided in the request body) |
| GET | /popular | List the top 3 most liked albums in order |
如果你想和我的结构一样的话你可以跟着我下面的命令操作：
|
1 2 3 4 5 6 7 8 9 |
$ cd $GOPATH/src $ mkdir -p recordstore/models $ cd recordstore $ touch main.go models/albums.go $ tree . ├── main.go └── models └── albums.go |
然后我们通过Redis CLI 保存一些专辑数据，这样我们等下就有测试数据了。
|
1 2 3 4 5 |
HMSET album:1 title "Electric Ladyland" artist "Jimi Hendrix" price 4.95 likes 8 HMSET album:2 title "Back in Black" artist "AC/DC" price 5.95 likes 3 HMSET album:3 title "Rumours" artist "Fleetwood Mac" price 7.95 likes 12 HMSET album:4 title "Nevermind" artist "Nirvana" price 5.95 likes 8 ZADD likes 8 1 3 2 12 3 8 4 |
我们将遵循MVC的模式来构建我们的项目，我们使用models/albums.go文件来实现我们Redis的逻辑处理，在models/albums.go文件中我们将使用init()函数来建立启动我们的Redis连接池。然后我们也会重构之前我们写的FindAlbum()函数。然后我们我们通过HTTP handlers来使用它：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 |
File: models/albums.go package models import ( "errors" // Import the Radix.v2 pool package, NOT the redis package. "github.com/mediocregopher/radix.v2/pool" "log" "strconv" ) // Declare a global db variable to store the Redis connection pool. var db *pool.Pool func init() { var err error // Establish a pool of 10 connections to the Redis server listening on // port 6379 of the local machine. db, err = pool.New("tcp", "localhost:6379", 10) if err != nil { log.Panic(err) } } // Create a new error message and store it as a constant. We'll use this // error later if the FindAlbum() function fails to find an album with a // specific id. var ErrNoAlbum = errors.New("models: no album found") type Album struct { Title string Artist string Price float64 Likes int } func populateAlbum(reply map[string]string) (*Album, error) { var err error ab := new(Album) ab.Title = reply["title"] ab.Artist = reply["artist"] ab.Price, err = strconv.ParseFloat(reply["price"], 64) if err != nil { return nil, err } ab.Likes, err = strconv.Atoi(reply["likes"]) if err != nil { return nil, err } return ab, nil } func FindAlbum(id string) (*Album, error) { // Use the connection pool's Get() method to fetch a single Redis // connection from the pool. conn, err := db.Get() if err != nil { return nil, err } // Importantly, use defer and the connection pool's Put() method to ensure // that the connection is always put back in the pool before FindAlbum() // exits. defer db.Put(conn) // Fetch the details of a specific album. If no album is found with the // given id, the map[string]string returned by the Map() helper method // will be empty. So we can simply check whether it's length is zero and // return an ErrNoAlbum message if necessary. reply, err := conn.Cmd("HGETALL", "album:"+id).Map() if err != nil { return nil, err } else if len(reply) == 0 { return nil, ErrNoAlbum } return populateAlbum(reply) } |
值得我们着重讲解一下的是pool.New()函数。在上面的代码中我们指定连接池的大小为10，它只是空闲时候在池中等待的数量。如果1o个连接都使用了的话，那时候将会调用 pool.Get()创建新的链接。
当你只有一个连接发出命令的话，比如上面的FindAlbum()函数，有可能会使用pool.Cmd()的快捷方式。这将自动从连接池中获取一个新的连接，执行给定的命令、然后将连接返回连接池。
这是我们重构了之后的FindAlbum()函数：
|
1 2 3 4 5 6 7 8 9 10 |
func FindAlbum(id string) (*Album, error) { reply, err := db.Cmd("HGETALL", "album:"+id).Map() if err != nil { return nil, err } else if len(reply) == 0 { return nil, ErrNoAlbum } return populateAlbum(reply) } |
好了，让我们来看看我们的main.go的文件如何实现的：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
File: main.go package main import ( "fmt" "net/http" "recordstore/models" "strconv" ) func main() { // Use the showAlbum handler for all requests with a URL path beginning // '/album'. http.HandleFunc("/album", showAlbum) http.ListenAndServe(":3000", nil) } func showAlbum(w http.ResponseWriter, r *http.Request) { // Unless the request is using the GET method, return a 405 'Method Not // Allowed' response. if r.Method != "GET" { w.Header().Set("Allow", "GET") http.Error(w, http.StatusText(405), 405) return } // Retrieve the id from the request URL query string. If there is no id // key in the query string then Get() will return an empty string. We // check for this, returning a 400 Bad Request response if it's missing. id := r.URL.Query().Get("id") if id == "" { http.Error(w, http.StatusText(400), 400) return } // Validate that the id is a valid integer by trying to convert it, // returning a 400 Bad Request response if the conversion fails. if _, err := strconv.Atoi(id); err != nil { http.Error(w, http.StatusText(400), 400) return } // Call the FindAlbum() function passing in the user-provided id. If // there's no matching album found, return a 404 Not Found response. In // the event of any other errors, return a 500 Internal Server Error // response. bk, err := models.FindAlbum(id) if err == models.ErrNoAlbum { http.NotFound(w, r) return } else if err != nil { http.Error(w, http.StatusText(500), 500) return } // Write the album details as plain text to the client. fmt.Fprintf(w, "%s by %s: £%.2f [%d likes] \n", bk.Title, bk.Artist, bk.Price, bk.Likes) } |
执行我们的代码：
|
1 |
$ go run main.go |
通过cURL来测试我们的请求，我们也可以通过postman之类的工具：
|
1 2 3 4 5 6 |
$ curl -i localhost:3000/album?id=2 HTTP/1.1 200 OK Content-Length: 42 Content-Type: text/plain; charset=utf-8 Back in Black by AC/DC: £5.95 [3 likes] |
使用事务
我们的第二个路由POST /likes
当我们的用户如果喜欢（点赞）我们的专辑的时候，我们需要执行2条不同的命令：使用HINCRBY来增加likes字段的值，和使用ZINCRBY来增加相应的score在我们的likes的有序集合中。
那么这里就会产生一个问题。理想情况下，我们希望这两个键在完全相同的时间递增作为一个原子操作，当一个key完成更新后，其它数据不会与其发生争抢。
解决这个问题的办法就需要我们使用Redis事务，这让我们运行多个命令在一起作为一个原子团。我们可以使用MULTI命令来开启事务，随后执行我们的HINCRBY 和ZINCRBY，最后执行EXEC命令
让我们在model中创建一个新的IncrementLikes()函数：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
File: models/albums.go ... func IncrementLikes(id string) error { conn, err := db.Get() if err != nil { return err } defer db.Put(conn) // Before we do anything else, check that an album with the given id // exists. The EXISTS command returns 1 if a specific key exists // in the database, and 0 if it doesn't. exists, err := conn.Cmd("EXISTS", "album:"+id).Int() if err != nil { return err } else if exists == 0 { return ErrNoAlbum } // Use the MULTI command to inform Redis that we are starting a new // transaction. err = conn.Cmd("MULTI").Err if err != nil { return err } // Increment the number of likes in the album hash by 1. Because it // follows a MULTI command, this HINCRBY command is NOT executed but // it is QUEUED as part of the transaction. We still need to check // the reply's Err field at this point in case there was a problem // queueing the command. err = conn.Cmd("HINCRBY", "album:"+id, "likes", 1).Err if err != nil { return err } // And we do the same with the increment on our sorted set. err = conn.Cmd("ZINCRBY", "likes", 1, id).Err if err != nil { return err } // Execute both commands in our transaction together as an atomic group. // EXEC returns the replies from both commands as an array reply but, // because we're not interested in either reply in this example, it // suffices to simply check the reply's Err field for any errors. err = conn.Cmd("EXEC").Err if err != nil { return err } return nil } |
我们也需要在main.go文件中为route添加一个addLike()handler：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 |
File: main.go func main() { http.HandleFunc("/album", showAlbum) http.HandleFunc("/like", addLike) http.ListenAndServe(":3000", nil) } ... func addLike(w http.ResponseWriter, r *http.Request) { // Unless the request is using the POST method, return a 405 'Method Not // Allowed' response. if r.Method != "POST" { w.Header().Set("Allow", "POST") http.Error(w, http.StatusText(405), 405) return } // Retreive the id from the POST request body. If there is no parameter // named "id" in the request body then PostFormValue() will return an // empty string. We check for this, returning a 400 Bad Request response // if it's missing. id := r.PostFormValue("id") if id == "" { http.Error(w, http.StatusText(400), 400) return } // Validate that the id is a valid integer by trying to convert it, // returning a 400 Bad Request response if the conversion fails. if _, err := strconv.Atoi(id); err != nil { http.Error(w, http.StatusText(400), 400) return } // Call the IncrementLikes() function passing in the user-provided id. If // there's no album found with that id, return a 404 Not Found response. // In the event of any other errors, return a 500 Internal Server Error // response. err := models.IncrementLikes(id) if err == models.ErrNoAlbum { http.NotFound(w, r) return } else if err != nil { http.Error(w, http.StatusText(500), 500) return } // Redirect the client to the GET /ablum route, so they can see the // impact their like has had. http.Redirect(w, r, "/album?id="+id, 303) } |
和之前一样我们来测试一下：
|
1 2 3 4 5 6 7 8 9 10 11 12 |
$ curl -i -L -d "id=2" localhost:3000/like HTTP/1.1 303 See Other Location: /album?id=2 Date: Thu, 25 Feb 2016 17:08:19 GMT Content-Length: 0 Content-Type: text/plain; charset=utf-8 HTTP/1.1 200 OK Content-Length: 42 Content-Type: text/plain; charset=utf-8 Back in Black by AC/DC: £5.95 [4 likes] |
使用Watch命令
好了还剩下我们最后一个route: GET /popular，这个route将会现实我们最受欢迎的3个专辑的详细内容，我们将在models/albums.go中创建FindTopThree()函数，这这个函数中我们需要：
1.使用ZREVRANGE命令，去获取3条专家的最高分，从我们的likes字段的有序集合中
2.通过HGETALL命令循环获取存储在键的散列的所有字段和值，把它们存到[]*Album slice中
这里同样也可能会发送数据竞争的情况，如果第二个第二个客户端也碰巧需要只想我们刚刚的操作，我们用户可能会得到不正确的数据。
这个问题的解决方法就是使用Redis WATCH命令与事务结合使用。WATCH可以让Redis监控某个key的变化，如果其它客户端要更改我们监控的key在我们执行EXEC之前，那么事务将会失败并且会返回Nil。如果没有其它客户端在我们执行EXEC命令之前修改我们的数据。那么我的操作将正常执行：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 |
File: models/albums.go package models import ( "errors" "github.com/mediocregopher/radix.v2/pool" // Import the Radix.v2 redis package (we need access to its Nil type). "github.com/mediocregopher/radix.v2/redis" "log" "strconv" ) ... func FindTopThree() ([]*Album, error) { conn, err := db.Get() if err != nil { return nil, err } defer db.Put(conn) // Begin an infinite loop. for { // Instruct Redis to watch the likes sorted set for any changes. err = conn.Cmd("WATCH", "likes").Err if err != nil { return nil, err } // Use the ZREVRANGE command to fetch the album ids with the highest // score (i.e. most likes) from our 'likes' sorted set. The ZREVRANGE // start and stop values are zero-based indexes, so we use 0 and 2 // respectively to limit the reply to the top three. Because ZREVRANGE // returns an array response, we use the List() helper function to // convert the reply into a []string. reply, err := conn.Cmd("ZREVRANGE", "likes", 0, 2).List() if err != nil { return nil, err } // Use the MULTI command to inform Redis that we are starting a new // transaction. err = conn.Cmd("MULTI").Err if err != nil { return nil, err } // Loop through the ids returned by ZREVRANGE, queuing HGETALL // commands to fetch the individual album details. for _, id := range reply { err := conn.Cmd("HGETALL", "album:"+id).Err if err != nil { return nil, err } } // Execute the transaction. Importantly, use the Resp.IsType() method // to check whether the reply from EXEC was nil or not. If it is nil // it means that another client changed the WATCHed likes sorted set, // so we use the continue command to re-run the loop. ereply := conn.Cmd("EXEC") if ereply.Err != nil { return nil, err } else if ereply.IsType(redis.Nil) { continue } // Otherwise, use the Array() helper function to convert the // transaction reply to an array of Resp objects ([]*Resp). areply, err := ereply.Array() if err != nil { return nil, err } // Create a new slice to store the album details. abs := make([]*Album, 3) // Iterate through the array of Resp objects, using the Map() helper // to convert the individual reply into a map[string]string, and then // the populateAlbum function to create a new Album object // from the map. Finally store them in order in the abs slice. for i, reply := range areply { mreply, err := reply.Map() if err != nil { return nil, err } ab, err := populateAlbum(mreply) if err != nil { return nil, err } abs[i] = ab } return abs, nil } } |
和之前一样在route中添加我们的handle：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
File: main.go func main() { http.HandleFunc("/album", showAlbum) http.HandleFunc("/like", addLike) http.HandleFunc("/popular", listPopular) http.ListenAndServe(":3000", nil) } ... func listPopular(w http.ResponseWriter, r *http.Request) { // Unless the request is using the GET method, return a 405 'Method Not // Allowed' response. if r.Method != "GET" { w.Header().Set("Allow", "GET") http.Error(w, http.StatusText(405), 405) return } // Call the FindTopThree() function, returning a return a 500 Internal // Server Error response if there's any error. abs, err := models.FindTopThree() if err != nil { http.Error(w, http.StatusText(500), 500) return } // Loop through the 3 albums, writing the details as a plain text list // to the client. for i, ab := range abs { fmt.Fprintf(w, "%d) %s by %s: £%.2f [%d likes] \n", i+1, ab.Title, ab.Artist, ab.Price, ab.Likes) } } |
通过curl 发送GET /popular route请求，我们将看到：
|
1 2 3 4 5 6 7 8 |
$ curl -i localhost:3000/popular HTTP/1.1 200 OK Content-Length: 147 Content-Type: text/plain; charset=utf-8 1) Rumours by Fleetwood Mac: £7.95 [12 likes] 2) Nevermind by Nirvana: £5.95 [8 likes] 3) Electric Ladyland by Jimi Hendrix: £4.95 [8 likes] |
英文原文链接：http://www.alexedwards.net/blog/working-with-redis
如何让Redis与Go密切配合
有疑问加站长微信联系（非本文作者）
