一个docker命令的访问流程

谢绝转载

序言

之前搞了一段时间docker volume plugin, 为了搞清楚plugin的工作机制,又研究了一阵docker源码的命令执行流程,程序员的记忆空间都是时刻刷新的,为了防止数据丢失,写下来方便以后温习.研究比较粗浅,有不对的地方请各位读者多提意见.

docker 版本：docker-ce(18.09)

本文会列出一些docker源码中的函数或者方法,因为篇幅原因会用'...'来省略一些内容,只留下我认为在调用流程中重要的一些内容. 在部分函数中会用注释标注关键点.

以下开始正文

一个docker命令的访问流程

以下是docker-ce代码的主要结构,主要关注前两个文件夹,即客户端和服务端.

下面通过docker命令的执行过程,来探究一下docker的代码调用流程.

客户端入口

客户端入口是docker.go中的main函数:

path	function name	line number
components/cli/cmd/docker/docker.go	main	172

func main() {
    // Set terminal emulation based on platform as required.
    stdin, stdout, stderr := term.StdStreams()
    logrus.SetOutput(stderr)

    dockerCli := command.NewDockerCli(stdin, stdout, stderr, contentTrustEnabled(), containerizedengine.NewClient)
    cmd := newDockerCommand(dockerCli)

    if err := cmd.Execute(); err != nil {
        ...
        os.Exit(1)
    }
}

main函数调用了newDockerCommand函数来初始化docker命令, 用到的是golang的命令行库Cobra.

path	function name	line number
components/cli/cmd/docker/docker.go	newDockerCommand	25

func newDockerCommand(dockerCli *command.DockerCli) *cobra.Command {
    ...
    cmd := &cobra.Command{
        Use:              "docker [OPTIONS] COMMAND [ARG...]",
        Short:            "A self-sufficient runtime for containers",
        SilenceUsage:     true,
        SilenceErrors:    true,
        TraverseChildren: true,
        Args:             noArgs,
        ...
    }
    cli.SetupRootCommand(cmd) #设置默认信息
  ...
    commands.AddCommands(cmd, dockerCli)#加载子命令

    ...
}

函数commands.AddCommands定义如下,可知加载了所有的子命令:

path	function name	line number
components/cli/cli/command/commands/commands.go	AddCommands	30

func AddCommands(cmd *cobra.Command, dockerCli *command.DockerCli) {
    cmd.AddCommand(
        ...
        // container
        container.NewContainerCommand(dockerCli),
        container.NewRunCommand(dockerCli),

        // image
        image.NewImageCommand(dockerCli),
        image.NewBuildCommand(dockerCli),

        ...

        // network
        network.NewNetworkCommand(dockerCli),

        ...

        // volume
        volume.NewVolumeCommand(dockerCli),
        ...
    )
}

以上就是docker命令行客户端的加载过程,也可以说是docker客户端命令执行的起点,我们通过具体命令的执行过程来看一下客户端命令是如何和服务端的代码对应起来的.以docker run命令为例.

docker run命令执行流程

根据上面的分析,我们找到docker run命令对应的函数NewRunCommand

path	function name	line number
components/cli/cli/command/container/run.go	NewRunCommand	35

func NewRunCommand(dockerCli command.Cli) *cobra.Command {
  ...
    cmd := &cobra.Command{
        Use:   "run [OPTIONS] IMAGE [COMMAND] [ARG...]",
        Short: "Run a command in a new container",
        Args:  cli.RequiresMinArgs(1),
        RunE: func(cmd *cobra.Command, args []string) error {
            copts.Image = args[0]
            if len(args) > 1 {
                copts.Args = args[1:]
            }
            return runRun(dockerCli, cmd.Flags(), &opts, copts)
        },
    }
    ...
    return cmd
}

命令运行则会运行runRun函数,经过一系列函数最终调用到同路径下的createContainer函数,可以发现docker run与docker create命令调用的是同一个函数:

path	function name	line number
components/cli/cli/command/container/create.go	createContainer	162

func createContainer(ctx context.Context, dockerCli command.Cli, containerConfig *containerConfig, name string, platform string) (*container.ContainerCreateCreatedBody, error) {
    ...

    //create the container
    response, err := dockerCli.Client().ContainerCreate(ctx, config, hostConfig, networkingConfig, name)
    ...
}

这里可以看到调用的是dockerCli.Client()的ContainerCreate方法,我们追踪溯源,看一下dockerCli.Client()是什么.

// Cli represents the docker command line client.
type Cli interface {
    Client() client.APIClient
    ...
}

通过Cli的接口定义可知Client()返回的是client.APIClient, 再看一下DockerCli中对接口方法Client()的具体实现:

path	function name	line number
components/cli/cli/command/cli.go	Client	82

// Client returns the APIClient
func (cli *DockerCli) Client() client.APIClient {
    return cli.client
}

通过接口的实现可知返回值就是cli.client,而cli.client是在newDockerCommand函数中调用dockerCli.Initialize初始化的:

path	function name	line number
components/cli/cli/command/cli.go	Initialize	168

func (cli *DockerCli) Initialize(opts *cliflags.ClientOptions) error {
    ...
    cli.client, err = NewAPIClientFromFlags(opts.Common, cli.configFile)
    ...
}

在NewAPIClientFromFlags中最终调用到了服务端的NewClientWithOpts函数,注意,连接客户端代码和服务端代码的关键来了:

path	function name	line number
components/engine/client/client.go	NewClientWithOpts	244

func NewClientWithOpts(ops ...func(*Client) error) (*Client, error) {
    client, err := defaultHTTPClient(DefaultDockerHost)
    if err != nil {
        return nil, err
    }
    c := &Client{
        host:    DefaultDockerHost,
        version: api.DefaultVersion,
        scheme:  "http",
        client:  client,
        proto:   defaultProto,
        addr:    defaultAddr,
    }

    ...

    return c, nil
}

首先,可知返回的c是一个Client类型的结构体指针,再看成员变量Client.client则是docker服务端的http客户端. Client的具体定义如下:

path	struct name	line number
components/engine/client/client.go	Client	68

// Client is the API client that performs all operations
// against a docker server.
type Client struct {
    // scheme sets the scheme for the client
    scheme string
    // host holds the server address to connect to
    host string
    // proto holds the client protocol i.e. unix.
    proto string
    // addr holds the client address.
    addr string
    // basePath holds the path to prepend to the requests.
    basePath string
    // client used to send and receive http requests.
    client *http.Client
    // version of the server to talk to.
    version string
    // custom http headers configured by users.
    customHTTPHeaders map[string]string
    // manualOverride is set to true when the version was set by users.
    manualOverride bool
}

在client.go 的相同路径下找到ContainerCreate方法,可知最后发出的是一个post请求:

path	function name	line number
components/engine/client/container_create.go	ContainerCreate	22

func (cli *Client) ContainerCreate(ctx context.Context, config *container.Config, hostConfig *container.HostConfig, networkingConfig *network.NetworkingConfig, containerName string) (container.ContainerCreateCreatedBody, error) {
    var response container.ContainerCreateCreatedBody

    ...

    serverResp, err := cli.post(ctx, "/containers/create", query, body, nil)
    if err != nil {
        if serverResp.statusCode == 404 && strings.Contains(err.Error(), "No such image") {
            return response, objectNotFoundError{object: "image", id: config.Image}
        }
        return response, err
    }

    err = json.NewDecoder(serverResp.body).Decode(&response)
    ensureReaderClosed(serverResp)
    return response, err
}

这里我们注意的是url的路径 "/containers/create".
那另一个疑问来了,该请求最终调用到什么地方去了?让我们话分两头,再去看看服务端的进程是如何启动的.

To be continued...

服务端程序

同样,我们找到入口函数,即服务端代码的main函数:

path	function name	line number
components/engine/cmd/dockerd/docker.go	main	52

func main() {
    ...
    cmd := newDaemonCommand()
    cmd.SetOutput(stdout)
    if err := cmd.Execute(); err != nil {
        fmt.Fprintf(stderr, "%s\n", err)
        os.Exit(1)
    }
}

main函数中只调用一个命令,那就是后台进程启动命令,没有子命令,newDaemonCommand函数如下:

path	function name	line number
components/engine/cmd/dockerd/docker.go	newDaemonCommand	18

func newDaemonCommand() *cobra.Command {
    opts := newDaemonOptions(config.New())

    cmd := &cobra.Command{
        Use:           "dockerd [OPTIONS]",
        Short:         "A self-sufficient runtime for containers.",
        SilenceUsage:  true,
        SilenceErrors: true,
        Args:          cli.NoArgs,
        RunE: func(cmd *cobra.Command, args []string) error {
            opts.flags = cmd.Flags()
            return runDaemon(opts)
        },
        ...
    }
    ...

    return cmd
}

类似客户端命令,这个命令执行后会调用到runDaemon函数,看一下进程启动都会做些什么

path	function name	line number
components/engine/cmd/dockerd/docker_unix.go	runDaemon	5

func runDaemon(opts *daemonOptions) error {
    daemonCli := NewDaemonCli()
    return daemonCli.start(opts)
}

NewDaemonCli()返回值是DaemonCli的结构体指针,先看下DaemonCli 的定义:

path	struct name	line number
components/engine/cmd/dockerd/daemon.go	DaemonCli	59

type DaemonCli struct {
    *config.Config
    configFile *string
    flags      *pflag.FlagSet

    api             *apiserver.Server
    d               *daemon.Daemon
    authzMiddleware *authorization.Middleware // authzMiddleware enables to dynamically reload the authorization plugins
}

再看DaemonCli的start方法,重点来了,这个方法包含整个docker进程的启动参数,我们只找到我们要找的关键部分:

path	function name	line number
components/engine/cmd/dockerd/daemon.go	start	74

func (cli *DaemonCli) start(opts *daemonOptions) (err error) {
    ...
    # 创建apiserver
    cli.api = apiserver.New(serverConfig)

    # 绑定监听端口
    hosts, err := loadListeners(cli, serverConfig)
    if err != nil {
        return fmt.Errorf("Failed to load listeners: %v", err)
    }

    ctx, cancel := context.WithCancel(context.Background())
    if cli.Config.ContainerdAddr == "" && runtime.GOOS != "windows" {
            ...
            # 启动containerd
            r, err := supervisor.Start(ctx, filepath.Join(cli.Config.Root, "containerd"), filepath.Join(cli.Config.ExecRoot, "containerd"), opts...)
            ...
            cli.Config.ContainerdAddr = r.Address()
            ...
    }
    ...

    # 创建pluginstore,用来保存plugin的相关信息
    pluginStore := plugin.NewStore()
    ...
    # 创建进程,并为守护进程设置一切服务
    d, err := daemon.NewDaemon(ctx, cli.Config, pluginStore)
    if err != nil {
        return fmt.Errorf("Error starting daemon: %v", err)
    }

    ...

    cli.d = d

    routerOptions, err := newRouterOptions(cli.Config, d)
    if err != nil {
        return err
    }
    routerOptions.api = cli.api
    routerOptions.cluster = c

    # 初始化路由
    initRouter(routerOptions)

    ...

    return nil
}

我们只关注调用流程,所以重点是初始化路由,这个函数的主要目的是绑定http请求到对应方法:

path	function name	line number
components/engine/cmd/dockerd/daemon.go	initRouter	480

func initRouter(opts routerOptions) {
    decoder := runconfig.ContainerDecoder{}

    routers := []router.Router{
        // we need to add the checkpoint router before the container router or the DELETE gets masked
        checkpointrouter.NewRouter(opts.daemon, decoder),
        container.NewRouter(opts.daemon, decoder),
        image.NewRouter(opts.daemon.ImageService()),
        systemrouter.NewRouter(opts.daemon, opts.cluster, opts.buildCache, opts.buildkit, opts.features),
        volume.NewRouter(opts.daemon.VolumesService()),
        build.NewRouter(opts.buildBackend, opts.daemon, opts.features),
        sessionrouter.NewRouter(opts.sessionManager),
        swarmrouter.NewRouter(opts.cluster),
        pluginrouter.NewRouter(opts.daemon.PluginManager()),
        distributionrouter.NewRouter(opts.daemon.ImageService()),
    }

    ...

    opts.api.InitRouter(routers...)
}

我们以容器为例具体看一下container.NewRouter():

path	function name	line number
components/engine/api/server/router/container/container.go	NewRouter	16

func NewRouter(b Backend, decoder httputils.ContainerDecoder) router.Router {
    r := &containerRouter{
        backend: b,
        decoder: decoder,
    }
    r.initRoutes()
    return r
}

具体的绑定过程在initRoutes中执行:

path	function name	line number
components/engine/api/server/router/container/container.go	initRoutes	31

func (r *containerRouter) initRoutes() {
    r.routes = []router.Route{
        // HEAD
        router.NewHeadRoute("/containers/{name:.*}/archive", r.headContainersArchive),
        // GET
        router.NewGetRoute("/containers/json", r.getContainersJSON),
        router.NewGetRoute("/containers/{name:.*}/export", r.getContainersExport),
        router.NewGetRoute("/containers/{name:.*}/changes", r.getContainersChanges),
        router.NewGetRoute("/containers/{name:.*}/json", r.getContainersByName),
        router.NewGetRoute("/containers/{name:.*}/top", r.getContainersTop),
        router.NewGetRoute("/containers/{name:.*}/logs", r.getContainersLogs, router.WithCancel),
        router.NewGetRoute("/containers/{name:.*}/stats", r.getContainersStats, router.WithCancel),
        router.NewGetRoute("/containers/{name:.*}/attach/ws", r.wsContainersAttach),
        router.NewGetRoute("/exec/{id:.*}/json", r.getExecByID),
        router.NewGetRoute("/containers/{name:.*}/archive", r.getContainersArchive),
        // POST
        # 找到重点了,"/containers/create"
        router.NewPostRoute("/containers/create", r.postContainersCreate),
        router.NewPostRoute("/containers/{name:.*}/kill", r.postContainersKill),
        router.NewPostRoute("/containers/{name:.*}/pause", r.postContainersPause),
        router.NewPostRoute("/containers/{name:.*}/unpause", r.postContainersUnpause),
        router.NewPostRoute("/containers/{name:.*}/restart", r.postContainersRestart),
        router.NewPostRoute("/containers/{name:.*}/start", r.postContainersStart),
        router.NewPostRoute("/containers/{name:.*}/stop", r.postContainersStop),
        router.NewPostRoute("/containers/{name:.*}/wait", r.postContainersWait, router.WithCancel),
        router.NewPostRoute("/containers/{name:.*}/resize", r.postContainersResize),
        router.NewPostRoute("/containers/{name:.*}/attach", r.postContainersAttach),
        router.NewPostRoute("/containers/{name:.*}/copy", r.postContainersCopy), // Deprecated since 1.8, Errors out since 1.12
        router.NewPostRoute("/containers/{name:.*}/exec", r.postContainerExecCreate),
        router.NewPostRoute("/exec/{name:.*}/start", r.postContainerExecStart),
        router.NewPostRoute("/exec/{name:.*}/resize", r.postContainerExecResize),
        router.NewPostRoute("/containers/{name:.*}/rename", r.postContainerRename),
        router.NewPostRoute("/containers/{name:.*}/update", r.postContainerUpdate),
        router.NewPostRoute("/containers/prune", r.postContainersPrune, router.WithCancel),
        router.NewPostRoute("/commit", r.postCommit),
        // PUT
        router.NewPutRoute("/containers/{name:.*}/archive", r.putContainersArchive),
        // DELETE
        router.NewDeleteRoute("/containers/{name:.*}", r.deleteContainers),
    }
}

通过以上的执行过程,我们找到了docker run最终调用的命令,接上文,继续docker run命令的执行.

继续docker run 命令的调用

通过路由绑定的http路径,我们找到了docker run调用的方法r.postContainersCreate:

path	function name	line number
components/engine/api/server/router/container/container_routes.go	postContainersCreate	446

func (s *containerRouter) postContainersCreate(ctx context.Context, w http.ResponseWriter, r *http.Request, vars map[string]string) error {
    ...

    ccr, err := s.backend.ContainerCreate(types.ContainerCreateConfig{
        Name:             name,
        Config:           config,
        HostConfig:       hostConfig,
        NetworkingConfig: networkingConfig,
        AdjustCPUShares:  adjustCPUShares,
    })
    ...
}

s.backend也就是前面DaemonCli的start方法中创建的deamon:

d, err := daemon.NewDaemon(ctx, cli.Config, pluginStore)

我们看一下deamon的定义:

path	struct name	line number
components/engine/daemon/daemon.go	Daemon	80

// Daemon holds information about the Docker daemon.
type Daemon struct {
    ID                string
    repository        string
    containers        container.Store
    containersReplica container.ViewDB
    execCommands      *exec.Store
    imageService      *images.ImageService
    idIndex           *truncindex.TruncIndex
    configStore       *config.Config
    statsCollector    *stats.Collector
    defaultLogConfig  containertypes.LogConfig
    RegistryService   registry.Service
    EventsService     *events.Events
    netController     libnetwork.NetworkController
    volumes           *volumesservice.VolumesService
    discoveryWatcher  discovery.Reloader
    root              string
    seccompEnabled    bool
    apparmorEnabled   bool
    shutdown          bool
    idMapping         *idtools.IdentityMapping
    // TODO: move graphDrivers field to an InfoService
    graphDrivers map[string]string // By operating system

    PluginStore           *plugin.Store // todo: remove
    pluginManager         *plugin.Manager
    linkIndex             *linkIndex
    containerdCli         *containerd.Client
    containerd            libcontainerd.Client
    defaultIsolation      containertypes.Isolation // Default isolation mode on Windows
    clusterProvider       cluster.Provider
    cluster               Cluster
    genericResources      []swarm.GenericResource
    metricsPluginListener net.Listener

    machineMemory uint64

    seccompProfile     []byte
    seccompProfilePath string

    diskUsageRunning int32
    pruneRunning     int32
    hosts            map[string]bool // hosts stores the addresses the daemon is listening on
    startupDone      chan struct{}

    attachmentStore       network.AttachmentStore
    attachableNetworkLock *locker.Locker
}

在相同路径下,找到Daemon的ContainerCreate方法:

path	function name	line number
components/engine/daemon/create.go	ContainerCreate	30

// ContainerCreate creates a regular container
func (daemon *Daemon) ContainerCreate(params types.ContainerCreateConfig) (containertypes.ContainerCreateCreatedBody, error) {
    return daemon.containerCreate(params, false)
}

对外可见方法ContainerCreate调用了私有方法containerCreate:

path	function name	line number
components/engine/daemon/create.go	containerCreate	34

func (daemon *Daemon) containerCreate(params types.ContainerCreateConfig, managed bool) (containertypes.ContainerCreateCreatedBody, error) {
    start := time.Now()
    if params.Config == nil {
        return containertypes.ContainerCreateCreatedBody{}, errdefs.InvalidParameter(errors.New("Config cannot be empty in order to create a container"))
    }

    os := runtime.GOOS
    if params.Config.Image != "" {
        img, err := daemon.imageService.GetImage(params.Config.Image)
        if err == nil {
            os = img.OS
        }
    } else {
        // This mean scratch. On Windows, we can safely assume that this is a linux
        // container. On other platforms, it's the host OS (which it already is)
        if runtime.GOOS == "windows" && system.LCOWSupported() {
            os = "linux"
        }
    }

    warnings, err := daemon.verifyContainerSettings(os, params.HostConfig, params.Config, false)
    if err != nil {
        return containertypes.ContainerCreateCreatedBody{Warnings: warnings}, errdefs.InvalidParameter(err)
    }

    err = verifyNetworkingConfig(params.NetworkingConfig)
    if err != nil {
        return containertypes.ContainerCreateCreatedBody{Warnings: warnings}, errdefs.InvalidParameter(err)
    }

    if params.HostConfig == nil {
        params.HostConfig = &containertypes.HostConfig{}
    }
    err = daemon.adaptContainerSettings(params.HostConfig, params.AdjustCPUShares)
    if err != nil {
        return containertypes.ContainerCreateCreatedBody{Warnings: warnings}, errdefs.InvalidParameter(err)
    }

    container, err := daemon.create(params, managed)
    if err != nil {
        return containertypes.ContainerCreateCreatedBody{Warnings: warnings}, err
    }
    containerActions.WithValues("create").UpdateSince(start)

    return containertypes.ContainerCreateCreatedBody{ID: container.ID, Warnings: warnings}, nil
}

由此,我们找到了客户端docker run执行命令后的整个代码流程.
本篇的主要目的是通过命令的代码流程, 把客户端的代码与服务端的代码连接起来,为以后探究其他命令的执行过程打下基础,简而言之,就是找到命令对应的服务端代码.

总结

1.起点

path	comment
components/cli/cli/command	客户端command定义

找到该命令调用的dockerCli.Client()下的函数名

2.然后跳转到对应的服务端的客户端代码

path	comment
components/engine/client	具体的http方法调用

3.然后跳转到对应的api server路由找到调用的函数

path	comment
components/engine/api/server/router	根据url找到具体的路由初始化

4.上一步找到的函数一般在daemon中定义具体执行

path	comment
components/engine/daemon	最终执行的方法