WSGIRef

2017-05-15

在上一篇文章 WSGI 的例子中我们多次利用了 wsgiref 来提供一个简单的 HTTP 服务器，但对其如何实现对 WSGI 应用的调用却没有深入了解。本文正是基于此点出发，以加深对 WSGI 的理解。

WSGIRef 本身很简单，在对 HTTPServer 的封装的基础上，实现了服务端的 WSGI 协议。其源代码量也是好少，很容易看懂。

本文打算用前文的例子来说清楚服务端是如何调用 WSGI 应用的：

# -*- coding: utf-8 -*-

from wsgiref.simple_server import make_server


# Application: with params environ and start_response (and optional one exc_info)
def simple_app(environ, start_response):
    """Simplest possible application object"""
    status = '200 OK'
    response_headers = [('Content-type', 'text/plain')]
    start_response(status, response_headers)
    return ['Hello World']  # !!!应用程序返回值需要为“可迭代对象”，如 list、生成器


# Server
httpd = make_server(
    '127.0.0.1',
    8051,
    simple_app
)

httpd.handle_request()

服务器初始化

在这个例子中，我们用 make_server 这个函数就构建了一个简单的服务器，其定义如下：

# wsgiref/simple_server.py

def make_server(
        host, port, app, server_class=WSGIServer, handler_class=WSGIRequestHandler
):
    """Create a new WSGI server listening on `host` and `port` for `app`"""
    server = server_class((host, port), handler_class)
    server.set_app(app)
    return server

下边我们来重点看下 WSGIServer 和 WSGIRequestHandler 这两个类。

WSGIServer

该类的继承关系为 WSGIServer -> BaseHTTPServer.HTTPServer -> SocketServer.TCPServer -> SocketServer.BaseServer。这里我们只需重点关注 WSGIServer 和 SocketServer.BaseServer 这两个类就行了。我们先来看下前者。

WSGIServer

这个类比较简单。服务端大部分重要工作都由其最顶层父类 SocketServer.BaseServer 完成，它提供对服务端 WSGI 应用的注册，看下代码就清楚了：

# wsgiref/simple_server.py

class WSGIServer(HTTPServer):
    """BaseHTTPServer that implements the Python WSGI protocol"""

    application = None

    def server_bind(self):
        """Override server_bind to store the server name."""
        HTTPServer.server_bind(self)
        self.setup_environ()

    def setup_environ(self):
        # Set up base environment
        env = self.base_environ = {}
        env['SERVER_NAME'] = self.server_name
        env['GATEWAY_INTERFACE'] = 'CGI/1.1'
        env['SERVER_PORT'] = str(self.server_port)
        env['REMOTE_HOST'] = ''
        env['CONTENT_LENGTH'] = ''
        env['SCRIPT_NAME'] = ''

    def get_app(self):
        return self.application

    def set_app(self, application):
        self.application = application

SocketServer.BaseServer

首先，比较重要的是，这个类提供了对请求处理类 RequestHandlerClass 的注册，方便以后处理请求时调用：

# SocketServer.py

class BaseServer:
    ......

    def __init__(self, server_address, RequestHandlerClass):
        """Constructor.  May be extended, do not override."""
        self.server_address = server_address
        self.RequestHandlerClass = RequestHandlerClass
        self.__is_shut_down = threading.Event()
        self.__shutdown_request = False

另外，这个类提供了两种服务器工作方式，即 serve forever 和 server once。这两种方式只是工作方式不一样，底层的请求处理还是一样的。本文用到的是后者，具体负责处理请求的函数是 handle_request：

# SocketServer.py

class BaseServer:
    ......

    def handle_request(self):
        """Handle one request, possibly blocking.

        Respects self.timeout.
        """
        # Support people who used socket.settimeout() to escape
        # handle_request before self.timeout was available.
        timeout = self.socket.gettimeout()
        if timeout is None:
            timeout = self.timeout
        elif self.timeout is not None:
            timeout = min(timeout, self.timeout)
        fd_sets = _eintr_retry(select.select, [self], [], [], timeout)
        if not fd_sets[0]:
            self.handle_timeout()
            return
        self._handle_request_noblock()

这里实际使用了 select 来对请求进行监听（轮询），最后调用函数 self._handle_request_noblock 对请求进行处理。这个函数最终会调用到 RequestHandlerClass：
self._handle_request_noblock -> self.process_request -> self.finish_request -> self.RequestHandlerClass(request, client_address, self)

而我们的 RequestHandlerClass 会最终继承自 SocketServer.BaseRequestHandler，当我们对它进行初始化的时候（处理一次请求初始化一次），它就会调用 handle 函数对请求进行处理：

# SocketServer.py

class BaseRequestHandler:
    ......

    def __init__(self, request, client_address, server):
        self.request = request
        self.client_address = client_address
        self.server = server
        self.setup()
        try:
            self.handle()
        finally:
            self.finish()

接下来请看我们实际注册到服务端的 WSGIRequestHandler。

WSGIRequestHandler

该类的继承关系为 WSGIRequestHandler -> BaseHTTPServer.BaseHTTPRequestHandler -> SocketServer.StreamRequestHandler -> SocketServer.BaseRequestHandler。这里我们要重点关注的是 WSGIRequestHandler 这个类。不过记住，处理请求时每次都会对其初始化一次，然后调用 handle 函数，见上一小节。

环境变量处理

直接上源代码吧：

# wsgiref/simple_server.py

class WSGIRequestHandler(BaseHTTPRequestHandler):
    server_version = "WSGIServer/" + __version__

    def get_environ(self):
        env = self.server.base_environ.copy()
        env['SERVER_PROTOCOL'] = self.request_version
        env['REQUEST_METHOD'] = self.command
        if '?' in self.path:
            path, query = self.path.split('?', 1)
        else:
            path, query = self.path, ''

        env['PATH_INFO'] = urllib.unquote(path)
        env['QUERY_STRING'] = query

        host = self.address_string()
        if host != self.client_address[0]:
            env['REMOTE_HOST'] = host
        env['REMOTE_ADDR'] = self.client_address[0]

        if self.headers.typeheader is None:
            env['CONTENT_TYPE'] = self.headers.type
        else:
            env['CONTENT_TYPE'] = self.headers.typeheader

        length = self.headers.getheader('content-length')
        if length:
            env['CONTENT_LENGTH'] = length

        for h in self.headers.headers:
            k, v = h.split(':', 1)
            k = k.replace('-', '_').upper()
            v = v.strip()
            if k in env:
                continue  # skip content length, type,etc.
            if 'HTTP_' + k in env:
                env['HTTP_' + k] += ',' + v  # comma-separate multiple headers
            else:
                env['HTTP_' + k] = v
        return env

    ......

请求处理

这里就比较重要了，是对请求的实际处理：

# wsgiref/simple_server.py

class WSGIRequestHandler(BaseHTTPRequestHandler):
    server_version = "WSGIServer/" + __version__
    ......

    def handle(self):
        """Handle a single HTTP request"""

        self.raw_requestline = self.rfile.readline()
        if not self.parse_request():  # An error code has been sent, just exit
            return

        handler = ServerHandler(
            self.rfile, self.wfile, self.get_stderr(), self.get_environ()
        )
        handler.request_handler = self  # backpointer for logging
        handler.run(self.server.get_app())

其中，ServerHandler 定义为：

# wsgiref/simple_server.py

class ServerHandler(SimpleHandler): # ServerHandler -> SimpleHandler -> BaseHandler，最后者才是真正对请求进行处理的类，请看下文
    server_software = software_version

    def close(self):
        try:
            self.request_handler.log_request(
                self.status.split(' ', 1)[0], self.bytes_sent
            )
        finally:
            SimpleHandler.close(self)

执行 handler.run 最终会调用 WSGI 应用，请看下文。

调用 WSGI 应用

前边我们说到，请求处理最终是由类 BaseHandler 进行处理的。服务端的 WSGI 标准就是由它封装的，并最终调用应用和处理响应。我们看下它提供的 run 函数就很清楚了：

# wsgiref/handlers.py

class BaseHandler:
    """Manage the invocation of a WSGI application"""
    ......

    def run(self, application):
        """Invoke the application"""
        # Note to self: don't move the close()!  Asynchronous servers shouldn't
        # call close() from finish_response(), so if you close() anywhere but
        # the double-error branch here, you'll break asynchronous servers by
        # prematurely closing.  Async servers must return from 'run()' without
        # closing if there might still be output to iterate over.
        try:
            self.setup_environ()    # 1. WSGI 标准要求的环境变量
            self.result = application(self.environ, self.start_response)    # 2. 按 WSGI 标准调用应用
            self.finish_response()    # 3. 对返回结果的处理
        except:
            try:
                self.handle_error()
            except:
                # If we get an error handling an error, just give up already!
                self.close()
                raise  # ...and let the actual server figure it out.

到这里，顿感焕然大悟！原来服务端就是这么准备环境变量，然后按 WSGI 标准调用应用的。

WSGI 环境变量构造

接下来，我们可以看下， WSGI 环境变量是如何构造的：

# wsgiref/handlers.py

class BaseHandler:
    """Manage the invocation of a WSGI application"""

    # Configuration parameters; can override per-subclass or per-instance
    wsgi_version = (1, 0)
    wsgi_multithread = True
    wsgi_multiprocess = True
    wsgi_run_once = False
    ......

    def setup_environ(self):
        """Set up the environment for one request"""

        env = self.environ = self.os_environ.copy()
        self.add_cgi_vars()

        # 添加跟 WSGI 相关的环境变量
        env['wsgi.input'] = self.get_stdin()
        env['wsgi.errors'] = self.get_stderr()
        env['wsgi.version'] = self.wsgi_version
        env['wsgi.run_once'] = self.wsgi_run_once
        env['wsgi.url_scheme'] = self.get_scheme()
        env['wsgi.multithread'] = self.wsgi_multithread
        env['wsgi.multiprocess'] = self.wsgi_multiprocess

        if self.wsgi_file_wrapper is not None:
            env['wsgi.file_wrapper'] = self.wsgi_file_wrapper

        if self.origin_server and self.server_software:
            env.setdefault('SERVER_SOFTWARE', self.server_software)

WSGI start_response 函数

还有，WSGI start_response是这样子构造的：

# wsgiref/handlers.py

class BaseHandler:
    """Manage the invocation of a WSGI application"""
    ......

    def start_response(self, status, headers, exc_info=None):
        """'start_response()' callable as specified by PEP 333"""

        if exc_info:
            try:
                if self.headers_sent:
                    # Re-raise original exception if headers sent
                    raise exc_info[0], exc_info[1], exc_info[2]
            finally:
                exc_info = None  # avoid dangling circular ref
        elif self.headers is not None:
            raise AssertionError("Headers already set!")

        self.status = status
        self.headers = self.headers_class(headers)
        assert type(status) is StringType, "Status must be a string"
        assert len(status) >= 4, "Status must be at least 4 characters"
        assert int(status[:3]), "Status message must begin w/3-digit code"
        assert status[3] == " ", "Status message must have a space after code"
        if __debug__:
            for name, val in headers:
                assert type(name) is StringType, "Header names must be strings"
                assert type(val) is StringType, "Header values must be strings"
                assert not is_hop_by_hop(name), "Hop-by-hop headers not allowed"
        return self.write

处理应用返回数据

# wsgiref/handlers.py

class BaseHandler:
    """Manage the invocation of a WSGI application"""
    ......

    def finish_response(self):
        """Send any iterable data, then close self and the iterable

        Subclasses intended for use in asynchronous servers will
        want to redefine this method, such that it sets up callbacks
        in the event loop to iterate over the data, and to call
        'self.close()' once the response is finished.
        """
        try:
            if not self.result_is_file() or not self.sendfile():
                for data in self.result:    # 到这里，我们应该明白，为什么应用返回的 body 要是“可迭代”的对象了吧！
                    self.write(data)
                self.finish_content()
        finally:
            self.close()

总结

用博文 wsgiref 源码解析来总结我们前边提到的工作流程挺好的：

另外，该博文还详述了一条 HTTP 请求的旅程：

服务器端启动服务，等到客户端输入 curl -i http://localhost:8000/ 命令，摁下回车键，看到终端上的输出，整个过程中，wsgi 的服务器端发生了什么呢？

服务器程序创建 socket，并监听在特定的端口，等待客户端的连接（注：本文例子是在调用 httpd.handle_request() 后进入等待状态）

客户端发送 http 请求

socket server 读取请求的数据，交给 http server

http server 根据 http 的规范解析请求，然后把请求交给 WSGIServer

WSGIServer 把客户端的信息存放在 environ 变量里，然后交给绑定的 handler 处理请求

HTTPHandler 解析请求，把 method、path 等放在 environ，然后 WSGIRequestHandler 把服务器端的信息也放到 environ 里

WSGIRequestHandler 调用绑定的 wsgi ServerHandler，把上面包含了服务器信息，客户端信息，本次请求信息得 environ 传递过去

wsgi ServerHandler 调用注册的 wsgi app，把 environ 和 start_response 传递过去

wsgi app 将reponse header、status、body 回传给 wsgi handler

然后 handler 逐层传递，最后把这些信息通过 socket 发送到客户端

客户端的程序接到应答，解析应答，并把结果打印出来。