如何在Go中自定义超时重试的http.Client或http.Transport？

从现在开始，我们要努力学习啦！今天我给大家带来《如何在Go中自定义超时重试的http.Client或http.Transport？》，感兴趣的朋友请继续看下去吧！下文中的内容我们主要会涉及到等等知识点，如果在阅读本文过程中有遇到不清楚的地方，欢迎留言呀！我们一起讨论，一起学习！

我想为标准 http.client 实现自定义 http.transport，如果客户端超时，它将自动重试。

附注由于某种原因，自定义 http.transport 是必备。我已经检查过 hashcorp/go-retryablehttp，但是它不允许我使用我自己的 http.transport。

这是我的尝试，自定义组件：

type customtransport struct {
    http.roundtripper
    // ... private fields
}

func newcustomtransport(upstream *http.transport) *customtransport {
    upstream.tlsclientconfig = &tls.config{insecureskipverify: true}
    // ... other customizations for transport
    return &customtransport{upstream}
}

func (ct *customtransport) roundtrip(req *http.request) (resp *http.response, err error) {
    req.header.set("secret", "blah blah blah")
    // ... other customizations for each request

    for i := 1; i <= 5; i++ {
        resp, err = ct.roundtripper.roundtrip(req)
        if errors.is(err, context.deadlineexceeded) {
            log.warnf("#%d got timeout will retry - %v", i, err)
            //time.sleep(time.duration(100*i) * time.millisecond)
            continue
        } else {
            break
        }
    }

    log.debugf("got final result: %v", err)
    return resp, err
}

调用者代码：

func main() {
    transport := newcustomtransport(http.defaulttransport.(*http.transport))
    client := &http.client{
        timeout:   8 * time.second,
        transport: transport,
    }

    apiurl := "https://httpbin.org/delay/10"

    log.debugf("begin to get %q", apiurl)
    start := time.now()
    resp, err := client.get(apiurl)
    if err != nil {
        log.warnf("client got error: %v", err)
    } else {
        defer resp.body.close()
    }
    log.debugf("end to get %q, time cost: %v", apiurl, time.since(start))

    if resp != nil {
        data, err := httputil.dumpresponse(resp, true)
        if err != nil {
            log.warnf("fail to dump resp: %v", err)
        }
        fmt.println(string(data))
    }
}

我的实现没有按预期工作，一旦客户端超时，重试实际上不会发生。请参阅下面的日志：

2020-07-15T00:53:22.586 DEBUG   begin to get "https://httpbin.org/delay/10"
2020-07-15T00:53:30.590 WARN    #1 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 WARN    #2 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 WARN    #3 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 WARN    #4 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 WARN    #5 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 DEBUG   got final result: context deadline exceeded
2020-07-15T00:53:30.590 WARN    client got error: Get "https://httpbin.org/delay/10": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2020-07-15T00:53:30.590 DEBUG   end to get "https://httpbin.org/delay/10", time cost: 8.004182786s

你能告诉我如何解决这个问题，或者有任何方法/想法来实现这样的 http.client 吗？

解决方案

请注意，http.client 的 timeout 字段或多或少已经过时。现在的最佳实践是使用 http.request.context() 进行超时。 – 脆弱

感谢@flimzy 的启发！我尝试使用上下文进行超时控制而不是 http.client 方式。代码如下：

func (ct *customtransport) roundtrip(req *http.request) (resp *http.response, err error) {
    req.header.set("secret", "blah blah blah")
    // ... other customizations for each request

    for i := 1; i <= 5; i++ {
        ctx, cancel := context.withtimeout(context.background(), 10*time.second)
        defer cancel()
        //reqt := req.withcontext(ctx)
        resp, err = ct.roundtripper.roundtrip(req.withcontext(ctx))
        if errors.is(err, context.deadlineexceeded) {
            log.warnf("#%d got timeout will retry - %v", i, err)
            //time.sleep(time.duration(100*i) * time.millisecond)
            continue
        } else {
            break
        }
    }

根据日志，它有效（注意日志中的时间戳，它实际上重试了）：

2020-07-16t00:06:12.788+0800    debug   begin to get "https://httpbin.org/delay/10"
2020-07-16t00:06:20.794+0800    warn    #1 got timeout will retry - context deadline exceeded
2020-07-16t00:06:28.794+0800    warn    #2 got timeout will retry - context deadline exceeded
2020-07-16t00:06:36.799+0800    warn    #3 got timeout will retry - context deadline exceeded
2020-07-16t00:06:44.803+0800    warn    #4 got timeout will retry - context deadline exceeded
2020-07-16t00:06:52.809+0800    warn    #5 got timeout will retry - context deadline exceeded
2020-07-16t00:06:52.809+0800    debug   got final result: context deadline exceeded
2020-07-16t00:06:52.809+0800    warn    client got error: get "https://httpbin.org/delay/10": context deadline exceeded
2020-07-16t00:06:52.809+0800    debug   end to get "https://httpbin.org/delay/10", time cost: 40.019334668s

不需要自定义http.client之类的东西。您可以简单地将获取操作包装到重试中——有很多可用的模块可以做到这一点：

package main

import (
    "io"
    "log"
    "net/http"
    "os"
    "time"

    "github.com/avast/retry-go"
)

func main() {

    r, err := fetchDataWithRetries("http://nonexistant.example.com")
    if err != nil {
        log.Printf("Error fetching data: %s", err)
        os.Exit(1)
    }
    defer r.Body.Close()
    io.Copy(os.Stdout, r.Body)
}

// fetchDataWithRetries is your wrapped retrieval.
// It works with a static configuration for the retries,
// but obviously, you can generalize this function further.
func fetchDataWithRetries(url string) (r *http.Response, err error) {
    retry.Do(
        // The actual function that does "stuff"
        func() error {
            log.Printf("Retrieving data from '%s'", url)
            r, err = http.Get(url)
            return err
        },
        // A function to decide whether you actually want to
        // retry or not. In this case, it would make sense
        // to actually stop retrying, since the host does not exist.
        // Return true if you want to retry, false if not.
        retry.RetryIf(
            func(error) bool {
                log.Printf("Retrieving data: %s", err)
                log.Printf("Deciding whether to retry")
                return true
            }),
        retry.OnRetry(func(try uint, orig error) {
            log.Printf("Retrying to fetch data. Try: %d", try+2)
        }),
        retry.Attempts(3),
        // Basically, we are setting up a delay
        // which randoms between 2 and 4 seconds.
        retry.Delay(3*time.Second),
        retry.MaxJitter(1*time.Second),
    )

    return
}

文中关于的知识介绍，希望对你的学习有所帮助！若是受益匪浅，那就动动鼠标收藏这篇《如何在Go中自定义超时重试的http.Client或http.Transport？》文章吧，也可关注golang学习网公众号了解相关技术文章。