Skip to content

consumer: reuse lookupd http client#333

Merged
ploxiln merged 1 commit into
nsqio:masterfrom
JieTrancender:perfHTTPTransport
Oct 17, 2021
Merged

consumer: reuse lookupd http client#333
ploxiln merged 1 commit into
nsqio:masterfrom
JieTrancender:perfHTTPTransport

Conversation

@JieTrancender

Copy link
Copy Markdown
Contributor

Problem

After some wrong happend on my machine, i found there are many TIME_WAIT status.
I have many nsqd topics and n consumers, consumer making http request which not using Keep-Alive causes this problem.

[xxx@xxx-inc.com server]$ netstat -an |grep 19991|awk '{print $6}'|sort|uniq -c
      2 ESTABLISHED
      1 LISTEN
   1401 TIME_WAIT

Solution

  1. using go http.Transport features(keepalives,timeouts,dualstack)
  2. reuse http client
[xxx@xxx-inc.com server]$ netstat -an |grep 19991|awk '{print $6}'|sort|uniq -c
   1217 ESTABLISHED
      1 LISTEN
    305 TIME_WAIT
  1. support setting lookupd http client
[xxx@xxx-inc.com server]$ netstat -an |grep 19991|awk '{print $6}'|sort|uniq -c
    131 ESTABLISHED
      1 LISTEN
    355 TIME_WAIT

Comment thread api_request.go Outdated
Comment thread config.go Outdated
Comment thread consumer.go Outdated
Comment thread consumer.go Outdated
Comment thread config.go Outdated
@JieTrancender

JieTrancender commented Sep 29, 2021

Copy link
Copy Markdown
Contributor Author

@ploxiln Thank you for your review, i have finished these modifications, please review it again.

Comment thread UPGRADING.md Outdated
#### Misc.

Un-exported `NewDeadlineTransport` and `ApiRequest`, which never should have been exported in the
Un-exported `ApiRequest`, which never should have been exported in the

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can leave this as it was; it's still true that NewDeadlineTransport was un-exported (and then now has been un-existed ;)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you undo this little change to UPGRADING, and squash down to one commit (with cleaned-up commit message), I'll merge.

Comment thread consumer.go
Timeout: r.config.LookupdPollTimeout,
KeepAlive: 30 * time.Second,
}).DialContext,
ResponseHeaderTimeout: r.config.LookupdPollTimeout,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems redundant with the http.Client Timeout but I guess it's fine.
(Dialer Timeout above could probably be a shorter default, probably matching the 10s TLSHandshakeTimeout below. But again, I guess it's fine, doesn't really hurt.)

@JieTrancender

JieTrancender commented Oct 12, 2021

Copy link
Copy Markdown
Contributor Author

emmm, can request a review to other people? @ploxiln

@ploxiln ploxiln changed the title feat: reuse http client consumer: reuse lookupd http client Oct 17, 2021
@ploxiln
ploxiln merged commit 61df38e into nsqio:master Oct 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants