to_nsq: added async flag#1332
Conversation
| err := producer.Publish(*topic, line) | ||
| var err error | ||
| if *async { | ||
| err = producer.PublishAsync(*topic, line, nil, nil) |
There was a problem hiding this comment.
This is "fire and forget". Are you trying to get better throughput, or ignore errors (perhaps from some producers)? I think it would be better to address each thing more specifically (but just the one you currently need for now).
- for better throughput, you could use
PublishAsync()for all producers, or for a batch of messages, and then check the publish success of the batch - to ignore errors for some producers or messages, that should be an explicit option
There was a problem hiding this comment.
It is for throughput, I'm at that point where to_nsq slows the software piping through it.
While we can live with a few miss on our side it totally does miss error reporting and probably a way to limit the number of simultaneous async requests.
I will improve this one further, feel free to close until then!
There was a problem hiding this comment.
Alternatively, rather than using PublishAsync, build an option to use multiple producers per destination nsqd (tradeoff would be more connections).
3 participants
Allows usage of async mode in
to_nsq