http client max connections

[David Smith]

I have a web crawler that crawls as many different websites as possible at
once. In this case, an async web client like aleph is perfect. My
question is, how many connections will aleph allow me to have at the same
time? Where is this limit set in the code? For the sake of the rest of
the system, I would like as few connections as possible to take all the
network bandwidth, I don't want to be trying to open new connections when
there is no bandwidth and so have a load of stuff queueing up inside
aleph/netty. Are there any ways to block creating a new connection until
there is sufficient resources for a connection to be worthwhile or is this
something that has to be set once in config and uses trial and error to get
right?

Thanks,

David.

[David Smith]

A more high level and relevant way to put this is actually, how to I
propagate backpressure with an aleph http client? I want to only take
messages from an input stream when it is worthwhile, not just keep taking
more and more messages as fast as they come in and have loads of open http
requests.

Post by David Smith
I have a web crawler that crawls as many different websites as possible at
once. In this case, an async web client like aleph is perfect. My
question is, how many connections will aleph allow me to have at the same
time? Where is this limit set in the code? For the sake of the rest of
the system, I would like as few connections as possible to take all the
network bandwidth, I don't want to be trying to open new connections when
there is no bandwidth and so have a load of stuff queueing up inside
aleph/netty. Are there any ways to block creating a new connection until
there is sufficient resources for a connection to be worthwhile or is this
something that has to be set once in config and uses trial and error to get
right?
Thanks,
David.

[ztellman]

Hi David,

Sorry for the delayed response, Google Inbox seems to think these messages
are "low priority" and I missed a bunch. To answer your questions:

* the max connections are defined
here: https://github.com/ztellman/aleph/blob/master/src/aleph/http.clj#L108,
if you want to change that you can define your own connection pool and
specify it using `:pool` when making a request.

* There's no easy way for Aleph to automatically detect when your bandwidth
is saturated, and use that as a backpressure signal. However, the
InputStream that represents the response body will automatically exert
backpressure if you're not consuming it, so no connection will be
over-allotted bandwidth you can't make use of.

Hope that helps,
Zach

Post by David Smith
A more high level and relevant way to put this is actually, how to I
propagate backpressure with an aleph http client? I want to only take
messages from an input stream when it is worthwhile, not just keep taking
more and more messages as fast as they come in and have loads of open http
requests.

Post by David Smith
I have a web crawler that crawls as many different websites as possible
at once. In this case, an async web client like aleph is perfect. My
question is, how many connections will aleph allow me to have at the same
time? Where is this limit set in the code? For the sake of the rest of
the system, I would like as few connections as possible to take all the
network bandwidth, I don't want to be trying to open new connections when
there is no bandwidth and so have a load of stuff queueing up inside
aleph/netty. Are there any ways to block creating a new connection until
there is sufficient resources for a connection to be worthwhile or is this
something that has to be set once in config and uses trial and error to get
right?
Thanks,
David.