Reynald Borer
2018-06-25 08:01:37 UTC
Hi folks,
I'm using Aleph as an http client to crawl multiple different URLs for
https://paper.li/ service.
On my testing environment, which is processing around 350 different URLs
per minute, Aleph pool seems to exhaust the heap (set to 9G) after a few
hours of processing. The culprit seems to be the Pool itself which is
retaining a lot of long & double instances.
I've posted my initial heap dump analysis
under https://github.com/ztellman/aleph/issues/394 . If you've already
experienced this kind of issue, your help is greatly appreciated.
I'll continue digging into aleph code to try to get a better understanding
of how it may happens.
Best regards,
Reynald
I'm using Aleph as an http client to crawl multiple different URLs for
https://paper.li/ service.
On my testing environment, which is processing around 350 different URLs
per minute, Aleph pool seems to exhaust the heap (set to 9G) after a few
hours of processing. The culprit seems to be the Pool itself which is
retaining a lot of long & double instances.
I've posted my initial heap dump analysis
under https://github.com/ztellman/aleph/issues/394 . If you've already
experienced this kind of issue, your help is greatly appreciated.
I'll continue digging into aleph code to try to get a better understanding
of how it may happens.
Best regards,
Reynald
--
You received this message because you are subscribed to the Google Groups "Aleph" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aleph-lib+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
You received this message because you are subscribed to the Google Groups "Aleph" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aleph-lib+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.