Website Physical Memory Gets Too High

[DemiG33k]

Hi, i love this tool been looking for it for ages please keep up the good work. I ran the crawler from my Linux box to my third party website server with 50k links or more for over two days to find that the website had crashed and a 500 error been returned from crawler output.
Durning the process I noticed on cpannel that the website physical memory was high, after I stopped the crawler the website came back online and my physical memory was back to normal.

I then ran the crawler again but changed the --max-concurrency setting to 2.
Although I’m probably about 2 hours into the crawl I can see the web sever physical memory rising again.

The above was my plan B, if this doesn’t work I need a plan C and am open to suggestions.

Although it may not solve the issue, in future versions it would be nice if you could start the crawler from where it left off, this would be useful also if the crawler itself crashed or even writing what it has crawled to the xml file while it’s running.. not after the crawl is complete. If these have been included already then I can only apologise in advance for missing them.

Thanks Neil

[DemiG33k]

3 days running no issues, until i find a socket hangup error (400) then the process was killed.. disappointed to say the least.