Webdumper gets stuck and hangs after running for too long...

[crayzn8]

I had the same problem with the old version. I grabbed the new version but it started doing the same thing after a while. I'm just extracting email addresses (and nothing else) from band profiles on http://www.purevolume.com

Are there log files or something I need to trash? I have "Infinite" checked for depth level, and "Do not allow" checked for external links.

I do a series of "browses" based on zip codes. Here is an example of one of the searches for a specific zip code:

http://www.purevolume.com/browse/?t=Art ... tion_city=

Then it will get through all those links, as well as all the ones on each of those pages. Then I enter a new zip code and start over with a new set of results.


Any reason why it just gets stuck after a series of different searches? I've tried restarting the program and the computer...cache as well. Seems like if I just don't use the program for a long period of time it starts working again.

Also, is there a way to have it remember what pages it had been to in the previous searches and skip those to save time? It goes through tons of the same pages on each new search right now. I know it does that for each individual search, I just would like it to remember the one before that, and before that...preferably.

Thanks for your help.

 

[stanbusk]

Re: Webdumper gets stuck and hangs after running for too lon

I guess this is because the server stops responding. I know of servers that will stop responding after intensive requests from the same IP. I guess it is some kind of security. Are you downloadingHTML files only ? What You can do is to press the 'Pause' button, wait a while and click on 'Resume'. The fact you say the program starts again after a while makes me think the server is indeed using some kind of security.

Also, is there a way to have it remember what pages it had been to in the previous searches and skip those to save time? It goes through tons of the same pages on each new search right now. I know it does that for each individual search, I just would like it to remember the one before that, and before that...preferably.

If you click on 'Pause' and then 'Resume' it will start where it left.

 

[crayzn8]

Re: Webdumper gets stuck and hangs after running for too lon

Well that brings up another good question.

When it gets stuck, I can hit pause and then start again from there. However, when that happens the total number of files to search in the URL goes up quite a bit. For example when I start searching a URL it might say it's processing "file 3 of 5,700". Then it will get stuck somewhere along the way and I hit pause and resume. Then the number of total files jumps to 8,300 usually. These are estimates and this happens pretty consistently. For the record, the exact same thing happens if I hit pause and restart from there (without the program getting stuck).

So where are these extra files to search coming from after I restart from a pause? This has happened on both versions of webdumper I have had.

As for my question about it remembering the previous searches, I don't think you understood me. I don't want it to search the same pages it already has in previous searches. It wastes a lot of time doing that. The reason I have to browse by zip code and do many different sessions is because the browse results on this site may show more than 75 pages of results, but that's all they let you access for some reason. I hit a wall, otherwise I would just let the program scan the entire site.

Any way around this?

 

[stanbusk]

Re: Webdumper gets stuck and hangs after running for too lon

It is weird because the 'Pause' simply disconnect the connections and 'Resume' reconnect them. It is pretty simple at code level. I will have a look anyway.

About your other problem, actually I did not design Web Dumper to extract email addresses from the web. To fix your problem I would better create a new product. I could easily remove 80% of the application code.

 

[crayzn8]

Re: Webdumper gets stuck and hangs after running for too lon

I was thinking maybe it's just starting over and adding the files it's already searched to the total for some reason. In any case it takes a lot longer to get through a URL search once paused, obviously.

A new product for this would be very helpful! I have been using MBM, Email Verifier, Bounce Handler, and Webdumper for almost 3 years now. Great products, I've really been able to do a lot with them.

Does email verifier have any drastic improvements now as opposed to a few years ago?

 

[stanbusk]

Re: Webdumper gets stuck and hangs after running for too lon

About eMail Verifier, it depends on the version you were using then, I am working on all the applications regularly. When I can make improvements, I do it... eMail Verifier is a very complex piece of software, it needs a regular maintenance to make sure it follows all new standards.