How to set limit for the maximum number of headers downloaded on first update?

[emike]

Is there any way to set a limit on the maximum number of headers downloaded on the first update?  I understand that you can set the number of days to expire, but this does not seem to affect the initial header download.  My news provider has excellent retention and it appears to me that regardless of what I set the expiration to be, the first header download is always whatever exists on the server being connected to.  If there was a way to set a limit on the number of headers it would certainly be helpful.

Thanks!

And btw, I think URD ROCKS!!!

[Styck]

Hey Emike,

Hmm the expire time should limit the initial header download; URD checks when the articles are outside the expire time (so recent enough to be included), and from there it should download every header up till the most recent one.

So setting it to 1 should limit the number of headers it will download. If you set the debug level to INFO then it should say in the log file how many articles it tries to download in an update, you could check there to see if it works OK.

Good luck and let us know if it worked

[emike]

I tested setting the expire time at 1 day and then at 90 days to try to set for the widest difference possible.  At 1 day expire time, the log info reported:

2009-02-10    10:46:06    NOTICE    Getting headers 1033370 to 2063261

At 90 days expire time, the log info reported:

2009-02-10    10:47:40    NOTICE    Getting headers 1033370 to 2063261


You indicated that:

Hmm the expire time should limit the initial header download; URD checks when the articles are outside the expire time (so recent enough to be included), and from there it should download every header up till the most recent one.

I am using giganews as my news provider.  I am wondering if giganews isn't providing the date information that your application is looking for to let you know how many headers to retrieve.  If there is any way to force a limit or alter a line of code so that I can impose this limit myself, I would greatly appreciate knowing how.

Thanks for your quick response and again, I absolutely love this application.  You guys ROCK!

[spearhead]

there is indeed an option to get the latest headers since a given date, unfortunately most usenet providers don't implement it. It's newnews. You can test that by logging in to your newsserver through telnet and type help. I need to log in there to check and I haven't an account there.

Some newsservers also do not support the next (and similar) commands that are needed for the update function currently. Or they take really long and urdd timesout.

I'm working on a new version of the update that actually searches backwards. It's mostly done tho and already in svn. That should resolve most issues really.

[emike]

thanks again for responding so quickly and its good to hear that a resolution is on the way.

I really love the application and appreciate your support.

[spearhead]

I committed the new update function to the svn repository

svn checkout http://urd.googlecode.com/svn/trunk/ urd-read-only

If you want to try it, it's here. (you do have to run the update.0.6.0-to0.6.1_mysql.sql (or sth) to update the database before restarting urd)

[emike]

I'm not familiar with svn or the process involved, but I went to the site you linked to and copied all of the files there and replaced the ones in my local installation.  I then ran the upgrade on my database as you recommended.  I then attempted another download and found that it still tries to get everything on the news server and does not pay attention to the expiration date on the initial update.

Thanks for the effort on my behalf and please let me know if I did something wrong.

Thanks!

[spearhead]

Using svn would be tremendously easier than copying all that stuff, in all the directories...

$config['use_alt_update'] 
what does this value give in config.php?

Otherwise, you might wanna set the loglevel to debug, and post the log here/ pm it to me. Then I can see what is going on.

the new function starts at the most recent post, than goes backwards in time until it finds a whole bunch msgs that are too old and then stops.

I could put a limit on the number of headers it downloads, but that would make it a lot more complicated.

[emike]

Great programmers are the wizards of this era, and you sir, are in the same league with Merlin himself!!!

IT WORKS!!!!!!!!!

Kudos to you and your team!!!  I am absolutely, completely and totally ecstatic!

Since I copied the files directly from the svn, my config.php wasn't updated as the svn site only has a config.php.default.  Once I added the use_alt_update statement to my config and set it true, my world changed for the better.

The log info now only says it is selecting the group, rather than giving information regarding the number of articles and where it will start and where it will end.  And then it gets quiet and goes about it's business.  The ETA for update completion was still huge, I think it said it would be finished in 1:18:32:02 or something else crazy like it had been doing before, but I gave it a chance and wouldn't you know it, about an hour or so later, BOOM!!!, it says it's finished.  I checked the article ages and none of them were more than a day old and since that's what I had set my expiration for BINGO!!! this baby is working like pure magic!

Thanks for your help, your patience while helping me and most of all for this entirely and completely awesome application. 

 

p.s.  I'm going to look around and see what I can do to support you and your efforts here.

[spearhead]

Cool! 

The ETA is still a bit flakey as it is calculated from all the available articles it hasn't downloaded yet. But it won't download all the articles anyway, but so it stops anyway when the articles become too old. I dont' really know yet how to fix that....

And any support is welcome really cheers