Never-Ending Header Update

[stevied]

I wrote a little perl program to give me the first and last message numbers in that group...

Here is the output...

Net::NNTP=GLOB(0x9b3378)<<< 281 News.GigaNews.Com
Net::NNTP=GLOB(0x9b3378)>>> GROUP alt.binaries.boneless
Net::NNTP=GLOB(0x9b3378)<<< 211 1290087701 1540544898 2830632598 alt.binaries.boneless
First: 1540544898 Last: 2830632598
Net::NNTP=GLOB(0x9b3378)>>> QUIT
Net::NNTP=GLOB(0x9b3378)<<< 205 goodbye

[stevied]

so here is what i have found out... changing the datatype in mysql to bigint will keep mysql from limiting the number.. however with a 32bit php system php will change the number from an int to a float and then all bets are off with the math.

[spearhead]

Thanx for finding out! There was indeed a fault in the db code; I changed the type to bigint.

Running this code after logging into mysql will fix that:

Code:

alter table groups change first_record first_record bigint;

I'll look into the numbers in php later.

If you want to experiment a little, you can set it up to get which articles it gets, by switching to debug level (admin/config). You can see these messages in the log then:

Code:

Getting articles X - Y

I'd like to see the values of those.

[stevied]

2009-12-31 09:55:03 NOTICE Selecting group: alt.binaries.boneless
2009-12-31 09:55:03 DEBUG fn:alt_get_headers() pid:22829 (8388608 8388608)
2009-12-31 09:55:03 DEBUG Total max: 5000000
2009-12-31 09:55:03 DEBUG Getting articles 1540578338 - 2147483648
2009-12-31 09:55:05 DEBUG Incoming connection from host localhost:57548
2009-12-31 09:55:05 DEBUG read line: [USER stevied]
2009-12-31 09:55:05 DEBUG read line: [PASS XXX]
2009-12-31 09:55:05 DEBUG read line: [DISKFREE h]
2009-12-31 09:55:05 DEBUG fn:do_diskfree() pid:20179 (8388608 8388608)
2009-12-31 09:55:05 DEBUG read line: [DISKFREE p1]
2009-12-31 09:55:05 DEBUG fn:do_diskfree() pid:20179 (8388608 8388608)
2009-12-31 09:55:05 DEBUG fn:add_parts() pid:22829 (11272192 11272192)
2009-12-31 09:55:05 DEBUG Total max: 5000000, counter: 5001


The issue is that it never will populate the pars section because i have it set to expire in 5 days  which i would bet that is bigger then the 2147483648

[spearhead]

1540578338 - 2147483648

These are the article numbers it is getting... all fitting perfectly well in a signed 32-bit int

[spearhead]

Running this code after logging into mysql will fix that:

Code:

alter table groups change first_record first_record bigint;

Should be

Code:

alter table groups change first_record first_record bigint unsigned default 0;
alter table groups change last_record last_record bigint unsigned default 0;
alter table groups change mid_record mid_record bigint unsigned default 0;

[stevied]

the issue is that there are high number articles...


when i ran my perl test program it returned 2830632598 as the last article

2830632598 is greater then 2147483648

php can not do unsigned integers so in the output that it is getting


2009-12-31 09:55:03 DEBUG Getting articles 1540578338 - 2147483648

urd will never get between 2147483648 and 2830632598

so urd is truncating the last article number to fit into the 32bit int

[spearhead]

Hmmm... I see. I'll write some code to make it work with gmp later this week... Shouldn't be too much work really.

[stevied]

thanks... I have been looking at gmp and have it installed so let me know and i will test it out for ya.

[spearhead]

I made a version that uses gmp for indexing.

You can try it out from the latest svn update. (if you really want to do some testing on URD, you'd better install it form svn as that is where the most recent copy is anyway - http://code.google.com/p/urd/source/checkout)

Be sure that in config.php, this is set:

Code:

$config['use_alt_update'] = TRUE;

Anyway if you run into any problems, let me know.

(and don't forget to load the gmp module for php - sudo apt-get install php5-gmp on ubuntu)

[stevied]

ok i am grabbing the code now and i will let you know how it goes.

[Reab4]

Any updates?

[spearhead]

The change to gmp seems to work quite well. If you wait a day or two you can try it out in the forthcoming URD release.

[Reab4]

I've always had never-ending header updates and blank progress bars whenever the group database becomes large (e.g. >3MB?). This is the reason I stayed below ~3 days as expiration time. The boneless group e.g. quickly explodes >5.5MB with even 1 day expiration time and causes this 'hanging' update.
I now work from svn and still have this issue.

How does this gmp work under debian, do I need to install some package? There doesn't seem to be a single gmp package.

[spearhead]

I don't know exactly what it is called under debian, but under PHP you only have to install php5-gmp. It also installs libgmp3c2. I suspect this is the under debian.

But if gmp doesn't work, it probably causes a lot of errors... is there nothing in the log file?

[thorwak]

I'm just gonna jump right in, and maybe make a fool out of myself.. But have you checked the task list?

For me, what happens whenever I update a group is that it get new headers, and if this is the first time, there is a nice progress bar in status update. After this is done, status goes back to 0%, still saying "updating headers" (or whatever it says, can't check right now) when in reality it really should change to "generating sets" (this is a minor bug I've been meaning to report, but haven't gotten around to it yet). If one checks the task list one can see that header updating is actually done though, and set generating has started. For this, there is never any progress task bar (it's always at 0% until done).  And it can take a loooong time..

Say you have an already huge table (a few MB seems very little, but say a few million headers - maybe that's what you meant?) and get new headers. This can actually take only a few seconds, depending on how many NEW articles there is on the server. In fact, it may be so quick that the first "updating headers" (with working progress status bar) may not have time to actually appear in the status field (has to do with page refresh) and the first thing you see is the "updating headers" that really is "generating sets", just labeled wrong. Since set generation currently starts from the very beginning (re-indexing already existing sets), it gets worse and worse as time/retention progresses, even if very few new headers are actually downloaded.

For me, this state can take hours or even days as I've posted in other threads. It DOES finish eventually, but makes using any real retention impossible, and very confusing since one doesn't understand what is going on a first

Could this be what you're talking about? The symptom you describe in your last post matches exactly what I see here.

[Reab4]

I haven't installed gmp. Aptitude came up with a lot of packages which contained 'gmp' - just didn't know what to choose.

thorwak: Hmm you may very well be right (or not ), Last weekend I gave up after 10 hours of waiting. I just killed the thread and purged/removed the group. Purging is really fast. The boneless group showed (only) 5.5MByte large, with a default expiration of 1 day (Expire time for incomplete sets: 7)

Since I only have 1 nntp thread to the usenet server, it blocks all pending tasks. I can't increase it - I think eweka doesn't allow more than 1 nntp link for header downloads, I get too many 'Connection limit reached'; Timeout occurred (10); 'Cannot authenticate to server' -errors. Even with 1 nntp connection I get these but far less.

I'm not too sure about it failing on 'updating headers' or 'generating sets'. If it were the first, I would block pending tasks. If it were the latter, it should not block pending tasks (right?). If it is still blocking pending tasks after 10 hours, it must be with true updating. (?)

I just included the boneless group again (6668974 posts, 6.7M, initial eta ~5h) to try to reproduce it. I'll block auto-update for it though.

And yes, the initial header download takes some time but finished properly. It is the incremental updating which seems the problem. Or may it be a database handling issue?

[thorwak]

It does seems like you are experiencing what I describe.

You can check the tasklist to see what is actually going on when it seems to "hang" (which it usually doesn't - it's just not always so good at telling you what it does  )

The default is to just allow one database intensive task at a time (generating sets is such a task - the worst by far actually), I guess this could block others tasks of yours. This can be changed in the configuration.

Purging is always fast, I guess it just drops the table, really.

[spearhead]

I don't think you can run the update without gmp. So that may be one problem, if it isn't included. php -m from the command line should show gmp among its list.

admin / control gives a more definite view of what is going on. You can see the urd task unmodified running. The name of gensets was printed on purpose as updating, since it is not interesting and perhaps confusing for an average user to know exactly what's going on. You can always look at control to figure out any details.

purging truncates the table, that is fast unless the table is locked.

[thorwak]

Thanks for setting the facts straight

The name of gensets was printed on purpose as updating, since it is not interesting and perhaps confusing for an average user to know exactly what's going on.

I agree in general; in this particular case though I think it's way to _little_ info provided since it confuses me and, it seems, other users. Besides - is the typical Usenet user really a typical non-technical user?

Maybe one could get to see the real gensets name if one is grandmaster lvl at least 

[Styck]

Maybe say 'analyzing new data' instead of 'generating sets'...
We could even add a flowchart in the help page to explain how URD works

[thorwak]

You laugh  But I think both ideas are actually great 

And if I may be funny: It should actually say "Analyzing new and old data" for the time being

What?

[thorwak]

I've been playing around with threads and nntp connections settings (as described in another thread) and one thing I just found out is that set generating seems to count as (still) occupying an nntp connection, even though no talking to the server should be necessary. I'm not 100% sure about this yet, but it seems to work this way. If it is, it's a bug (and an easy fix I guess - just a flag that should be reset somewhere) and explains what you're seeing, Reab4!

[spearhead]

The flag exists indeed (urdd/urdd_command.php), but it is set to FALSE

[thorwak]

Funny, I could have sworn I had a DL that used all but one of the NNTP connections, and when the genset finished one the remaining connection was allocated to the DL as well.

I guess it's possible I got confused by total number of threads being the limit though, and that setting not being updated on the fly.