Never-Ending Header Update

[Reab4]

21:58... Since the last update (19:24:42) I have not seen the progress bar progress. This was also the last log entry. There is an ETA (7:23:43, fluctuates) however, which I did not see when it (seemingly) got stuck last time.
Also from the network history I can see it has been getting header data for the last few hours.

The steady red 100k line is caused by the header downloads. Eweka has a 100kbyte/s header download limit.

The normal behavior, seen from the log, is
- selecting group
- updating binary info   (gives an ETA and proper progress report)
- updating set info        (gives no proper progress report)

20 Apr 2010    19:24:42    NOTICE    Selecting group: alt.binaries.boneless
20 Apr 2010    19:24:42    INFO    using authentication
20 Apr 2010    19:24:41    INFO    Connecting to NNTP server: newsreader.eweka.nl:119
20 Apr 2010    17:26:50    NOTICE    Updating set info for alt.binaries.boneless complete
20 Apr 2010    17:26:45    NOTICE    Updating set info for alt.binaries.boneless
20 Apr 2010    16:29:32    NOTICE    Updating binary info for alt.binaries.boneless
20 Apr 2010    16:14:28    NOTICE    Selecting group: alt.binaries.boneless
20 Apr 2010    16:14:27    INFO    using authentication
20 Apr 2010    16:14:27    INFO    Connecting to NNTP server: newsreader.eweka.nl:119
20 Apr 2010    16:02:10    NOTICE    Updating set info for alt.binaries.boneless complete
20 Apr 2010    16:02:07    NOTICE    Updating set info for alt.binaries.boneless
20 Apr 2010    15:15:43    NOTICE    Updating binary info for alt.binaries.boneless
20 Apr 2010    14:33:14    NOTICE    Selecting group: alt.binaries.boneless
20 Apr 2010    14:33:13    INFO    using authentication
20 Apr 2010    14:33:13    INFO    Connecting to NNTP server: newsreader.eweka.nl:119
20 Apr 2010    12:50:03    NOTICE    Updating set info for alt.binaries.boneless complete
20 Apr 2010    12:50:01    NOTICE    Updating set info for alt.binaries.boneless
20 Apr 2010    12:23:32    NOTICE    Updating binary info for alt.binaries.boneless

I'll see what tomorrow brings..

Update:
20 Apr 2010    21:52:05    NOTICE    Updating binary info for alt.binaries.boneless

hehehe

[thorwak]

There you go. (It's not really practical to have headers above the amount 10-20M currently; it just takes too long to index. In boneless this is just 2-3 days unfortunately (4M+ per day)

[Reab4]

Just looked at the log. Last thing I did is set it to autoupdate-every-hour.
The updating binaries takes ~2 hours while the updating setinfo seems finished within minutes. It skipped the 1:00 and 5:00 cycle due to being busy. But no freezing.
I'll enable the other groups and set it to autoupdate-every-3-hours.

Is there a way to speed things up? - I mean is urd getting (some) headers it already has or is it getting the absolute latest ones, which it doesn't have?

The boneless group is now already at 10M, while the initial update was at 5M. Is this due to the incomplete sets set at 7-days expiration (default expiration=1 day)?

[spearhead]

URD shouldn't get any articles it already downloaded before. It keeps track of what it has downloaded and goes from there. Even if you interrupt and update.

You may look into mysqltuner.pl to find some speed up in the database.

[Reab4]

I've kind of come to a best solution to dealing with large groups:

(1)
"Default expire time (in days): "     =1
This is the how many days of headers it will initially collect.

(2)
"Age of removed files: "      =4   (was 8 )
"Age of volatile database info :"    =4   (was 8 )
Setting this lower yields a much lower and manageable database. Boneless ~5MB  (was>12MB)

(3)
"Expire after update: "      no  (un-tick)
With large groups and (3) ticked, the exipiration phase causes urd to become unresponsive. In firefox the urd page is not loading and causes other tabs not to load as well. This can be unlocked by pressing ESC un the urd tab. Even with this un-ticked, processing the large groups takes some time and can cause urd to become unresponsive - you just have to wait. I also set the autoupdate of large groups to 3 hours, starting at 0:50 and the smaller ones to 1 hour, starting at 0:00.

(4)
Since (3) is unticked, I:
"Optimise database:"     daily
"Clean database of volatile information : "   daily



From (2) I would expect that the database contains data from up to 4 days old. I can't find data>1day however. Is this normal?

[spearhead]

I think the browse page also limits by the expire value

[thorwak]

I never have any problems with URD, or the system in general, being unresponsive during updating, even with tasks that take many hours to complete. Searching sets etc is almost as fast as without these tasks running.

If you have the option, try having the DB files (/var/lib/mysql) on a physically separate disk - this well help a lot since the system disk won't be thrashed anymore and system load won't go crazy. Even just a separate partition for /var/lib/mysql may help a little bit, but physically different (and fast disk) helps the most. On a raid setup, a separate LUN is important (has to do with command queuing, per LUN)

[Reab4]

hmm  , indeed sounds like a system performance issue. It is anyway good to hear that it is not reproducible on your side. It doesn't surprise me, if I think of it, since running a proper 1080p from hda has not been doable even before urd was installed. hellanzb/rar/par caused too much disk activity.
I have all located on hda (=also / disk) , just to put hdc and hdd to sleep after 30mins.
l may run an experiment to have mysql on hdc for a while.

What about the 4 days expiration setting? How is this reflected by what the user can see?
With the settings I kind of expected:
    get-1-day-old-headers -> store in 4-day-history-database

[thorwak]

Well I CAN reproduce it - when I had everything, including the DB, in the same VM and on the same disk, system load could go up over 30 I even managed to crash the kernel a couple of times, was a long time ago I managed to do that.. Those problems went away however since I moved the DB outside of the VM, but most importantly to it's own disks, used for nothing else. System load is usually around 1.4-2.5 during large set updates, and since most of that is going to dedicated disks, other tasks and system in general run just fine.

par2/unrar is different, here all you can do is throw lots of RAM and fast disks at the problem. I can currently stream 1080p to my media player while unraring without hickups, but my last setup couldn't do that. You need plenty of RAM (for file cache) and fast disks. If course it would help a lot here too if you would par and unrar on disks other than those used for viewing files. Adjusting nice levels for urdd and it's child processes would help too, but I haven't had the need to experiment with this myself (I got a new home server recently so the HW being fast solves a lot of the performance issues I had earlier)

Since you have 3 physical drives you can improve plenty by using them. Spread the disk IO. Don't let your HW sleep dude 

The "default expiration" from admin config is just the value that will be chosen for you when you subscribe to new groups, it doesn't change anything by itself. The expiration can be changed for the individual groups afterwards too, and what happens is what one can expect: The next time expire is run (by default it's run automatically after an update) articles (and sets) older than this setting are removed and no longer available for download (you can save an NZB file though if you may want to download something later)

Edit: If you do a purge and full header DL, the expiration setting will be respected and you will end up with 4 days retention. If you started out with a setting of 1 however, and then increase it to 4 and update headers, it doesn't mean it will grow to 4 immediately. What happens is that nothing will be deleted from that group by "expire" until it's 4 days old, that is in 3 days in this example. Hope that clears it up

edit: typo

[Reab4]

The system is a 2.6GHz hyperthreading with 1GB RAM. It's already an oldy but suits me well for now. Consumption is 55~85W 24/7. You can see stats at 85.146.213.173. The system load is 6.5 when DL/rar/par and urd-handling, which is a bit high.

I just changed the mysql DB to hdc and upped the expiration to 3 days. Lets see how it behaves.

I still don't fully grasp the (2) setting. If (1) default expiration (or individual group expiration) is set to 1, then how will (2), set to 4, will have any effect. I.e. all>1day will be deleted due to (1), right?

There is also a "Expire time for incomplete sets (in days, 0 to disable): " setting. It overrides the (1) and (2) setting?

Many thanks for explaining.   

[spearhead]

The incomplete sets options remove incomplete sets (given the percentage) after the given time, regardless of the per-set expire date. So if a group has expire set to 10 days and incomplete is set to 3 all incomplete sets will be removed from that group after 3 days. All other sets after 10 days.

I'll have to dig in the browse page to recall how exactly the hidden, expire and all the other options work. There is another bug in browse that you can't search in hidden groups, even when that this the selected group. I'll fix that all later.

[Reab4]

After I moved the mysql DB to hdc I have not had a single hickup. Expire is up to 5 days and boneless has had 20MB of data. Surely gathering/processing all the data takes some time but the processing itself doesn't cause problems.
Thankyou very much for clearing the view.

[spearhead]

That's good. I'm working a bit on the sets generating function. It's stable yet, but you might wanna try it out. Also see this thread http://urdland.com/forum/index.php?topic=252.msg1751#new

[Reab4]

I noticed an update_db for 1.0.4 to 1.0.5 in svn (1437?). Do I need to run something to get it working?. I'm still at 1423 (i think). I'm at work now, can't hack.
How do I manually update the db again?

[spearhead]

Yea the update script is needed. If you can't run it, you'd have to add a dirty (smallint) to each parts_XXX and binaries_XXX table.

[thorwak]

Reab4, I'm happy I was able to help out a little. Cool to hear you got your performance issues sorted out!

There are great things in progress in the 1.0.5 svn code, but like Spearhead says it's a bit unstable currently. You might want to backup your DB if you care about it and in general hold on to your hat

Please report any findings though - the more testers the better!

Edit: It's being discussed here: http://urdland.com/forum/index.php?topic=252.25