Performance issues with URD

[spearhead]

URD can appear really slow sometimes with larger news groups. However this is usually a database problem. For mysql users I recommend to use mysqltuner (http://www.mysqltuner.com) and for postgresql pgturen (http://pgfoundry.org/projects/pgtune/). Problems mostly appear in the update group and generating sets functionality. In the past I've seen some issues with the browse page too but that should be fixed by now.

Anyway, use this thread to discuss speed issues with URD and fixes for it.

[spearhead]

The database itself is typically the culprit when it comes to speed issues. Esp when URD has to handle huge newsgroups, it may slow down significantly. Especially hurt can be the browse sets page and the update / generating sets commands.

Mostly speed can be improved gigantly by improving the database settings, the tools described in the earlier post can be very helpful. As a starting point for mysql I recommend my-huge.cnf that comes with most installations.

The most important settings are:
key_buffer
max_allowed_packet
table_cache
join_buffer_size
sort_buffer_size
read_rnd_buffer_size

(don't forget to restart mysql and URDD after changing!)

Typically I find that mysql is speedier than postgres on all accounts, but this may be a setting issue nevertheless. The newly added (1.0.6) feels lots slower tho.

In any case I'd like to hear users' experiences with URD speed. What settings you use, what database type etc etc.

For users that upgraded to newer versions of URD, be sure to check that the indices are correct parts_* and binaries_* and setdata tables.  Many things have changed in that direction and indices can easily affect the speed by a factor 100. In case of doubts, or if you don't know how to fix it, be sure to post here.

Also if URD speed troubles you be sure to post here.

[d420]

What do you recommend as far as mysql settings? I have 8GB of RAM and using innodb. For the most part, it all runs pretty smooth except for very large groups where the groups will hang on generating the sets, I think. I am still on urd version 1.0.4 so will upgrading have any positive effect?

[spearhead]

What do you recommend as far as mysql settings? I have 8GB of RAM and using innodb. For the most part, it all runs pretty smooth except for very large groups where the groups will hang on generating the sets, I think. I am still on urd version 1.0.4 so will upgrading have any positive effect?

Are you really using innodb? the mysql config in URD says myisam.

You could take this as a starting point:
https://code.google.com/p/urd/source/browse/trunk/branches/devel/examples/my.cnf

Key_buffer usually has the biggest impact on speed.  Also try to run that mysqltuner, it helps greatly.

In v1.0.5 the set generation algorithm hasn't changed. The speed of the browse page has tho.

[d420]

Ahhhh maybe running innodb is part of the issue... I'll see what happens when I remove it. Hmmmm....

Thanks for the info!

[spearhead]

Uhm... I actually confused the numbers.... to summarise:
1.0.5 has a gensets speed up, especially for large groups (groups >2 M headers will benefit most) and also saves >30% diskspace in dthe DB
1.0.6 (which is supposed to be released shortly) has the browse page speed up immensly (factor 5 is common)

(and much more gory details are here: https://code.google.com/p/urd/source/browse/trunk/branches/devel/docs/CHANGELOG)

[lnxusr]

I installed URD yesterday and added one newsgroup (alt.biniaries.teevee).  After telling it to generate sets, it said it would take over a day.  After updating my.cnf to the values in the link provided by d420, nothing changed.  I decided to totally remove URD and reinstall from scratch.  The update time is now down to 14 hours.  This is unacceptable.  It could take weeks to fully index even a handful of newsgroups at that rate.

My newsserver has almost 200 million articles in that group alone.  There needs to be some way to tell URD to not index all articles from the start.  I wouldn't mind starting from, say, 3 or 6 months ago to begin with and go from there, or possibly being able to specify starting with a specified number of articles.  URD should be smart enough to detect if the specified number, or even date, is cutting a set of posts off in the middle.  It could either round up or down to get or eliminate a set of matching articles, or prompt on what to do.

My server is a AMD Athlon x2 6000+ (duel core 3GHZ) with 4 gig of ram.  If it takes this long for one newsgroup, there is something seriously wrong with URD.

Edit:  Ok, hang on here.  In the hour or so that URD started updating a.b.teevee, saying it would take 14 hours, and me writing this post, it seems a.b.teevee is done.  Perhaps URD is using MicroSofts time/percentage estimation logarithm?  A.b.multimedia says it'll take 12 hours to complete.  We'll see how long it actually takes.

[spearhead]

taking a random set of my.cnf values won't probably help much, since it depends especially on the amount of memory you have available and some other settings. Try using mysqltuner to properly tune mysql instead. (also don't forget to restart mysql). It may take a few iterations to get the proper settings. Finetuning it can actually save up to several factors of speed improvements. Esp this parameter is important "Temporary tables created on disk" as it can easily slow down queries by a factor 10-100. Also there is a lot of IO going on to the disks. (I heard some ppl had very good results with SSD here). This URD setting can also help sometimes: "Maximum headers per batch".

That said, indexing works in reverse, so the latest batch of articles is added first and it works backwards and the expire time tells it when to stop per group. There is also a global hard limit on the number of articles that is fetched per update (default is unlimited). Generating sets is really hard on the database so that is where it helps most to tune.

The estimation is a bit shakey, I know. 

[senax]

For me setting:

innodb_flush_log_at_trx_commit = 0

in /etc/my.cnf helped the most. Older versions of mysql had this set this way.

See here: http://dev.mysql.com/doc/refman/4.1/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit

Check for lots of writes using 'iostat -x 1', my disks went from 100% busy to almost idle.

Regards,
Senax