Problems with 0.6.1 version - Slow downloads (200-300 Kbits)

[VinZz]

Hello,
Il have upgraded from 0.5.5 to 0.6.1
Thank you for all the news features, il was great !

But I have a problem with that new version.
I have a celeron 1.4Ghz and 1Go of Ram server and now, the downloads are very very slow.
With the 0.5.5, it was around 4 and 5 Mbits.
Now, it was around 200-300 Kbits maxi.

Thank you for your help.

[VinZz]

Sorry for the post into the bad category ...

[spearhead]

Okay, I need some more info. What connection type are you using (SSL?). Are there any messages in the log file? What is the cpu usage (is it high for the URDD processes)?

Are you sure there are no other processes running that eat bandwidth? Is trickle enabled in admin/config?

[VinZz]

I am not using SSL.
I have the same parameters as into th 0.5.5 (I have keep the config file)
There is nothing into the logs.
I have 100 Mb symetric connexion.

Process :

Code:

CPU load averages: 56.14 (1 mins) , 42.54 (5 mins) , 28.86 (15 mins)

ID    Owner    CPU    Command   
13551 www-data 11.1 % /usr/bin/yydecode -e -b -f
13556 www-data 11.0 % /usr/bin/yydecode -e -b -f
13555 www-data 9.3 % /usr/bin/yydecode -e -b -f
10802 www-data 2.5 % /usr/bin/php /******/urdd/urdd.php -D
10797 www-data 2.4 % /usr/bin/php /******/urdd/urdd.php -D
10799 www-data 2.3 % /usr/bin/php /******/urdd/urdd.php -D
10800 www-data 2.2 % /usr/bin/php /******/urdd/urdd.php -D
10801 www-data 2.1 % /usr/bin/php /******/urdd.php -D
10798 www-data 1.9 % /usr/bin/php /******/urdd/urdd.php -D

There is actually only one running download (170Kb)

Trickle was called : "/usr/bin/trickle" but no shapping enabled.

[VinZz]

I have juste put the download in pause and :

CPU load averages:    3.09 (1 mins) , 43.81 (5 mins) , 50.68 (15 mins)

[VinZz]

I think it's caused to yydecode :

Code:

19432 www-data 12.0 % /usr/bin/yydecode -e -b -f
19434 www-data 10.4 % /usr/bin/yydecode -e -b -f
19448 www-data 10.0 % /usr/bin/yydecode -e -b -f
19435 www-data 9.5 % /usr/bin/yydecode -e -b -f

When I stard a download, there is a correct speed arount 1 300 Kb and next, it was decreasing to 200 Kb after 10 minutes (time to yydecode to take all the CPU)

[spearhead]

If the load goes to 100% it's yydecode. YYdecode processes should be really shortlived tho (one article / process). It maybe disk access that is slowing it down or a million other things. Frankly there hasn't been many changes is the download code itself.So I doubt that that is the problem.

you might try to increase this one Socket timeout in admin config. And see if that helps.

Otherwise atm I have little to go on really.

[VinZz]

I have changes socket timeout from 30 to 300 and it do the same :

15378    www-data    10.5 %    /usr/bin/yydecode -e -b -f
15379    www-data    10.5 %    /usr/bin/yydecode -e -b -f
15384    www-data    9.0 %    /usr/bin/yydecode -e -b -f
15385    www-data    6.0 %    /usr/bin/yydecode -e -b -f
15386    www-data    4.0 %    /usr/bin/yydecode -e -b -f

I have nothing changed onto the server but URD ...

[spearhead]

I have changes socket timeout from 30 to 300 and it do the same :

15378    www-data    10.5 %    /usr/bin/yydecode -e -b -f
15379    www-data    10.5 %    /usr/bin/yydecode -e -b -f
15384    www-data    9.0 %    /usr/bin/yydecode -e -b -f
15385    www-data    6.0 %    /usr/bin/yydecode -e -b -f
15386    www-data    4.0 %    /usr/bin/yydecode -e -b -f

I have nothing changed onto the server but URD ...

These aren't really weird, actually... yydecoding is quite heavy on the CPU. How long do these processes persist? More than a few seconds? Perhaps you could lower the number of threads to a server, or the total number of nntp threads.

Can you post (or PM or PM for my email address) a large section of a log file while running in debug mode (log level is set to debug in admin/config)?

[VinZz]

With only one NNTP connexion, it was running to 300 Ko/s

I have set priority of urdd process (and childrens) to -10.

Code:

1.38 (1 mins) , 1.35 (5 mins) , 1.31 (15 mins)

27118 www-data 62.0 % /usr/bin/yydecode -e -b -f
11000 www-data 15.0 % /usr/bin/php /*****/urdd/urdd.php -D

I set the logs now.

[spearhead]

the load of yydecode is not excessive. It should run only very shortly (much less than a second) but one process for each article. They run sequential per thread.

Can you see how long these yydecode threads are active? Normally when ps or top is running I rarely see them.

Is the disk showing excessive activity (such as caused by swapping)? (eg top shows the amount of swap memory used).

In fact I haven't any idea why you get so little throughput. The logs show nothing special, and the articles / time should be ok (40 articles in ~20 seconds)

[doenerali69]

i've exactly the same problem. did you solve it?!

[VinZz]

Hi,
Not yet.
I will update to URD 1.0.1

[spearhead]

You might try to add the -l option to yydecode. Perhaps that helps

In 1.0.1 this can be done in admin/config

in 0.6.x this must be done in the code (function/download_functions.php)

[doenerali69]

it says: "The file cannot be found or is not executable by the webserver "usr/bin/yydecode -l""



[EDIT:] aah sry i got it, but it doesn't fix the problem

[spearhead]

Can you see how long these yydecode threads are active? Normally when top or so is running I rarely see them. The should really just be a fraction of a second each. (it is run for each article independently).

Also please post/PM/PM for email the log file with log level set to debug.

[spearhead]

Also please try to run the test script in this tarball (never mind the image in it):
http://urdland.com/yydecode_test.tgz

run as
time bash test.sh

and post the timing output

[doenerali69]

ali@vampire:~# time bash test.sh

real    0m33.254s
user    0m22.780s
sys     0m10.720s

doesn't look good i think... is there any alternative to yydecode?

logfile follows...

[spearhead]

Shouldn't be any problem

This is what I get:

real   0m49.525s
user   0m33.762s
sys   0m14.561s

And URD runs just fine with it. I get max throughput (700kBps) and it is limited only by the available bandwidth.

You'd also might want to post the help/debug output (with urdd running)

Also , do you have any other threads running at the same time? Like updating newsgroups? How is the load on mysql/postgres?

[doenerali69]

i've send you a pm

here is what "top" shows me:

top - 22:41:31 up 21 min,  1 user,  load average: 3.23, 2.73, 1.71
Tasks: 110 total,   8 running, 102 sleeping,   0 stopped,   0 zombie
Cpu(s): 87.2%us, 10.8%sy,  0.0%ni,  0.7%id,  0.0%wa,  0.2%hi,  1.2%si,  0.0%st
Mem:   2061744k total,   995524k used,  1066220k free,    12296k buffers
Swap:  1646620k total,        0k used,  1646620k free,   851428k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
14894 www-data  20   0  1804  516  428 R   29  0.0   0:00.88 yydecode
14897 www-data  20   0  1804  508  428 R   24  0.0   0:00.73 yydecode
 8673 www-data  20   0 40908  14m 2428 S   10  0.7   0:57.80 php
 8674 www-data  20   0 40908  14m 2428 S   10  0.7   1:01.02 php
 8681 www-data  20   0 40908  14m 2428 S   10  0.7   1:02.43 php
 8677 www-data  20   0 40908  14m 2428 S    9  0.7   0:59.99 php
14900 www-data  20   0  1804  504  428 R    8  0.0   0:00.24 yydecode
13093 www-data  20   0 38484  11m 3592 R    7  0.6   0:01.83 apache2
 8685 www-data  20   0 40908  14m 2428 S    7  0.7   0:57.67 php
14904 www-data  20   0  1804  508  428 R    6  0.0   0:00.19 yydecode
14906 www-data  20   0  1804  516  428 R    5  0.0   0:00.15 yydecode
 8683 www-data  20   0 40908  14m 2428 S    5  0.7   0:56.78 php
13098 www-data  20   0 37964  11m 3580 S    4  0.6   0:01.67 apache2
 8679 www-data  20   0 40908  14m 2428 S    3  0.7   0:59.58 php
14882 root      20   0  2448 1164  912 R    1  0.1   0:00.04 top
 1673 root      15  -5     0    0    0 S    0  0.0   0:00.62 kjournald

[spearhead]

I see that you use 7 threads, it may be that processes are competing for resources. Can you try to lower that to e.g. 3 threads

also the throughput URD shows in transfers is a bit unreliable. Can you run this command:
vnstat -tr 20

It's far more accurate.

Neither the logs nor the config give any "funny" values. Actually the download times don't look that bad. 3 mins for 7 threads and 50 articles / thread

[doenerali69]

while downloading with 7 threads:
=========================
#vnstat -tr 20
20356 packets sampled in 20 seconds
Traffic average for eth0

      rx         6.92 Mbit/s           595 packets/s
      tx       235.60 kbit/s           422 packets/s

while downloading with 3 threads:
=========================
# vnstat -tr 20
15670 packets sampled in 20 seconds
Traffic average for eth0

      rx         5.39 Mbit/s           464 packets/s
      tx       180.40 kbit/s           319 packets/s

[spearhead]

If that is only the traffic that URD is generating it isn't too bad is it?
Or is your and your usenet providers bandwidth a lot higher?

[doenerali69]

i'm using a vDSL50+ (50Mbit) connection,

i tried downloading from usenet with 32 threads to the server on my notebook (winxp) with an average rate of 1,8mBYTE/sec

with urd i've only between 600 and 885kbyte/s

[spearhead]

Interesting ... your vnstat says Mbit /s ... while mine only says kByte /s ... hence my confusion

I'll look into it later.