v1.0.3 task listing on firefox 100% load

[Reab4]

Viewing the task list seems to take 100% system load.
I can only (when I get a slot) close the tab and reload urd.
I already tried clearing the cache to no avail.

btw, love the im/export settings and other added stuff.

[spearhead]

I suspect that the task list is just too long, and it task too long to get all the data. (that firefox hangs is a ff problem, imo the browser should be resilient against these kinda DoSes, but because it is single threaded it isn't)

Anyway, a quick fix may be in line 69 of html/ajax_admintasks.php
change

Code:

$timeval = 0;

ts

Code:

$timeval = 1;

In the long run a proper pageskipper is needed

[Reab4]

I changed it but I'm not sure that this is the case. The task page keeps refreshing, taking 100% processing from firefox. The whole browser becomes unresponsive.
The list itself flashed and vanishes and so does the activity/status window in the bottom-left corner. The quickest way to get out of this kinda deadlock is to close the tab and reload urd.

Also the new 'delete this set' action results an a timeout due to this action taking quite long. Doesn't matter if this a single set or a list of sets.
The action itself is performed ok but needs a manual page refresh.
 

[spearhead]

I presume you tested reloading the page with F5.

Does the firefox error log give any information?

[Reab4]

Yes I did that, including ^F5 and deleted the cache too.
It behaves the same on another machine. It just seems too much processing as both the task list and the progress box are constantly refreshing. This is severe as the other tabs seem unresponsive.
When I get home I'll restart the server as there have been some updates on php, mysql (i think) and apache and even the kernel. I've noticed the server itself becomes quite unresponsive after I ask a task list. Then I'm not able to ask the normal web pages.
For now I stay clear of the task list page.

About the 'delete this set'. I think the global 'delete this set' button (on top of the list) shows some odd behavior. If I use a single delete (on the right of a set) it seems ok. If I use the global, it can take ages and results in a timeout notification, even though there is only 1 set to delete. Also If there are say 10 sets listed and I manually delete 1 (so then there are 9)...if I then press the global delete, one set is added to the list to make it ten again and nothing is deleted. (all in case Max nr of lines per page: 10).

[Reab4]

Oh and the error log is almost empty:
Warning: Error in parsing value for 'cursor'.  Declaration dropped.
Source File: http://85.146.213.173/urd/html/smarty/templates/default/style.css
Line: 100

[spearhead]

in urdd/urdd_client.php

try replacing the functions delete_set() and delete_set_rss() with this code

Code:

function delete_set(array $setid)
    {
        $setid_list = '';
        if ($setid == array())
            return FALSE;
        $cnt = 0;
        foreach ($setid as $s) {
            $setid_list .= " $s";
            $cnt++;
            if ($cnt > 10) {
                list($code, $resp, $data) = $this->send_multi_command(get_command(COMMAND_DELETE_SET) . " $setid_list");
                $cnt = 0;
                $setid_list = '';

            }
        }
        if ($cnt > 0)
                list($code, $resp, $data) = $this->send_multi_command(get_command(COMMAND_DELETE_SET) . " $setid_list");

        if ($code == 200)
            return TRUE;
        else
            return FALSE;
    }
    function delete_set_rss(array $setid)
    {
        $setid_list = '';
        if ($setid == array())
            return FALSE;
        $cnt = 0;
        foreach ($setid as $s) {
            $setid_list .= " $s";
            $cnt++;
            if ($cnt > 10) {
                list($code, $resp, $data) = $this->send_multi_command(get_command(COMMAND_DELETE_SET_RSS) . " $setid_list");
                $cnt = 0;
                $setid_list = '';
            }
        }
        if ($cnt > 0)
            list($code, $resp, $data) = $this->send_multi_command(get_command(COMMAND_DELETE_SET_RSS) . " $setid_list");
        if ($code == 200)
            return TRUE;
        else
            return FALSE;
    }

That may fix the delete set problem. I'll look into the tasks later.

[Reab4]

The task list starts with 'show all'.
If I narrow that down to e.g. queued/1day, it becomes 'normal' again

The delete issue posted above is a minor issue indeed.

The behavior of a delete action seems to have more impact on firefox - it kind of results in freezing of ff, in the sense that no other ff tab is able to refresh until the delete task finishes. If the delete list is long, it may result in a timeout. To prevent this timeout, I press ESC and wait for the Activity block to appear again. Then I press search again to update the list.

[spearhead]

Pretty weird really. It seems like this is a database issue. Is the database process high in the cpu usage? Do you run the optimize database frequently? 'cause these queries are really light really. Does the browse sets give problems too? That is probably the heaviest web page in URD, db wise.

[Reab4]

Only the delete action causes this hold-up. I understand you don't have this problem with firefox. The browse sets behaves like it used to.
The browse sets action doesn't seem hold up the browser, possibly because it is finished presenting a list in a short time. So perhaps the freezing is also present but unnoticeable. Whereas the delete action takes more time for the server to complete. If you delete 1 set, if then behaves like the 'browse sets' action: ff regains control quickly. But if you delete a list of say 100 sets from the a.b.hdtv.x264 of 5GB each it may take a while and in the meantime ff is waiting and is unable to refresh the other open browser tabs.
Therefore perhaps the delete function behaves normally after all...not nice but like the other actions.
A similar behavior is seen with opening the (all) task list: ff waits for it to complete. But after it finishes, it is updated. So ff never gets time for other tasks. For the end user it looks like ff is freezing or 100% .

The database is optimized daily (automatically).

I've been playing with the delete function and now see that the deleted headers are just retrieved again from the usenet server (right?). So I stopped deleting unused sets. I just let the auto-purge function (>7 days) do it for me, like it used to do. I.e. I kind of don't use the delete function due to this.

[spearhead]

Well the browser waits for the command to complete on URDD. But deleting a set is relatively light, perhaps if all these sets are really large that it takes too much time. I can easily transform the process to run in the background, but that means that the browser screen isn't updated directly.

Headers are never collected twice, unless you purge the group. So the sets shouldn't resurface really. Perhaps the command never finishes completely.

I'll look into this later

[Reab4]

How about only marking them as 'to be deleted', remove them from the users' view and let some 'clean/purge the database'-routine handle it in the background daily or when idling. From the user's point of view it would be as fast as moving sets to the hidden list. Just a thought.

[spearhead]

Try out this patch  http://urdland.com/patch-1262-1264.diff

by running in the urd directory:

Code:

patch -p0 -i < patch-1262-1264.diff

Use this to test if it's safe first:

Code:

patch -p0 -i --dry-run < patch-1262-1264.diff

Or if you use svn run svn up.

I changed the delete set stuff + some tweaks to admin/tasks. Let me know if it helps.

[Reab4]

patch: **** Can't open patch file --dry-run : No such file or directory

So I did:
patch -p0 --dry-run -i patch-1262-1264.diff
and then
patch -p0 -i patch-1262-1264.diff

I now get a nice progress bar which is actually not progressing but finished nevertheless.
Now I did something stupid. I pressed the delete again as the list was still present. It became empty after a manual page refresh. The result is now that I have a "removing set:" activity which doesn't go away. In the task page (works ok now) there is a 'removing set: ... ' item which doesn't respond to a cancel and which doesn't go away after an urdd/mysql restart.
Log after cancel action: WARNING    Thread not found

How hufter-proof is it.
http://lmgtfy.com/?q=hufter-proof+means

[Reab4]

The delete action is still not very usable though. The sets resurface after an update. I tried it on a relatively small group (a.b.ebooks.technical). It just gathered sets again from up to 6 days.

[spearhead]

did you restart the urd daemon after patching?

Can you post / pm me a log file while setting log level in admin/config to debug from the line that looks like

Code:

[...] DEBUG fn:queue_delete_set()  [...]

non responsive items in status/downloads can always be removed in admin / control with clean all volatile data button (the blank one)

[Reab4]

There was no mention of something like: [...] DEBUG fn:queue_delete_set()  [...]. Only a 'Warning: thread not found'.
An urdd or mysql restart didn't help.
The clean all volatile data button did the trick though.

[spearhead]

I'm a little confused here. The queue_delete_set is what is called as soon as you click a button in the new version, the old one runs do_delete_set. Do you see any of these? Do you see any DEBUG messages at all?

Can you please do the following:
Set log level to debug mode first in admin/config. And clear the volatile info again in admin/control.
Terminate the urdd daemon first (click urdd is online in the status screen) if it is running.
Hop over to a command line and cd to the urd directory? (eg cd /usr/share/urd)
run ./urdd.sh -F -N -U 2> ./urdd-test.log
Then go back to the browser - any urd screen, wait til urdd is online to appear again. Go to browse sets / group sets and press on a "delete set" button on a set here and there. Never mind any warnings.
Go over to browse sets/ rss sets, if you have any and press some delete sets a few times.
Wait a minute or so and go back to the terminal where urd is running
press control-c to kill urdd. The urdd-test.log should be pretty big by now. PM/post the contents please.

[Reab4]

I'll do that "./urdd.sh -F -N -U 2> ./urdd-test.log" tomorrow but here's a debug log when I delete it without redirecting. First I global-delete a set. The screen is not reset so the set is stil visible. Then I global-delete again:

2010-Feb-14    20:28:29    ERROR    Set not Found 'fa5218ad23dd81dd415fdc501ec7eb96'
2010-Feb-14    20:28:29    WARNING    Child Died? Could not execute SQL query "SELECT "binaryID" FROM binaries_ WHERE "setID" = 'fa5218ad23dd81dd415fdc501ec7eb96'" mysql error: [1146: Table 'urddb.binaries_' doesn't exist] in EXECUTE("SELECT "binaryID" FROM binaries_ WHERE "setID" = 'fa5218ad23dd81dd415fdc501ec7eb96'")
2010-Feb-14    20:28:29    WARNING    
2010-Feb-14    20:27:18    INFO    Deleting 0 binaries
2010-Feb-14    20:27:15    INFO    Deleting 0 binaries
2010-Feb-14    20:26:53    INFO    Deleting 0 binaries
2010-Feb-14    20:26:50    INFO    Deleting 0 binaries
2010-Feb-14    20:25:48    INFO    Deleting 0 binaries
2010-Feb-14    20:25:36    INFO    Deleting 0 binaries
2010-Feb-14    20:22:10    INFO    Deleting 396 binaries

[spearhead]

ah this helps...

I think changing  urdd/do_functions.php will help here in function do_delete_set()

change

Code:

if ($res === FALSE) {
            write_log("Set not Found $setid", LOG_ERR);
            $cmt .= "Set not Found $setid ";

        }

to

Code:

if ($res === FALSE) {
            write_log("Set not Found $setid", LOG_ERR);
            $cmt .= "Set not Found $setid ";
            continue;
        }

(adding a continue)

[Reab4]

Yes, I think it is fixed now.

[spearhead]

Awesome. Thanx for helping in improving URD again!