[resloved] problems auto-downloading...

[ProD]

Made sure I got the relevent check boxes checked - at the admin level and at the user level.

Put in some search terms.

Browsing the group shows highlighted items so that part seems to work fine.

Updated the group and after the update steps are completed (update headers, generate sets, expire) I have the following pop up into my log:

Code:

Something went wrong while auto-downloading: Could not execute SQL query "SELECT "name", "pass" FROM users WHERE "ID" = '8'" mysqli error: [2006: MySQL server has gone away] in EXECUTE("SELECT "name", "pass" FROM users WHERE "ID" = '8' LIMIT 1")

It seems to hang for a LONG time on expiring btw... I don't remember this long of a hang before enabling auto download.  I noticed that while the long hang is present in expiring phase it doesn't error on small groups.  I tried doubling the default_socket_timeout (to 120) in php.ini (in cli)... didn't seem to help.

I don't know if a timeout is the right thing to be chasing from that error it only occurred to me from some time on google and an odd thread found.  I also doubled the memory_limit to 256.

Just for fun I tried modifying the timeout to 1 hour (3600 seconds) and still got the error...

[spearhead]

This is a memory problem. There is another thread around here that has the same issue. It happens when the amount of matches is high. The auto-download code should be re-written to overcome this.

[spearhead]

I just committed an  an update for this problem. It seems to work better now (actually two bugs). Be sure to update your database by running the update.sh script (optionally with 1.0.6 as the first parameter. It may take a long time tho, depending on the database size).

[ProD]

I have been fiddling with the new version for a bit now (surprised about another speed change in the positive).  I didn't run the update.sh script previous to looking through the new version though after having run the script it did nothing (already updated message).  My guess it the install process updated the database (ran the script).

Automatic set downloading (matched strings) is indeed working... Though is it supposed to get everything that matches every time?  I was expecting that new sets would be checked for auto-download...

(artificial) Example: group set to 1 month before expiring sets. Group set to auto-update daily.  Group expected to have 1 new (matching) set added each day.  After updating the new item plus the previous month's matching sets are added to download. - so instead of 1 new item get 30 new items set to download (the previous 29 were already selected for download previously).

btw: love the nzb feature - has made testing much less painful...

[spearhead]

Installing will set up the new database. In case you upgrade you can just overwrite all the URD files and run the update.sh to upgrade the DB without losing data

I'll look into the autodl function again. It should only download them once indeed.

[spearhead]

I think the problem only manifests when you use download as nzb.

To fix try to edit functions/functions.php function auto_download_rss() _and_ in
function auto_download()

After the line

Code:

$sql .= 'WHERE (usersetinfo."statusread" != 1 OR usersetinfo."statusread" IS NULL) AND ';

add the line

Code:

$sql .= "(usersetinfo.\"statusnzb\" != 1 OR usersetinfo.\"statusnzb\" IS NULL) AND ";

[ProD]

Sorry for the delay... didn't notice the new post for some reason.
Tested it out after making the change to functions.php - also restarted urd, apache, and mysql for good measure.  Still generated a ton of NZB files.  I will try it again without the nzb option and see what happens. Another refresh of headers today should generate no new sets so this should be a good test (at least from not downloading everything).

[update] no new downloads with the manual update and auto-download nzb disabled (but auto-download enabled).  Will put in a new post if the next test works - auto-update headers and auto-download. [/update]

[ProD]

[edited, see below]

Update: Not so good.
A) I unchecked the option to halt unknown passworded downloads and not the NZB downloads so it was correct behavior downloads weren't starting (at least that day) since I had NZB turned on.

B) I had a lot of nzb files, this was with the extra line of code instead of just a few from the last few days.

C) I correctly turned off NZB auto-download leaving regular auto-download on and ended up with the large number of items again so whatever it is doesn't seem unique to nzb files.  Thankfully I now know where that clear all downloads button is (and that it's there) .

[edit]
I got a database query set up to count off the number of unread user sets...  I have a suspicion that may take a few days to check...  I think my activity in response to the seemingly duplicate auto-download entries (stop and clear all downloads) might be messing things up.  NOTE that this is not my reaction with the nzbs since that's a negligible load on the system.

Specifically I noticed that the count is dropping and I am not 100% certain the added downloads are duplicates.
<shakes head>...

[/edit]

[spearhead]

Off the top of my head, switching autodl to nzbs is an exclusive set from fully autodl'ed. That is after you switch it will autodl anything it hadn't before but as nzb instead of full dl or vice versa.

The flags for when thinsgs are dl'ed or nzb'ed are switched when the data is added to the queue, not when downloaded.

If your query takes that long, either your database is _huge_ or the query is wrong. sth like select count(*) from setdata left join usersetinfo on setdata.id = usersetinfo.setid and userid = 2 where statusread =1 or statusnzb =1 ; might do it. (you may have to run same case statements in there - you can find these in the code tho). this is just a bit of untested query, but essentially it.

[ProD]

Lets not discount the slowness of the system.  For reference it can't even use half the bandwidth available to it on header downloading, even with new, empty, groups.  Also I didn't expect the process to take a couple of days but leaving myself room to be able to monitor the process in person.

I am confident that it was my own doing (at least the auto-download stuff).  Basically I was killing the process before it could complete and being daily/weekly items (and in a state of concern to keep the system from virtually halting) I didn't catch that the sets were in fact newer than the previous ones.

Sorry for dragging this on.

I am going to wait for the groups to populate with something new, update with auto-nzb and then delete the created nzbs and do another update (I still have that line in the functions.php).

Code:

$sql .= "(usersetinfo.\"statusnzb\" != 1 OR usersetinfo.\"statusnzb\" IS NULL) AND ";

[ProD]

OK.  I have updated sets twice in a row (nzb auto-download), moving the nzb files out of the way before the 2nd update.  It is creating duplicate sets (made sure).  Notably it is not generating nzb files for everything, just items that haven't been auto-downloaded (nzb option OFF) yet.

If there are any other code bits you would like me to test I can.  The thing I was keen about is in fact working (no thanks to my activities).  Thanks much!

[spearhead]

So I guess this is resolved then. Or are there still issues?

[ProD]

The main thread of this does appear to be resolved.

Thanks.