Auto Set Download to Per Newsgroup Directories?

[Bonedaddy]

Apologies if this is obvious or documented somewhere I haven't found.

Is it possible to have all sets found be automatically downloaded to a different directory based on the newsgroup name?

[spearhead]

yes there is. There is an option in preference to set it called "Download directory format:" since 1.0.5. It has a bunch of options there including the group name.

[Bonedaddy]

Thanks very much.  However I'm running 1.0.5 and don't see those in preferences (even as a Grandmaster user?)

Is it after 1.0.5?  Do I need to upgrade?

Or am I looking in the wrong place.   

Thanks!

[Bonedaddy]

I'm sorry, I'm an idiot. 

I see it now, last item.

Now I just need to find out how to autodownload all sets found?

Thanks

[spearhead]

There is an autodownload option in admin/configuration you need to enable. Then hop back to (user) preferences tag auto download there too. And fill in % as a search term.  That should do it, but I haven't tried it tho.

[Bonedaddy]

Ah, yes, I do see those options and have tried it.

Unfortunately I get:

30 Oct 2010    16:14:29    NOTICE    Something went wrong while auto-downloading

which looks like download_sets() failed but it doesn't log the exception message text so I'm not sure why.

Although now that I stopped urdd and restarted it it all seems to be downloading OK!

Thanks!  I have to say that urd is pretty slick!!!

[spearhead]

Interesting. It may be that it hits some limit somewhere, memory or buffersize of mysql or sth like that. Anyway I added the exception message now to the error line

Thanx ...

[Bonedaddy]

Thanks.  It is happening again.  It seems that database  is throwing:

Could not execute SQL query "SELECT "name", "pass" FROM users WHERE "ID" = '2'" mysql error: [2006: MySQL server has gone away] in EXECUTE("SELECT "name", "pass" FROM users WHERE "ID" = '2' LIMIT 1")

The server is still there (I can use mysql to look at things anyway.)

Do I need to increase a timeout somewhere?

Thanks

[spearhead]

Probably this line in mysql's my.cnf (be sure to take my_huge.cnf as a starting point, urd is a heavy db user)

Code:

max_allowed_packet = 16M

You may have to increase the value even more.

[Bonedaddy]

Thanks.  I found the sample my-huge.cnf but can't run it for some reason.  I increased all the values in my my.cnf especially the max_allowed_packet.

The bigger problem was I ran out of space on my mysql fs.    Added another 333 GB  but will have to watch it.  :-)

I have one final question though.  Is there a way to get a "mostly" flat file save for the downloads?

For example, I'd like all files in <configured download directory>/<group name>/file1.jpg...  So, no username component and no set name component.

Thanks for the great software!

[spearhead]

There is none such option. Username is always there and so is the setname... some custom scripts can fix that for you tho. Set the allow scripts option in admin/config and set the right script to run in preferences. There is an example you can expand. A script can basically be any type of script (bash, php, perl, python...) and is run after each download for that download. Moving all files in a directory shouldn't be that hard to implement.

[Bonedaddy]

Thanks very much!  I'll take a look at the scripts.

I can't believe urd hasn't been picked up by more distributions!

[spearhead]

You're welcome. If you run in to any troubles, let me know.

and any kind of promotion is welcomed btw 

[Bonedaddy]

Thanks.  I seem to still be getting errors trying to autodownload:

05 Nov 2010    15:30:59    NOTICE    Could not execute SQL query "SELECT name,pass FROM users WHERE ID = '2'" mysql error: [2006: MySQL server has gone away] in EXECUTE("SELECT name,pass FROM users WHERE ID = '2' LIMIT 1")

I ended up changing the query a little as the double quotes around name, pass, and ID gave me no results in the mysql client.

I have max_allowed_packet at 32M (though the reply to that query should be quite small, just the single row with name and password?)

Do you have any suggestions for fixing or debugging further?

Thanks

[spearhead]

Don't change the queries! Change your MYSQL client config:

Code:

SET sql_mode='ANSI_QUOTES'

The quoting style for vanilla mysql is pretty retarded and not ansi compliant. This is fixed with the above statement.

Probably the query that was run prior to this query caused mysql to err.
It's probably the function auto_download in functions/functions.php that causes the trouble.

You should probably also glance at the mysql log. It may give some information on what happened. There also tools like mtop that can be useful when debugging mysql.

Running urdd from the command line is also a good thing when wanting to do some debugging (you may get some rights warnings tho here and there if you started it before from the webclient). And of course setting log level to debug mode helps.

[Bonedaddy]

Yes, I did see that after the fact.  :-)

I tracked it down to the DatabaseConnection constructor failing in the connect().

I added a $this->force_new_connection = 1; after the NewADOConnection.

This seems to help a lot but I don't know if that's chance or not  :-)

[spearhead]

I have one final question though.  Is there a way to get a "mostly" flat file save for the downloads?

For example, I'd like all files in <configured download directory>/<group name>/file1.jpg...  So, no username component and no set name component.

I've just added this to the current svn release. Please try it out.
If you start the directory format with # it will skip the setname, you can (re-)add it with %s anyway. Username stays in there tho.
E.g. #/%g/ would mean all files are moved to
<configured download directory>/<username>/<group name>/

plain /%g/ would mean all files are moved to
<configured download directory>/<username>/<group name>/<setname>/

I haven't really tested it a great deal and it isn't documented yet. But it should work. Try it out if you will.

[Bonedaddy]

Thanks, I'll give it a try.  Probably tomorrow.

Thanks for all the work and support!

[Bonedaddy]

OK, I updated via svn and I'm still seeing it put the files in set name directories so maybe I'm not specifying right.  I've tried #/%g/ and #%g

Although I don't know if that is reread when restoring a download or if I need to wait for a new download to start?

Also, the help text for that field when I hover is cut off after four  entries or so (because it goes down from the field.)

Thanks though!

[spearhead]

Have you restarted URDD?

Either syntax should fix it tho. Setting is read when the download is moved after everything else is finished.

Which browser do you use?

[Bonedaddy]

Firefox 3.6.12

I'll double check all that as long as I know my syntax is OK.

Thanks.

[Bonedaddy]

Well, I'm back to having the problem:

urdd: NOTICE Could not execute SQL query "SELECT  "name", "pass" FROM users WHERE "ID" = '2'" mysql error: [2006: MySQL server has gone away] in EXECUTE("SELECT  "name", "pass" FROM users WHERE "ID" = '2' LIMIT 1")

Again, it's fine from mysql so I have some kind of connection issue between urdd and mysql.

Forcing a new connection doesn't do anything (as the adodb implies for mysql.)

I don't see anything in the mysql logs or apache logs

I've run mysqltuner and done what's suggested there

Any suggestions for trying to debug this?

Thanks

[spearhead]

Not really. The problem you have is with the query that happens _before_ the one you see.

Google showed the source is mostly wait_timeout or max_allowed_packet  in my.cnf. You may have to increase both (to pretty large values). I guess that at one point in the auto download code either the query becomes too large or it takes too long to pump it into the database. How many sets do you want it to dowload anyway?

You could enable urd query logging by adding this to config.php

Code:

define('QUERY_LOG', TRUE);

You can then

Code:

tail -f /tmp/urd_query.log

To see what happens. The last successful query is probably the issue (not showing the URD webpages will limit the amount of queries a great deal btw)

[Bonedaddy]

OK.  It's odd though.  Whatever it is before must be confusing adodb since it says it's connected but doesn't seem to be when it tries to do that.

If I do a DB->disconnect() and then connect() in select_query() I don't see the problem (but of course that's not a good solution.)  I'm not sure why DB->isConnected() returns true when it definitely doesn't seem connected.

I'll check out the query log.  UPDATE: I don't seem to get a query log with that  define.  I don't see QUERY_LOG or query_log in any of the php files either?

My max_allowed_packet is 32M.  wait_timeout is the default 28800 which would seem more than enough?

I'm not sure how many sets.  I have autodownload turned on and around 30 groups or so configured.

Thanks!

[Styck]

FYI:

If you're running the latest (svn) version, there's a QUERY_LOG check in db.class.php, and it will write the results to the QUERY_LOG_FILE (this should be set to '/tmp/urd_query.log' in defines.php).