Feature req (minor): "system default" / "no change" on file perms

[thorwak]

Minor issue (more like an annoyance, really) :

I have urdd save everything to a globally exported nfs that media readers etc on my network can access, rw. The NFS export is configured as all_squash, anonuid and anongid, so the owner change and mode change urdd tries to do on files generates annoying error messages sometimes (nothing breaks though so it's really not a big problem).

There should however, for completeness, be a setting like "system default" in the owner and permissions lists, or a checkbox "change file owner and permissions" that can be disabled to counter situations like this, me thinks

If noone feels like doing it I might fix it myself at some point (unless someone tells me not to, of course..)

[spearhead]

Is this at startup? or after downloading?

on downloading if you set the group to the blank value it won't touch it. I added this to permissions as well.

[thorwak]

Sorry, I actually can't remember - I just remember seeing it a couple of days ago and thinking I should turn permission changing off. Now when I finally got around to it I realized there was no such setting, so I reported it now. I have hardly done any downloading (only various header operations and "previewing" and such so probably it was something to do with this), but having a blank value for permissions like there is for group should do the trick.

Thanks!

[spearhead]

Well if you run into it again (probably at urdd startup then) please post the log file

[thorwak]

Will do. Again, sorry, I know this was a terrible bug report...

[thorwak]

(wrong thread)

[thorwak]

Here's an example:
2010-04-16   16:41:37   INFO   Downloaded 48 articles
2010-04-16   16:42:35   INFO   Successful download 55
2010-04-16   16:43:53   NOTICE   Cannot set permission; incorrect value
2010-04-16   16:43:53   INFO   Successfully decompressed download 55 (1 archives)

Group set to (blank) and owner "Don't change" so I guess something is still a bit off.

[spearhead]

Can you try again? I altered the error message so that it shows the value now
(ps works fine here  )

[thorwak]

Hmm something has happened here, urdd freaks out whenever I try to download something. Sometimes it had time to start DLing a bit, sometime not. I'm not sure yet whether the DB has gone bonkers (I did a major expire in a group, but not the one I try to DL from) or if it's one of the latest svn updates that has a problem... Here's what happens in the logs during a session.

Code:

Apr 16 23:54:24 urd urdd: INFO Connecting to NNTP server: usenetproxy.2600.hz:120
Apr 16 23:54:24 urd urdd: INFO using authentication as gnxxxxxx
PHP Warning:  mysql_connect(): Can't connect to MySQL server on 'urddb' (4) in /var/www/urd/functions/adodb/drivers/adodb-mysql.inc.php on line 364

Warning: mysql_connect(): Can't connect to MySQL server on 'urddb' (4) in /var/www/urd/functions/adodb/drivers/adodb-mysql.inc.php on line 364
Apr 16 23:54:24 urd urdd: CRITICAL An error occured during startup of URD daemon: Could not connect to database : mysql error: [0: Connection error to server 'urddb' with user 'urd_user'] in CONNECT(urddb, 'urd_user', '****', urddb)
Apr 16 23:54:24 urd urdd: CRITICAL
Apr 16 23:54:24 urd urdd: NOTICE is there another URD daemon running?
Apr 16 23:54:24 urd urdd: ERROR Exception: Could not execute SQL query "SELECT "value" FROM preferences WHERE "option" = 'urdd_startup' AND "userID" = 0" mysql error: [2006: MySQL server has gone away] in EXECUTE("SELECT "value" FROM preferences WHERE "option" = 'urdd_startup' AND "userID" = 0 LIMIT 2147483647")
Apr 16 23:54:24 urd urdd: ERROR
Apr 16 23:54:24 urd urdd: INFO shutdown handler called as parent
Apr 16 23:54:24 urd urdd: NOTICE Connecting to NNTP server.
Apr 16 23:54:24 urd urdd: INFO Connecting to NNTP server: usenetproxy.2600.hz:120
urd:~/urd-svn/urd# Apr 16 23:54:25 urd urdd: INFO using authentication as gnxxxxxx

From the logs it looks like the DB goes down but this is not happening in reailty, it's there all the time and the webui works fine etc, just that urdd shuts down and I have to start it again 

[spearhead]

Weird stuff... perhaps restarting the database server will help

[thorwak]

Tried restarting the mysqld, no difference. I guess if you can still DL stuff fine with current SVN it must be a local problem here, and then it kind of have to be related to the expire I did. 730 days down to 365 in a 78M group. That took a while and I kind of did a couple of "svn up" in the meantime so I guess all bets are off

The weird thing is though that urdd works fine until it starts forking (I guess?) to create DL threads.. You didn't touch any stuff related to that in the last hours?

Oh well, I'm too tired right now to do any meaningful diagnostic work. It'll have to wait until tomorrow

[spearhead]

does any of the other stuff work? update? expire?
the only change i made was with the perms error

[thorwak]

Yes.. I purged a group, updated, generated sets - and started downloading, fine so far... I guess there could be a very weird bug concerning very small DLs.. Now I tried one of the newly ones created (and a few 100 MB as usual), the smaller ones were stuff like posted NZB, single .flacs and the likes, 1-10 MB. Maybe threading breaks because it's not enough data for all threads? Can you try a very small DL?

I'll do some more testing here to pinpoint.

EDIT: I also have those broken DLs listed as active, even though urdd is idle (I stopped the big one manually and that was fine too)

EDIT2: Yep, seems to be size related... 8 threads on a big file from the same group that kept crashing urdd is fine, taking something that is around 1 MB crashes urdd. New bug I guess

[thorwak]

So, here is the data we really wanted

2010-04-13   07:51:16   DEBUG   fn:set_permissions() pid:6415 (8388608 8388608)
2010-04-13   07:50:49   DEBUG   fn:set_permissions() pid:6403 (8388608 8388608)

Edit: Really tired now, this is old stuff obviously

Edit2: I don't get the error anymore. I changed some permissions earlier when I was gonna watch some TV. I'll see if I can reproduce and get you the proper debug output tomorrow. Have to go to bed now. But that "small file" bug is pretty bad, maybe that has higher prio 

ttyl!

[thorwak]

I realize this thread has gone a bit offtopic, maybe we should continue in the "bug & quirks"-thread, but just making sure: Have you been able to reproduce the "small set DL crashes urdd"-bug?

[spearhead]

No everything works just perfectly alright here.

Can you switch to debug log level in admin/config? and repost the log?

[thorwak]

No everything works just perfectly alright here.

Can you switch to debug log level in admin/config? and repost the log?

Ok, will do it later since I have a big indexing job running right now that I don't want to crash. It's funny though you're not seeing the same.. Everything is working fine with the kind of DLs I've been doing so far, 8 NNTP connections at the same time, no problem. Then I select something very small (typically 1 MB) and it crashes as it tries to divide the small DL into several connections. I'm fairly sure it has to do with there not being enough posts to divide it over all the NNTP connections, maybe it's already finished once the other connections come up, or whatever.

I'll try to run a session with full debug tonight and see what comes out.

[spearhead]

alternatively you could check if there are corruptions in the db

[thorwak]

It seems to affect all groups I DL from and it makes no sense big DLs would work while small wouldn't, indexing and everything fine if the DB is corrupt... (I've epxerienced it being FUBAR once before, while it was in a VM, and it behaved comepletely different)

Checking the DB fully takes a long time as well since it's around 50 GB again, but of course I will make sure it's not something silly like that. Maybe I can put a completely clean install in a new DB actually. Will see. I wanna complete the freaking translation dammit

[spearhead]

well if a dl table is corrupt.... but if you can dl larger stuff, that's really weird. Does previewing work ok?

O and the plan is to release 1.0.4 on next monday 

[thorwak]

Yes, previewing an .NFO file works fine (just tried it again to make sure). On the other hand, this doesn't seem to ever try to use several connections? That seems to be the keys (new threads being started)

The DL table is fine I think (I removed all the broken tasks yesterday)

Cool with 1.0.4 being released. I'll make sure I finish the translation. At the manual now. Btw, you might want to go over the manual (and maybe the FAQ) since it's outdated - for instance the the description of scehduled updates seems plain wrong (so probably for an older version). I'll change to the best of my knowledge in the Swedish version but I think you should do the English "master" and then I'll reflect any changes in the Swedish one that might be missing and/or if I was wrong about something.