Yes I understand.

Other groups seem to work fine.
I'll observe for a few days and see what happens.
Thanks.

Must be eweka. The errors below are the reason I switched to urd1.1.0.
There are a lot of phony boneless groups there but this one has 2.8G of articles, so it must be the right one.

Oct 24 2011    08:39:21    ERROR    Error while updating alt.binaries.boneless: No article(s) selected (No articles selected) (420)
Oct 24 2011    08:39:21    ERROR    No article(s) selected (No articles selected) (420)
Oct 24 2011    08:39:20    WARNING    Cannot get messages:
Oct 24 2011    08:38:03    ERROR    Error while updating alt.binaries.boneless: No article(s) selected (No articles selected) (420)
Oct 24 2011    08:38:03    ERROR    No article(s) selected (No articles selected) (420)
Oct 24 2011    08:38:02    WARNING    Cannot get messages:
Oct 24 2011    08:36:58    ERROR    Error while updating alt.binaries.boneless: No article(s) selected (No articles selected) (420)
Oct 24 2011    08:36:58    ERROR    Error while updating alt.binaries.boneless: No article(s) selected (No articles selected) (420)
Oct 24 2011    08:36:58    ERROR    No article(s) selected (No articles selected) (420)
Oct 24 2011    08:36:58    ERROR    No article(s) selected (No articles selected) (420)
Oct 24 2011    08:36:57    WARNING    Cannot get messages:
Oct 24 2011    08:36:57    WARNING    Cannot get messages:

For some reason I've has issues in the past, especially with the boneless group. It would after some time just hang while updating. I would spend lots of time trying to figure out why the boneless group wouldn't update anymore. So in the end I chose a fresh install.

Anyway this morning it was the a.b.hdtv.x264 group, yesterday the a.b.movies. It is possible, since I did a 5-day-expire update it took too long or so to update. Update speed is ~400kB/s. So perhaps they have some time limit in place.

The update grouplist went ok and so does NZB downloading to a folder seen by hellanzb.

Oct 24 2011    04:03:44    WARNING    Cannot get messages:
Oct 24 2011    04:03:44    ERROR    Unexpected response (500): [Max connect time exceede.] (500)
Oct 24 2011    04:03:44    ERROR    Error while updating alt.binaries.hdtv.x264: Unexpected response (500): [Max connect time exceede.] (500)
Oct 24 2011    03:58:36    WARNING    Cannot get messages:
Oct 24 2011    03:58:36    ERROR    Unexpected response (500): [Max connect time exceede.] (500)
Oct 24 2011    03:58:36    ERROR    Error while updating alt.binaries.hdtv.x264: Unexpected response (500): [Max connect time exceede.] (500)
Oct 24 2011    02:55:10    WARNING    Cannot get messages:
Oct 24 2011    02:55:10    ERROR    Unexpected response (500): [Max connect time exceede.] (500)
Oct 24 2011    02:55:10    ERROR    Error while updating alt.binaries.hdtv.x264: Unexpected response (500): [Max connect time exceede.] (500)
Oct 24 2011    02:04:27    WARNING    Cannot get messages:
Oct 24 2011    02:04:27    ERROR    Unexpected response (500): [Max connect time exceede.] (500)
Oct 24 2011    02:04:27    ERROR    Error while updating alt.binaries.hdtv.x264: Unexpected response (500): [Max connect time exceede.] (500)
Oct 24 2011    01:40:24    WARNING    Cannot get messages:
Oct 24 2011    01:40:24    ERROR    Unexpected response (500): [Max connect time exceede.] (500)
Oct 24 2011    01:40:24    ERROR    Error while updating alt.binaries.hdtv.x264: Unexpected response (500): [Max connect time exceede.] (500)
Oct 24 2011    01:13:38    WARNING    Cannot get messages:
Oct 24 2011    01:13:38    ERROR    Unexpected response (500): [Max connect time exceede.] (500)
Oct 24 2011    01:13:38    ERROR    Error while updating alt.binaries.hdtv.x264: Unexpected response (500): [Max connect time exceede.] (500)
Oct 24 2011    00:30:37    ERROR    Unexpected response (500): [Max connect time exceede.] (500)
Oct 24 2011    00:30:37    ERROR    Error while updating alt.binaries.hdtv.x264: Unexpected response (500): [Max connect time exceede.] (500)
Oct 24 2011    00:30:37    WARNING    Cannot get messages:
Oct 23 2011    22:04:37    ERROR    Error while updating alt.binaries.hdtv.x264: Unexpected response (500): [Max connect time exceede.] (500)
Oct 23 2011    22:04:37    WARNING    Cannot get messages:
Oct 23 2011    22:04:37    ERROR    Unexpected response (500): [Max connect time exceede.] (500)
Oct 23 2011    21:49:38    WARNING    Cannot get messages:
Oct 23 2011    21:49:38    ERROR    Unexpected response (500): [Max connect time exceede.] (500)
Oct 23 2011    21:49:38    ERROR    Error while updating alt.binaries.hdtv.x264: Unexpected response (500): [Max connect time exceede.] (500)
Oct 23 2011    20:57:40    WARNING    Cannot get messages:
Oct 23 2011    20:57:40    ERROR    Unexpected response (500): [Max connect time exceede.] (500)
Oct 23 2011    20:57:40    ERROR    Error while updating alt.binaries.hdtv.x264: Unexpected response (500): [Max connect time exceede.] (500)
Oct 23 2011    18:30:36    ERROR    Error while updating alt.binaries.hdtv.x264: Unexpected response (500): [Max connect time exceede.] (500)
Oct 23 2011    18:30:36    ERROR    Unexpected response (500): [Max connect time exceede.] (500)

The x264 hasn't updated yet.
I just added the boneless group with a 1-day expire.
If the error persists I'll try the $e->gettraceasstring().

ERROR   Error while updating alt.binaries.movies: Unexpected response (500): [Max connect time exceede.] (500)

I decided to upgraded URD from 1.0.8 to 1.1.0
I deleted the /usr/share/urd dir, dpkg -r and then -i on the .deb file.
Then rebuild the database/accounts from scratch.
...Pretty much like I did the previous versions.

Any idea how to analyze this?

Thanks - again overlooked that 

Has this been removed in 1.0.6?

  There is a setting (not under preferences!): 'block popup windows'

I already did that. The new install was indeed behaving odd. But the problem persists.
It just hit me... I switched from firefox to safari and can't get it working on safari, whatever setting for tabs I use:

'Open pages in tabs instead of windows:
- never
- automatically
- always

Any ideas?

..used to (v1.0.5) open a new window/tab where the NFO was shown but now only downloads the NFO and can be viewed in the ViewFiles>PreviewFiles section.
Is there some setting to restore the old behavior? I.e. instantly see the NFO?

btw, just switched to 1.0.6 and am as always very impressed by your good work. Thanks!

Oops haven't looked there in a while. Should do the trick, thanks.

Spam is getting worse by the day. Today I had 17000 sets of .rar files <5MB.
Would it be possible to add a flexible filter to move all these directly to the hidden section?

So I had a hdd crash and installed a fresh db.
I noticed a few oddities in the setup of urd:
- Startup of urdd.sh fails due to yyencode not being installed. Why is this fatal?
- During setup, a password is asked to create >8chars and at least 2 numbers. After importing my global setup, the password had changed back to the old one.

Thanks for the info. I decided to take the quick route to get it over with and delete the whole db.

Got a hdd crash. urddb_svn is actually not on the crashed hdd but probably due to the IDE (yes IDE) controller things got messy.

Could not execute SQL query "LOCK TABLE parts_18318 write" mysql error: [144: Table './urddb_svn/parts_18318' is marked as crashed and last (automatic?) repair failed] in EXECUTE("LOCK TABLE parts_18318 write")

What to do?
Due to this, urd is cycling activating/failing/pausing the boneless update.

Edit: svn 1628

I put up the urdlog generated by ./urdd.sh -F >urdlog 2>&1
http://85.146.213.173/homepage/_downloads/misc/
Not sure if it's any good 'cause the expire went smooth.

With the last /tmp/urdd/log the problem was also gone.
I also put it up.

But I anyway activated the expire-on-update and the activity block has gone now for quite some time. This is why I initially discarded this option.
The mysql processlist:
+--------+-------+-----------+-----------+---------+------+--------------+------------------------------------------------------------------------------------------------------+
| Id     | User  | Host      | db        | Command | Time | State        | Info                                                                                                 |
+--------+-------+-----------+-----------+---------+------+--------------+------------------------------------------------------------------------------------------------------+
| 561878 | Reab4 | localhost | urddb_svn | Query   | 1317 | Sending data | DELETE FROM parts_3 WHERE "date" < 1274717001  AND "binaryID" NOT IN ( SELECT "binaryID" FROM userse |
| 561913 | Reab4 | localhost | urddb_svn | Query   | 1309 | Locked       | DELETE FROM parts_13955 WHERE "date" < 1274717009  AND "binaryID" NOT IN ( SELECT "binaryID" FROM us |
| 563896 | Reab4 | localhost | urddb_svn | Sleep   |  803 |              | NULL                                                                                                 |
| 563902 | Reab4 | localhost | urddb_svn | Sleep   |  802 |              | NULL                                                                                                 |
| 563903 | Reab4 | localhost | urddb_svn | Query   |  801 | Locked       | SELECT COUNT(setdata."ID") AS cnt  FROM setdata  LEFT JOIN usergroupinfo ON (usergroupinfo."groupid" |
| 563904 | Reab4 | localhost | urddb_svn | Sleep   |  801 |              | NULL                                                                                                 |
| 563905 | Reab4 | localhost | urddb_svn | Sleep   |  801 |              | NULL                                                                                                 |
| 563906 | Reab4 | localhost | urddb_svn | Sleep   |  801 |              | NULL                                                                                                 |
| 563917 | Reab4 | localhost | urddb_svn | Sleep   |    5 |              | NULL                                                                                                 |
| 563919 | Reab4 | localhost | urddb_svn | Sleep   |  106 |              | NULL                                                                                                 |
| 563920 | root  | localhost | NULL      | Query   |    0 | NULL         | show processlist                                                                                     |
+--------+-------+-----------+-----------+---------+------+--------------+------------------------------------------------------------------------------------------------------+
11 rows in set (0.00 sec)


Why are some locked?
Also the process list doesn't seem to change (at all).
It just returned and firefox became responsive again.

About an hour ago I web-restarted and the problem persisted.
Now I started it with ./urdd.sh -F >urdlog 2>&1 and the expire went smooth.
After a web-start the problem is also gone.

I'll re-evaluate the expire option. Sorry for the inconvenience.

Edit: typo

Btw, I test the expire by clicking expire in the newsgroup list for a small group. It then seems to crash. I haven't tried the expire after update yet because of the major system load in my last experience.
 
ps -ax | grep php yields:
20091 ?        Ss     4:54 /usr/bin/php /usr/share/urd_svn/urdd/urdd.php -D

In admin/control there are no threads running. Probably that's why it's not responding well to cancel/pause.

With a 1% expire processbar on small group:
+--------+-------+-----------+-----------+---------+------+-------+------------------+
| Id     | User  | Host      | db        | Command | Time | State | Info             |
+--------+-------+-----------+-----------+---------+------+-------+------------------+
| 552536 | Reab4 | localhost | urddb_svn | Sleep   |    0 |       | NULL             |
| 552646 | root  | localhost | NULL      | Query   |    0 | NULL  | show processlist |
| 552683 | Reab4 | localhost | urddb_svn | Sleep   |    1 |       | NULL             |
| 552684 | Reab4 | localhost | urddb_svn | Sleep   |    1 |       | NULL             |
| 552687 | Reab4 | localhost | urddb_svn | Sleep   |    0 |       | NULL             |
| 552688 | Reab4 | localhost | urddb_svn | Sleep   |    0 |       | NULL             |
+--------+-------+-----------+-----------+---------+------+-------+------------------+
6 rows in set (0.03 sec)

But this is actually not the reason I discarded the expire-after-update option.

Hmmm.
It's still at 1%, since 9:36
Normally the expiring action causes quite a load on my system. In this particular testcase I couldn't cancel it but killed it with "clear ALL volatile information"
Just tried it again - same results.
Tried it on another small group - same results.
Page responsiveness (firefox) is sluggish during this action and the expire task cannot be canceled nor paused.
Any ideas what to look for in the log?

Yes they are both fixed.
1.0.5 seems to work like a well-oiled snappy machine over here.

Well...except for the expire-after-update, which Which takes quite a while, even on small groups. So I don't use it. This means that the database is getting larger and larger by the day. Daily cleanup options do not include limiting the database to discard all data >some days  (I think).
Would be nice to have this option to run it at night to keep the database small.
Do you think it would be useful?
On the other hand, I haven't had problems with the large database so far.

Edit: To test I just expire on a.b.m.cooking (59045 posts). After 1 hour still at 1% progress.

I think it looks much more informative now, even though there is no ETA.
Thanks!

Some cosmetics...
- When a make-nzb is canceled, the task is gracefully ended but an empty/invalid file is still made.
- In the GroupSets page..when only the last tab of multiple tabs is globally hidded, an empty list/tab is returned. Would be nice to show the now-last tab or the first tab. However clicking search again shows the first tab again so no major issue.

I have to take off my hat - the nzb solution is most elegant.
I haven't had a problem yet so it seems solved. Thanks!

A drawback of having 1 Maximum database intensive threads is that when it's busy, no other (short-time) intensive thread may get processed for a while, so I set it to 2.

Another word about the progress bar; It has been discussed before...
Is it possible distinguish (add text above progress bar) between:
- 'updating headers' or 'downloading headers'
- 'generating sets' or 'processing headers'
As the end user is looking at the progress bar nearing the end with 'updating' and then seeing that it starts all over again due to generating sets is really depressing.
 

Ah I once tried the php -f update_db_xxx_to_yyy_mysql.sql which gave errors.
I'll remember it the next time. Just updated and imported the settings which went ok, except for 'group not found' message. This is normal as the grouplist was not complete. Seems a chicken<>egg problem.

It should actually be me thanking you for the good support. I use the program daily with satisfaction. There are some oddities which I don't understand but am still wondering if it's me or the program.
I'm glad I can be of some assistance to you in perfecting the program.

[installation script 4/6]

Make_NZB was giving me real headaches so just decided to update from svn to 1554.
Noticed a change in update_db1.0.4_to_1.0.5 so removed the .installed and start from scratch. Also because the db was growing too large and I still haven't found out how to contain the size of it. Whatever setting I do, it grows by the day and contains data older than the clean database settings (age of remove files etc.). Anyway, that's not why I'm bitching.
The installation script 4/6 yields this error:

Could not execute SQL query " DROP TABLE IF EXISTS `post_messages` CREATE TABLE `post_messages` ( `id` bigint(20) NOT NULL auto_increment, `userid` int(10) NOT NULL default '0', `groupid` int(10) NOT NULL default '0', `subject` varchar(512) NOT NULL default '', `poster_id` varchar(512) NOT NULL default '', `poster_name` varchar(512) NOT NULL default '', `message` text DEFAULT '' NOT NULL, PRIMARY KEY (`id`), )ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci" mysql error: [1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'CREATE TABLE `post_messages` ( `id` bigint(20) NOT NULL auto_increment, ' at line 1] in EXECUTE(" DROP TABLE IF EXISTS `post_messages` CREATE TABLE `post_messages` ( `id` bigint(20) NOT NULL auto_increment, `userid` int(10) NOT NULL default '0', `groupid` int(10) NOT NULL default '0', `subject` varchar(512) NOT NULL default '', `poster_id` varchar(512) NOT NULL default '', `poster_name` varchar(512) NOT NULL default '', `message` text DEFAULT '' NOT NULL, PRIMARY KEY (`id`), )ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci")
 

Not quite right. You end up with a status quo where there's an empty progress bar in the activity block and some MAKE_NZB activities with 0%.

See the screenshot:
Screen shot 2010-05-16 at 13.34.12.png
urdlog_20100516_1335.zip

But yes the logfile isn't humongous anymore and increasing the nr of total threads to begin with seems to solve it.
But I think (speculation) even with the higher amount of threads it can still happen if a (few?) MAKE_NZB is initiated while all threads are already used at that moment.

Edit:
On top of that, all new NZBs added are part of this 'forgotten' thread and are never executed.