I checked alt.binaries.emulators.mame and alt.binaries.emulators.misc both are set to 90 retention in urd and have binaries up to 12w old.  I tried adding alt.binaries.ebook (not a.b.e-book) and set it to 365 day retention, it showed binaries up to 51w old.

Newsleecher displays the older articles fine, that's why I could tell there were missing binaries.  I will check other groups closer to see how far back they go and if they are missing any.

NC gave no response , but telnet worked:

Code:

telnet news.newshosting.com 119
Trying 209.197.12.12...
Connected to news.iad.newshosting.com.
Escape character is '^]'.
200 NNRP unlimited.newshosting.com Service Ready - support@newshosting.com (fe49.iad)
authinfo user USER
381 PASS required
authinfo pass PASS
281 OK (Posting Allowed)
group alt.binaries.emulators.pinball
211 65535 678393 743927 alt.binaries.emulators.pinball
quit
Goodbye
Connection closed by foreign host.

Newshostings retention is 600 days.

urdd.log (to big to posting here):
http://pastebin.org/411006


Help/Debug:

PHP Settings
System name   Linux server.com 2.6.32-23-generic #37-Ubuntu SMP Fri Jun 11 07:54:58 UTC 2010 i686 (none)
PHP_version   5.3.2-1ubuntu4.2 (should be at least 5.1.2)
Loaded php.ini file   /etc/php5/apache2/php.ini

URD svn revision   No svn used

Safe_mode   off (should be off)
Magic_quotes_gpc   off (should be off)
Safe_mode_gid   off (should be off)
Register_globals   off (should be off)

File_uploads   on (should be on)

Post_max_size   64M (Should be 2M or more)
Upload_max_filesize   64M (should be 2M or more)
Memory_limit   512M (should be 128M or more)

Allow_url_fopen   on (should be on)

These settings may cause trouble; empty values should always work tho:
Open_basedir   ''
Disable_functions   ''
Disable_classes   ''

Some basic system settings useful for debugging download directory problems
Apache userid   www-data (33)
Apache groupid   www-data (33)
URD Install path   /usr/share/urd
download_dir   /home/user/Downloads/
dl_dir_rights   drwxrwxrwx (0777)
URDD groups setting   
URDD permissions setting   0644

To fix problems to do with download directory settings

On the command line type:
# sudo chgrp -R www-data '/home/user/Downloads/'
# sudo chmod -R g+rwx '/home/user/Downloads/'

Neither of the commands should give any errors. Restart URDD. In case you still run in to trouble re-run the commands again after starting URDD. Also double check the setting value for group in admin/config ("URDD groups setting" above) and the permissions setting in admin/config ("URDD permissions setting" above). These values should be either blank or group = www-data permissions = 0664

Debug output of URD settings
Global configuration
URD_version   1.0.5
admin_email   user3@hotmail.com
allow_global_scripts   0
allow_user_scripts   0
auto_download   0
auto_expire   1
auto_login   
auto_reg   0
check_nntp_connections   1
cksfv_path   /usr/bin/cfv
clean_db_age   3
clean_dir_age   3
compress_nzb   0
db_intensive_maxthreads   1
default_expire_time   5
default_language   english.php
default_template   default
dir_cd   
dlpath   /home/user/Downloads/
expire_incomplete   0
expire_percentage   80
file_path   /usr/bin/file
forumpassword   xxxxxx
forumusername   
global_hidden_files_list   
global_hiddenfiles   0
group   
group_filter   alt.bin*
gzip_pars   -q -c
gzip_path   /bin/gzip
index_page_root   browse
keep_interesting   0
log_level   7
max_dl_name   42
maxbuttons   10
maxdl   400
maxexpire   365
maxfilesize   0
maxheaders   10000
maxul   100
nice_value   0
nntp_maxdlthreads   0
nntp_maxthreads   8
par_pars   c -q
parse_nfo   1
period_cd   0
period_cdb   0
period_getinfo   0
period_ng   0
period_opt   0
period_sendinfo   0
period_update   8
permissions   0644
preferred_server   20
queue_size   20000
rar_pars   a -ed -inul -idp -m5 -r -ep1 -y
rar_path   /usr/bin/rar
register   0
replacement_str   
run_update   0
scheduler   on
sendmail   0
settype_0   %T %C %P %n
settype_1   %T %C %P %n %(y) %m %a %l %s %

• %N

settype_100   %T %C %P %n
settype_2   %T %C %P %n %(y) %f %g %N
settype_3   %T %C %P %n %f %g %

• %N

settype_4   %T %C %P %n %(o) %N
settype_5   %T %C %P %n %e %m %a %

• %N

settype_6   %T %C %P %n - %A %(y) %f %g
settype_7   %T %C %P %n %(y) %m %a %l %s %

• %N

settype_8   %T %C %P %n %(o) %N
settype_9   %T %C %P %e %n %m %a %

• %N

shaping   0
socket_timeout   30
subdownloader_pars   -c -q --rename-subs -D
subdownloader_path   /usr/bin/subdownloader
tar_path   /bin/tar
time1_cd   
time1_cdb   
time1_getinfo   
time1_ng   
time1_opt   
time1_sendinfo   
time1_update   12
time2_cd   
time2_cdb   
time2_getinfo   
time2_ng   
time2_opt   
time2_sendinfo   
time2_update   0
total_max_articles   0
trickle_path   /usr/bin/trickle
un7zr_pars   x -y -bd
un7zr_path   /usr/bin/7za
unace_pars   x -y
unace_path   /usr/bin/unace
unarj_pars   x -y -i
unarj_path   /usr/bin/arj
unpar_pars   r -q
unpar_path   /usr/bin/par2
unrar_pars   x -y -kb -o- -c- -idp
unrar_path   /usr/bin/rar
unzip_pars   -qq -n
unzip_path   /usr/bin/unzip
update_text   
update_type   0
update_version   
urdd_host   localhost
urdd_maxthreads   10
urdd_pars   
urdd_path   /usr/share/urd/urdd.sh
urdd_port   11666
urdd_restart   1
urdd_startup   100
url   http://localhost/urd/
usenet_server   
webdownload   0
yydecode_pars   -e -b -f
yydecode_path   /usr/bin/yydecode
yyencode_pars   -q -m130000 -o
yyencode_path   /usr/local/bin/yencode

Session info
last_login   1279689888
last_active   1279735594
menudisplay   1
statusdisplay   1
challenge   ch2Jixyk5c4AHVoP
challenge_timeout   1279820176

 

User preferences
blocked_terms   
button1   none
button10   none
button2   none
button3   none
button4   none
button5   none
button6   none
button7   none
button8   none
button9   none
cancel_crypted_rars   0
defaultsort   date DESC
delete_files   1
download_text_file   1
format_dl_dir   
global_scripts   
hidden_files_list   
hiddenfiles   0
index_page   manual
language   english.php
mail_user   0
mail_user_sets   0
maxsetname   80
maxsetsize   0
minngsize   1
minsetsize   0
poster_email   
poster_name   
pref_level   10
rarfile_size   1000
recovery_size   5
screen_height   600
screen_width   800
search_terms   
search_type   LIKE
setcompleteness   0
setsperpage   100
skip_int   0
subs_lang   en, nl
taskmax1   5
taskmax2   8
template   default
unpar   1
unrar   1
use_auto_download   0
user_scripts   

Usenet Servers
id   20
name   Newshosting
hostname   news.newshosting.com
threads   15
connection   ssl
port   563
plain_port   119
secure_port   563
authentication   1
priority   10
retention   0
posting   1
compressed_headers   0
username   user3
password   xxxxxx

 

URDD startup test results
Required module curl   Succeeded -- Required PHP module loaded :curl
Optional module openssl   Succeeded -- Optional PHP module loaded :openssl
Required module pcntl   Succeeded -- Required PHP module loaded :pcntl
Required module pcre   Succeeded -- Required PHP module loaded :pcre
Required module posix   Succeeded -- Required PHP module loaded :posix
Required module sockets   Succeeded -- Required PHP module loaded :sockets
Required module SPL   Succeeded -- Required PHP module loaded :SPL
Optional module xmlrpc   Succeeded -- Optional PHP module loaded :xmlrpc
Required module gmp   Succeeded -- Required PHP module loaded :gmp
Optional module xmlreader   Succeeded -- Optional PHP module loaded :xmlreader
Optional module xmlwriter   Succeeded -- Optional PHP module loaded :xmlwriter
xml_parser_create   Succeeded -- Required modules has been loaded
Directory /home/user/Downloads/tmp/   Succeeded -- Directory usable
Directory /home/user/Downloads/done/   Succeeded -- Directory usable
Directory /home/user/Downloads/preview/   Succeeded -- Directory usable
Directory /home/user/Downloads/nzb/   Succeeded -- Directory usable
Directory /home/user/Downloads/spool/   Succeeded -- Directory usable
Directory /home/user/Downloads/scripts/   Succeeded -- Directory usable
Directory /home/user/Downloads/post/   Succeeded -- Directory usable
Magpie Cache dir   Succeeded -- Magpie directory accessable
Memory limit   Succeeded -- Memory limit above 128M
Server: news.newshosting.com:563   Succeeded -- Server: news.newshosting.com:563 enabled with 15 connections
urdland   Failed -- No URDLand account provided. Extra set info cannot be received

URDD settings: Daemon online
URDD Status
Uptime: 0 days 2:15:29
Queued: 0
Running: 0
Scheduled: 6
Users: 1

URDD settings
urdd_port : 11666
urdd_listen_host : 127.0.0.1
urdd_listen_host6 : ::1
urdd_logfile : /tmp/urdd.log
urdweb_logfile : /tmp/urdweb.log
urdd_log : file|stderr
urdd_min_loglevel : 7
urdd_daemonise : FALSE
urdd_debug_level : 255
urdd_timeout : 30
urdd_pidfile : /usr/share/urd/urdd.pid
urdd_uid : urd
urdd_gid : urd
scheduler : TRUE
test_servers : FALSE
test_urdland : TRUE
connect_urdland : TRUE
old_update : FALSE
disable_nntp : FALSE
log_file : /tmp/urdd.log
urdd_restart : TRUE
check_nntp_connections : 1
update_disable_keys : TRUE
find_servers : FALSE

URDD version
1.0.5 (stable)

URDD jobs, queue, threads, users
Threads:
Queue:
Jobs:
#1. @ 07/24/10 12:00:00 (repeat every 604800 sec) 8 u:user cmd:CHECK_VERSION
#2. @ 07/22/10 01:00:00 (repeat every 86400 sec) 9 u:user cmd:UPDATE 35
#3. @ 07/22/10 01:00:00 (repeat every 86400 sec) 10 u:user cmd:UPDATE 3413

This is what I got:
http://img714.imageshack.us/img714/2122/screenshotach.png
Seems like for some reason URD isn't detecting complete sets as well as yours is.  All of my listings for that group show partial sets, and only lists the ones at 3 weeks.  Yours show complete sets and older sets.

The binaries that are 155 days old is a total of 2.9gb , the 209 day old posts is over 400 total binaries.
I didn't notice this in other groups.

I purged a.b.emulators.pinball, then updated.  Still only 3 weeks of binaries, the following are the logs:

From Admin tasks:

Description    Progress    Status    ETA    Added    Last updated    Comment    Actions
Expiring alt.binaries.emulators.pinball    100%    Finished       22:36:09    22:36:10    Deleted 0 articles    
Generating sets for alt.binaries.emulators.pinball    100%    Finished       22:35:56    22:36:09    Generated sets data complete    
Updating alt.binaries.emulators.pinball    100%    Finished       22:34:47    22:35:56    1339 articles/s - total 65534 articles    
Purging alt.binaries.emulators.pinball    100%    Finished       22:34:31    22:34:32    Finished


From urdd.log

Jul 20 22:34:47 server.com urdd: INFO Connecting to NNTP server: ssl:news.newshosting.com:563
Jul 20 22:34:49 server.com urdd: INFO using authentication as user
Jul 20 22:34:49 server.com urdd: NOTICE Selecting group: alt.binaries.emulators.pinball
Jul 20 22:35:56 server.com urdd: NOTICE Updating binary info for alt.binaries.emulators.pinball
Jul 20 22:35:58 server.com urdd: NOTICE Total is 2195
Jul 20 22:36:08 server.com urdd: NOTICE Updating set info for alt.binaries.emulators.pinball
Jul 20 22:36:09 server.com urdd: NOTICE Updating set info for alt.binaries.emulators.pinball complete
Jul 20 22:36:09 server.com urdd: INFO Deleting 0 binaries

I might have found another problem.  I use newshosting, which has 600 day retention.  I have set URD's expire to 365 for alt.binaries.emulators.pinball.  Although URD only displays the binaries from the last month.  There are other binaries that were posted there from 154 days ago and 208 days ago.  Does URD think that if there is a large gap of no postings, the newsgroup has no more posts and quit retrieving binaries?

Completely working:
Newshosting

yes it is installed

First time I tried to install, I had to install a couple of other apps to continue (yencode, and the subtitles downloader)  Then I ran into lots of problems trying to use MySQL, with MySQLI eventually working.  Then at the step where you set the login for URD, I used a password with less than 8 characters, and had to set another login using a better password. 

This working install I used the same settings, so I'm not sure what is different.  Maybe at the stage where I set the password something was not done correctly.  Its possible I restarted installation to use the login I wanted to use originally.  Is this information set in the database?  I did drop the database a couple of times, but that might have been when I was having problems selecting MySQL.

I wish I could give better information of what made it work, but not really sure.  Thanks for your help.

After a complete removal of URD and reinstall, it seems to be working correctly.

I do have a couple of questions, is the max number of days articles kept in the database 365 days?  My provider has 600 day retention, but selecting that resulted in a bogus expire setting message. 

Also I am a little confused about the automatic updates and @time settings.  I have selected automatic updates hourly, but what should I have for @time?  is this minutes:seconds?  If its hour:minutes, does it ignore the hour part and just update at the :minutes?  For example if I set hourly, and for @time 1:15 will updated be at 12:15, 1:15, 2:15 etc?

I tried selecting both hourly, and setting a time, both gave that error

when I try adding through the web interface, it just pops up a window that says "Not a number"

Running it from command line it starts ok.  When I tried generating sets for all newsgroups, I received:

Code:

Jul 16 12:02:51 server.com urdd: INFO Connecting to NNTP server: ssl:news.newshosting.com:563
Jul 16 12:02:53 server.com urdd: INFO using authentication as user
Jul 16 12:02:53 server.com urdd: NOTICE We are started as root. Dropping priviliges
Jul 16 12:02:53 server.com urdd: NOTICE Dropping priviliges to group urd
Jul 16 12:02:53 server.com urdd: NOTICE Dropping priviliges to user urd
Jul 16 12:04:37 server.com urdd: INFO Connecting to NNTP server: ssl:news.newshosting.com:563
Jul 16 12:04:38 server.com urdd: INFO using authentication as user
Jul 16 12:04:38 server.com urdd: NOTICE Getting grouplist from server.
Jul 16 12:04:45 server.com urdd: NOTICE Done with grouplist from server. Inserted 0, Updated 7040
Jul 16 12:05:03 server.com urdd: WARNING Child Died? Could not execute SQL query "DELETE FROM parts_3324 WHERE "date" < 1278864303 " mysqli error: [1146: Table 'urddb.parts_3324' doesn't exist] in EXECUTE("DELETE FROM parts_3324 WHERE "date" < 1278864303 ")
Jul 16 12:05:03 server.com urdd: WARNING
Jul 16 12:05:03 server.com urdd: WARNING Child Died? Could not execute SQL query "SELECT count("binaryID") AS cnt FROM binaries_3324 WHERE "date" < 1278864303 " mysqli error: [1146: Table 'urddb.binaries_3324' doesn't exist] in EXECUTE("SELECT count("binaryID") AS cnt FROM binaries_3324 WHERE "date" < 1278864303 ")
Jul 16 12:05:03 server.com urdd: WARNING

I have already tried using my root user which results in the same problems.  For some reason some of the functions of URD just do nothing.  For example clicking on the icon to start URD daemon does nothing.  When trying to select a group to subscribe to does nothing (just gives a message empty value).  The only way I was able to connect to my news server was manually adding the values in the database, and removing the servers I didn't use.  I have tried using both Firefox and Opera, both have the same results.  I use MySQL for several other applications, so don't want to remove and reinstall.

The way I was successfully able to get through the 6 initial steps was to use urd as user name, set the password, and the root user, and password and use mysqli.  Just using the root user as the database user resulted in errors, and anythign other than mysqli also did too.  After that I edited dbconfig.php and changed the user to the root user, and used phpmyadmin to manually add my news server.  I tried manually subscribing to some groups, and was partially successful, but didn't know what to set for values to tell it when to update.

here are some more errors from /tmp/urdd.log:

Code:

Jul 15 19:55:10 server.com urdd: INFO using authentication as user
Jul 15 19:55:10 server.com urdd: NOTICE Selecting group: alt.binaries.emulators.misc
Jul 15 19:55:12 server.com urdd: ERROR Error while updating alt.binaries.emulators.misc: Could not execute SQL query "LOCK TABLE parts_3324 write" mysqli error: [1146: Table 'urddb.parts_3324' doesn't exist] in EXECUTE("LOCK TABLE parts_3324 write")
Jul 15 19:55:12 server.com urdd: ERROR  (0)
Jul 15 19:55:13 server.com urdd: INFO Connecting to NNTP server: ssl:news.newshosting.com:563
Jul 15 19:55:15 server.com urdd: INFO using authentication as user
Jul 15 19:55:15 server.com urdd: NOTICE Selecting group: alt.binaries.emulators.misc
Jul 15 19:55:17 server.com urdd: ERROR Error while updating alt.binaries.emulators.misc: Could not execute SQL query "LOCK TABLE parts_3324 write" mysqli error: [1146: Table 'urddb.parts_3324' doesn't exist] in EXECUTE("LOCK TABLE parts_3324 write")

The following is what I get whenever I navigate to localhost/urd

Code:

Fatal error: Uncaught exception 'Exception' with message '('default_language')' in /usr/share/urd/functions/functions.php:449 Stack trace: #0 /usr/share/urd/functions/checkauth.php(57): get_config(Object(DatabaseConnection), 'default_languag...') #1 /usr/share/urd/html/index.php(51): require_once('/usr/share/urd/...') #2 {main} thrown in /usr/share/urd/functions/functions.php on line 449

I get

Code:

ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'insert into stats (userid, action, value) values (2,1,1)' at line 2

Code:

SET sql_mode='ANSI_QUOTES';
insert into stats (userid, action, value) values (2,1,1);

worked fine, (adding the ; at the end of the first line).

When trying to configure URD I am unable to add new servers, I use Newshosting.  When navigating away from the URD page, I am unable to use URD anymore unless I remove the .installed file, empty the database and start over.

I just tried manually importing URD_mysql_db.sql then using MySQLi and was able to get to step 5/6 and 6/6

Using MySQL Improved:

Code:

Storing default usenet servers Failed
Could not execute SQL query "INSERT INTO usenet_servers ("name", "hostname", "port", "secure_port", "threads", "connection", "authentication", "username", "password", "priority", "retention", "compressed_headers", "posting") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)" Could not execute SQL query "INSERT INTO usenet_servers ("name", "hostname", "port", "secure_port", "threads", "connection", "authentication", "username", "password", "priority", "retention", "compressed_headers", "posting") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)" mysqli error: [1146: Table 'urddb.usenet_servers' doesn't exist] in EXECUTE("INSERT INTO usenet_servers ("name", "hostname", "port", "secure_port", "threads", "connection", "authentication", "username", "password", "priority", "retention", "compressed_headers", "posting") VALUES ('4usenet', 'news.4usenet.nl', 119, 563, 6, 'ssl', 1, 'XXXXXXX', 'XXX', 0, 0, 0, 0)")
Storing default search buttons Failed
Could not execute SQL query "INSERT INTO searchbuttons ("name", "image_url", "search_url") VALUES (?, ?, ?)" Could not execute SQL query "INSERT INTO searchbuttons ("name", "image_url", "search_url") VALUES (?, ?, ?)" mysqli error: [1146: Table 'urddb.searchbuttons' doesn't exist] in EXECUTE("INSERT INTO searchbuttons ("name", "image_url", "search_url") VALUES ('All4Divx', 'buttons/btn_all4divx.png', 'http://www.all4divx.com/subtitles/$q/any/1')")

The output is the following repeated several times:

Code:

#0 urd_autoload(ADORecordSet_ext_empty)
#1 spl_autoload_call(ADORecordSet_ext_empty)
#2 class_exists(ADORecordSet_ext_empty) called at [/usr/share/urd/functions/adodb/adodb.inc.php:1048]
#3 ADOConnection->_Execute(SET sql_mode='ANSI_QUOTES', ) called at [/usr/share/urd/functions/adodb/adodb.inc.php:1015]
#4 ADOConnection->Execute(SET sql_mode='ANSI_QUOTES', ) called at [/usr/share/urd/functions/db.class.php:406]
#5 DatabaseConnection->execute_query(SET sql_mode='ANSI_QUOTES') called at [/usr/share/urd/functions/db.class.php:379]
#6 DatabaseConnection->connect() called at [/usr/share/urd/functions/db.class.php:101]
#7 DatabaseConnection->__construct(mysql, localhost, user, password, urddb, 1) called at [/usr/share/urd/install/install.4.inc.php:169]
#8 include(/usr/share/urd/install/install.4.inc.php) called at [/usr/share/urd/install/install.php:207]
#0 urd_autoload(ADORecordSet_ext_empty)
#1 spl_autoload_call(ADORecordSet_ext_empty)
#2 class_exists(ADORecordSet_ext_empty) called at [/usr/share/urd/functions/adodb/adodb.inc.php:1048]
#3 ADOConnection->_Execute(SET sql_mode='ANSI_QUOTES', ) called at [/usr/share/urd/functions/adodb/adodb.inc.php:1015]
#4 ADOConnection->Execute(SET sql_mode='ANSI_QUOTES', ) called at [/usr/share/urd/functions/db.class.php:406]
#5 DatabaseConnection->execute_query(SET sql_mode='ANSI_QUOTES') called at [/usr/share/urd/functions/db.class.php:379]
#6 DatabaseConnection->connect() called at [/usr/share/urd/functions/db.class.php:409]
#7 DatabaseConnection->execute_query(SET sql_mode='ANSI_QUOTES') called at [/usr/share/urd/functions/db.class.php:379]
#8 DatabaseConnection->connect() called at [/usr/share/urd/functions/db.class.php:101]
#9 DatabaseConnection->__construct(mysql, localhost, user, password, urddb, 1) called at [/usr/share/urd/install/install.4.inc.php:169]
#10 include(/usr/share/urd/install/install.4.inc.php) called at [/usr/share/urd/install/install.php:207]

I am using MySQL, the only options I have is MySQL, MySQL with transactions, and MySQL Improved.  with Transactions also fails with the same errors, Improved gives a different error page.

I'm using PHP version:

PHP 5.3.2-1ubuntu4.2 with Suhosin-Patch (cli) (built: May 13 2010 20:01:00)
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies

on Ubuntu 10.04

Code:

php -r "throw new exception ('foo');"

gave:

Code:

Fatal error: Uncaught exception 'Exception' with message 'foo' in Command line code:1
Stack trace:
#0 {main}
  thrown in Command line code on line 1

Adding echo "$class\n"; gave the following:

Code:

ADORecordSet_ext_empty
ADORecordSet_ext_empty
ADORecordSet_ext_empty
...........
ADORecordSet_ext_empty
ADORecordSet_ext_empty
Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20 bytes) in /usr/share/urd/functions/autoincludes.php on line 74

I am getting this error in step 3/6 when trying to set up URD 1.05  "Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 3 bytes) in /usr/share/urd/functions/autoincludes.php  on line 73 "  I originally set the memory_limit for PHP to 128M and got this error, tried 256M and received the same, then tried 512, and also got the error.