Memory exhausted

[spearhead]

I still have to fix the installer so you can reuse the same name ...

Thanx for your patience. And don't hesitate to drop by in case you run into any troubles, or got some ideas how to improve in general 

[spearhead]

Did you have the package php5-adodb installed? Cause I can reproduce the error exactly when I installed it. It's gone again when I remove the package.

(the package is an accelerator for php adodb written in C and not necessary for URD)

[fusa]

yes it is installed

[fusa]

I might have found another problem.  I use newshosting, which has 600 day retention.  I have set URD's expire to 365 for alt.binaries.emulators.pinball.  Although URD only displays the binaries from the last month.  There are other binaries that were posted there from 154 days ago and 208 days ago.  Does URD think that if there is a large gap of no postings, the newsgroup has no more posts and quit retrieving binaries?

[spearhead]

I might have found another problem.  I use newshosting, which has 600 day retention.  I have set URD's expire to 365 for alt.binaries.emulators.pinball.  Although URD only displays the binaries from the last month.  There are other binaries that were posted there from 154 days ago and 208 days ago.  Does URD think that if there is a large gap of no postings, the newsgroup has no more posts and quit retrieving binaries?

Urd retrieves all headers until it considers a bunch too old.  It doesn't look at the gap in between.

Something may have gone wrong in the update even tho URD should pick up from there then. Is there anything in the log file or in admin/tasks that shows what happened?

[fusa]

I purged a.b.emulators.pinball, then updated.  Still only 3 weeks of binaries, the following are the logs:

From Admin tasks:

Description    Progress    Status    ETA    Added    Last updated    Comment    Actions
Expiring alt.binaries.emulators.pinball    100%    Finished       22:36:09    22:36:10    Deleted 0 articles    
Generating sets for alt.binaries.emulators.pinball    100%    Finished       22:35:56    22:36:09    Generated sets data complete    
Updating alt.binaries.emulators.pinball    100%    Finished       22:34:47    22:35:56    1339 articles/s - total 65534 articles    
Purging alt.binaries.emulators.pinball    100%    Finished       22:34:31    22:34:32    Finished


From urdd.log

Jul 20 22:34:47 server.com urdd: INFO Connecting to NNTP server: ssl:news.newshosting.com:563
Jul 20 22:34:49 server.com urdd: INFO using authentication as user
Jul 20 22:34:49 server.com urdd: NOTICE Selecting group: alt.binaries.emulators.pinball
Jul 20 22:35:56 server.com urdd: NOTICE Updating binary info for alt.binaries.emulators.pinball
Jul 20 22:35:58 server.com urdd: NOTICE Total is 2195
Jul 20 22:36:08 server.com urdd: NOTICE Updating set info for alt.binaries.emulators.pinball
Jul 20 22:36:09 server.com urdd: NOTICE Updating set info for alt.binaries.emulators.pinball complete
Jul 20 22:36:09 server.com urdd: INFO Deleting 0 binaries

[spearhead]

I don't see anything particularly odd in there. But it seems to find only 2195 headers. There probably isn't posted much.

Does it work on groups with more posts?

[fusa]

The binaries that are 155 days old is a total of 2.9gb , the 209 day old posts is over 400 total binaries.
I didn't notice this in other groups.

[spearhead]

http://img514.imageshack.us/img514/6475/screenshotpniball.jpg
is what I got on 300 days on that group.

[fusa]

This is what I got:
http://img714.imageshack.us/img714/2122/screenshotach.png
Seems like for some reason URD isn't detecting complete sets as well as yours is.  All of my listings for that group show partial sets, and only lists the ones at 3 weeks.  Yours show complete sets and older sets.

[spearhead]

can you post (or PM) the output of the help/debug page (with URDD online)

Also can change to debug mode (in admin/config - loglevel) and post/pm the output from the command that says update XXX (xxx is the group number) till it is finished with generating sets? Please purge the group beforehand.

[Styck]

Hi,
What I see is that Spearhead has older posts (30w) than you (only 3w).

(Spearhead probably filters out stuff that is incomplete as a preference).

Are you sure your newsserver has a retention that matches 30 weeks?

[fusa]

Newshostings retention is 600 days.

urdd.log (to big to posting here):
http://pastebin.org/411006


Help/Debug:

PHP Settings
System name   Linux server.com 2.6.32-23-generic #37-Ubuntu SMP Fri Jun 11 07:54:58 UTC 2010 i686 (none)
PHP_version   5.3.2-1ubuntu4.2 (should be at least 5.1.2)
Loaded php.ini file   /etc/php5/apache2/php.ini

URD svn revision   No svn used

Safe_mode   off (should be off)
Magic_quotes_gpc   off (should be off)
Safe_mode_gid   off (should be off)
Register_globals   off (should be off)

File_uploads   on (should be on)

Post_max_size   64M (Should be 2M or more)
Upload_max_filesize   64M (should be 2M or more)
Memory_limit   512M (should be 128M or more)

Allow_url_fopen   on (should be on)

These settings may cause trouble; empty values should always work tho:
Open_basedir   ''
Disable_functions   ''
Disable_classes   ''

Some basic system settings useful for debugging download directory problems
Apache userid   www-data (33)
Apache groupid   www-data (33)
URD Install path   /usr/share/urd
download_dir   /home/user/Downloads/
dl_dir_rights   drwxrwxrwx (0777)
URDD groups setting   
URDD permissions setting   0644

To fix problems to do with download directory settings

On the command line type:
# sudo chgrp -R www-data '/home/user/Downloads/'
# sudo chmod -R g+rwx '/home/user/Downloads/'

Neither of the commands should give any errors. Restart URDD. In case you still run in to trouble re-run the commands again after starting URDD. Also double check the setting value for group in admin/config ("URDD groups setting" above) and the permissions setting in admin/config ("URDD permissions setting" above). These values should be either blank or group = www-data permissions = 0664

Debug output of URD settings
Global configuration
URD_version   1.0.5
admin_email   user3@hotmail.com
allow_global_scripts   0
allow_user_scripts   0
auto_download   0
auto_expire   1
auto_login   
auto_reg   0
check_nntp_connections   1
cksfv_path   /usr/bin/cfv
clean_db_age   3
clean_dir_age   3
compress_nzb   0
db_intensive_maxthreads   1
default_expire_time   5
default_language   english.php
default_template   default
dir_cd   
dlpath   /home/user/Downloads/
expire_incomplete   0
expire_percentage   80
file_path   /usr/bin/file
forumpassword   xxxxxx
forumusername   
global_hidden_files_list   
global_hiddenfiles   0
group   
group_filter   alt.bin*
gzip_pars   -q -c
gzip_path   /bin/gzip
index_page_root   browse
keep_interesting   0
log_level   7
max_dl_name   42
maxbuttons   10
maxdl   400
maxexpire   365
maxfilesize   0
maxheaders   10000
maxul   100
nice_value   0
nntp_maxdlthreads   0
nntp_maxthreads   8
par_pars   c -q
parse_nfo   1
period_cd   0
period_cdb   0
period_getinfo   0
period_ng   0
period_opt   0
period_sendinfo   0
period_update   8
permissions   0644
preferred_server   20
queue_size   20000
rar_pars   a -ed -inul -idp -m5 -r -ep1 -y
rar_path   /usr/bin/rar
register   0
replacement_str   
run_update   0
scheduler   on
sendmail   0
settype_0   %T %C %P %n
settype_1   %T %C %P %n %(y) %m %a %l %s %

• %N

settype_100   %T %C %P %n
settype_2   %T %C %P %n %(y) %f %g %N
settype_3   %T %C %P %n %f %g %

• %N

settype_4   %T %C %P %n %(o) %N
settype_5   %T %C %P %n %e %m %a %

• %N

settype_6   %T %C %P %n - %A %(y) %f %g
settype_7   %T %C %P %n %(y) %m %a %l %s %

• %N

settype_8   %T %C %P %n %(o) %N
settype_9   %T %C %P %e %n %m %a %

• %N

shaping   0
socket_timeout   30
subdownloader_pars   -c -q --rename-subs -D
subdownloader_path   /usr/bin/subdownloader
tar_path   /bin/tar
time1_cd   
time1_cdb   
time1_getinfo   
time1_ng   
time1_opt   
time1_sendinfo   
time1_update   12
time2_cd   
time2_cdb   
time2_getinfo   
time2_ng   
time2_opt   
time2_sendinfo   
time2_update   0
total_max_articles   0
trickle_path   /usr/bin/trickle
un7zr_pars   x -y -bd
un7zr_path   /usr/bin/7za
unace_pars   x -y
unace_path   /usr/bin/unace
unarj_pars   x -y -i
unarj_path   /usr/bin/arj
unpar_pars   r -q
unpar_path   /usr/bin/par2
unrar_pars   x -y -kb -o- -c- -idp
unrar_path   /usr/bin/rar
unzip_pars   -qq -n
unzip_path   /usr/bin/unzip
update_text   
update_type   0
update_version   
urdd_host   localhost
urdd_maxthreads   10
urdd_pars   
urdd_path   /usr/share/urd/urdd.sh
urdd_port   11666
urdd_restart   1
urdd_startup   100
url   http://localhost/urd/
usenet_server   
webdownload   0
yydecode_pars   -e -b -f
yydecode_path   /usr/bin/yydecode
yyencode_pars   -q -m130000 -o
yyencode_path   /usr/local/bin/yencode

Session info
last_login   1279689888
last_active   1279735594
menudisplay   1
statusdisplay   1
challenge   ch2Jixyk5c4AHVoP
challenge_timeout   1279820176

 

User preferences
blocked_terms   
button1   none
button10   none
button2   none
button3   none
button4   none
button5   none
button6   none
button7   none
button8   none
button9   none
cancel_crypted_rars   0
defaultsort   date DESC
delete_files   1
download_text_file   1
format_dl_dir   
global_scripts   
hidden_files_list   
hiddenfiles   0
index_page   manual
language   english.php
mail_user   0
mail_user_sets   0
maxsetname   80
maxsetsize   0
minngsize   1
minsetsize   0
poster_email   
poster_name   
pref_level   10
rarfile_size   1000
recovery_size   5
screen_height   600
screen_width   800
search_terms   
search_type   LIKE
setcompleteness   0
setsperpage   100
skip_int   0
subs_lang   en, nl
taskmax1   5
taskmax2   8
template   default
unpar   1
unrar   1
use_auto_download   0
user_scripts   

Usenet Servers
id   20
name   Newshosting
hostname   news.newshosting.com
threads   15
connection   ssl
port   563
plain_port   119
secure_port   563
authentication   1
priority   10
retention   0
posting   1
compressed_headers   0
username   user3
password   xxxxxx

 

URDD startup test results
Required module curl   Succeeded -- Required PHP module loaded :curl
Optional module openssl   Succeeded -- Optional PHP module loaded :openssl
Required module pcntl   Succeeded -- Required PHP module loaded :pcntl
Required module pcre   Succeeded -- Required PHP module loaded :pcre
Required module posix   Succeeded -- Required PHP module loaded :posix
Required module sockets   Succeeded -- Required PHP module loaded :sockets
Required module SPL   Succeeded -- Required PHP module loaded :SPL
Optional module xmlrpc   Succeeded -- Optional PHP module loaded :xmlrpc
Required module gmp   Succeeded -- Required PHP module loaded :gmp
Optional module xmlreader   Succeeded -- Optional PHP module loaded :xmlreader
Optional module xmlwriter   Succeeded -- Optional PHP module loaded :xmlwriter
xml_parser_create   Succeeded -- Required modules has been loaded
Directory /home/user/Downloads/tmp/   Succeeded -- Directory usable
Directory /home/user/Downloads/done/   Succeeded -- Directory usable
Directory /home/user/Downloads/preview/   Succeeded -- Directory usable
Directory /home/user/Downloads/nzb/   Succeeded -- Directory usable
Directory /home/user/Downloads/spool/   Succeeded -- Directory usable
Directory /home/user/Downloads/scripts/   Succeeded -- Directory usable
Directory /home/user/Downloads/post/   Succeeded -- Directory usable
Magpie Cache dir   Succeeded -- Magpie directory accessable
Memory limit   Succeeded -- Memory limit above 128M
Server: news.newshosting.com:563   Succeeded -- Server: news.newshosting.com:563 enabled with 15 connections
urdland   Failed -- No URDLand account provided. Extra set info cannot be received

URDD settings: Daemon online
URDD Status
Uptime: 0 days 2:15:29
Queued: 0
Running: 0
Scheduled: 6
Users: 1

URDD settings
urdd_port : 11666
urdd_listen_host : 127.0.0.1
urdd_listen_host6 : ::1
urdd_logfile : /tmp/urdd.log
urdweb_logfile : /tmp/urdweb.log
urdd_log : file|stderr
urdd_min_loglevel : 7
urdd_daemonise : FALSE
urdd_debug_level : 255
urdd_timeout : 30
urdd_pidfile : /usr/share/urd/urdd.pid
urdd_uid : urd
urdd_gid : urd
scheduler : TRUE
test_servers : FALSE
test_urdland : TRUE
connect_urdland : TRUE
old_update : FALSE
disable_nntp : FALSE
log_file : /tmp/urdd.log
urdd_restart : TRUE
check_nntp_connections : 1
update_disable_keys : TRUE
find_servers : FALSE

URDD version
1.0.5 (stable)

URDD jobs, queue, threads, users
Threads:
Queue:
Jobs:
#1. @ 07/24/10 12:00:00 (repeat every 604800 sec) 8 u:user cmd:CHECK_VERSION
#2. @ 07/22/10 01:00:00 (repeat every 86400 sec) 9 u:user cmd:UPDATE 35
#3. @ 07/22/10 01:00:00 (repeat every 86400 sec) 10 u:user cmd:UPDATE 3413

[spearhead]

I don't see any weird stuff in either.

Estimating header count: 0 0  678393 743928 0 365
First guess : 65535

says it's getting 64K articles (which is slightly peculiar), with article numbers starting at 678393 and ending at 743928, expire set to 365 days.

You could validate this by  running this on the command line

Code:

nc news.newshosting.com 119
authinfo user USERNAME
authinfo pass PASSWORD
group alt.binaries.emulators.pinball
quit

You should see a line like this after the group command

211 1 737352 737352 alt.binaries.emulators.pinball

the two "737352"s here indicate the lowest and highest article number here. They should be 678393 743928  in your case.

(if you haven't nc installed, you can use telnet instead)

[fusa]

NC gave no response , but telnet worked:

Code:

telnet news.newshosting.com 119
Trying 209.197.12.12...
Connected to news.iad.newshosting.com.
Escape character is '^]'.
200 NNRP unlimited.newshosting.com Service Ready - support@newshosting.com (fe49.iad)
authinfo user USER
381 PASS required
authinfo pass PASS
281 OK (Posting Allowed)
group alt.binaries.emulators.pinball
211 65535 678393 743927 alt.binaries.emulators.pinball
quit
Goodbye
Connection closed by foreign host.

[spearhead]

The numbers seem to match, so urd is getting all the headers.

Code:

211 65535 678393 743927

So that is not the problem. It may simply be that you newsgroup doesn't have all the articles (maybe the group "crashed" on the server somehow). You said it'd work well with other groups right?

[fusa]

Newsleecher displays the older articles fine, that's why I could tell there were missing binaries.  I will check other groups closer to see how far back they go and if they are missing any.

[fusa]

I checked alt.binaries.emulators.mame and alt.binaries.emulators.misc both are set to 90 retention in urd and have binaries up to 12w old.  I tried adding alt.binaries.ebook (not a.b.e-book) and set it to 365 day retention, it showed binaries up to 51w old.

[spearhead]

Other groups seem to work fine then, it seems. Weird. Maybe there are other reasons URD drops these.

The image I posted showed only sets that are more than 90% complete or less than 3 days old (you can set that in admin/config)

Note that these completeness %s are just elaborate guesses and no absolute truths. You can click on it and chose "show set info" to get more details about a particular set.

Not much to go on for me tho.