New features for this great project.

[Arie]

Hi,
First of all, let me complement you with this great, and fast developing project. I have used it for a few months now, and it was exactly what I was looking for; a great web application for usenet.URD ROCKS! Also the Dutch translation is perfect, so there must be some 'cloggies' in the team
I do have a few requests to, hopefully, improve URD.
- Performance is an issue. When I download a large file, my system (Ubuntu server 8.0.4, CPU Intel Xeon 3.2Ghz, RAM 8GB) becomes very sluggish on the other web sites I'm hosting. If you need more info, please let me know.
- When I search for certain objects, let's say an album name, and the result comes up with all separate songs, I would like to be able to select them all, and download them with one click/button.
- When I browse sets, I would like the same page buttons on the bottom of the page, as on the top. This was the case in older versions, if I remember well.
- When I log off, it would be nice to not see the application buttons, because it's confusing. People may think they can actually control URD.

Many thanx in advance & keep up the good work!

[spearhead]

Thanx for your response

Both developers are Dutch. So that's where the Dutch translation comes from.

Performance is probably more likely due to bandwidth and not processor performance. You may want to play with the traffic shaping (admin/config). If not please provide more info. Your system should well enough be able handle URD without any problems.

We are currently contemplating how to improve the browse page. We have the same issues as you have.

Buttons would be easy to fix I guess.

Hiding the menu is not a problem too I guess. Then again, they easily see it won't work unless they log in.

[Styck]

Hey Arie

We've given you credits for the installation 'manual' on the download tab
Nice to drop by! Good points indeed, the performance issue might be due to the fact that URD is using quite a bit of PHP power, which could affect those websites you mentioned.

Very high on my todo-list is a community-based-browse page; where it's possible for URD users to combine and split sets, and provide feedback, and the other users can use that info to improve their sets. So basically someone merges a few sets, and everybody else will see the merged set.

This is probably going to take a few releases before it is implemented :p

[Arie]

Performance is probably more likely due to bandwidth and not processor performance. You may want to play with the traffic shaping (admin/config).

My maximum download bandwidth sometimes exceeds 18Mb, and my usenet server limits me to about 750Kb, so bandwidth shouldn't be a problem, right?
When I check my system with top, the main cpu activity is php, and yydecode, when downloading with URD. The average cpu load is about 10% per thread.
I wonder what the load is on other systems. Maybe others could look into this, and let us know. Just as an example - maybe not the right one - for torrents I run torrentflux-b4rt. With about 10 simultaneous download I have never had a comparable load on my server.
Turning down my download max to 400Kb didn't ease the load.

[spearhead]

nope probably not bandwidth then.

I'll check into it. What log level do you run URD on? I hope not debug... info or notice should be good.

Newsgroups are far less efficient than torrent qua overhead. It's text-based and has to be decoded. PHP isn't very good at doing number crunching, so that is offloaded to yydecode (which is a compiled C program). Anyway I'll look into it.

Also you might consider to add noatime to your /etc/fstab for the filesystem URD is storing info on, as well as mysql.

[Arie]

I have just upgraded to svn 0.5.3, noughty me The funny thing is that I now get full band width, 1877 KB/s!!! So the 'fact' that my usenet server limited me, was not quite right. My other web sites are responding a bit snappier as well.
Great and thanx again!

[spearhead]

Interesting as I can't recall making any significant changes there. At least in the downloading binaries part. I did make a fix in the updating of headers that sped things up a lot.

Anyway I'm glad it works for you!

[dvb3eak]

It would be nice to be able to run "follow up" script. Such a script would be called once the URD finished working on a task no matter if the decompression of par2repair failed or went successfully. I would propose to pass a path to the tmp/xx  ord done/xxx  directory with the stuff related to the task. In my case I am moving the files some place else, and sometimes fixing, but the point is - user can do some extra steps by just simple reconfiguration.

Once again, great job URD ROCKS !!! 

[spearhead]

It would be nice to be able to run "follow up" script. Such a script would be called once the URD finished working on a task no matter if the decompression of par2repair failed or went successfully. I would propose to pass a path to the tmp/xx  ord done/xxx  directory with the stuff related to the task. In my case I am moving the files some place else, and sometimes fixing, but the point is - user can do some extra steps by just simple reconfiguration.

Once again, great job URD ROCKS !!! 

this is gonna be a security nightmare. The script would have to run under the same priviliges as the webserver (www-data on ubuntu) or URD and won't have much access right outside webserver/urd activity.

Worse, putting user generated scripts on a server and then run them is possible way to break into a machine. putting some exploit code in /tmp (any upload will do) and running it from this script will do that. It's hardly possible to know what the script will do... putting some exploit code in /tmp (any upload will do) and running it from this script will do that.

So I'm not sure if this really is a good idea. Alternatively you might run it as a simple cron job... and polls the directories.

[dvb3eak]

Cron job is what i am doing today, but the problem is - how to recognize if URD is done with the directory (or set in tmp ) ?
It is often shows 100% and on GUI it is in "finished" group yet directory stays in tmp/

I recognize by running the full cycle (par2repair, unrar etc which is time , CPU, memory consuming).

can you at least set sort of a semaphore (file with certain name)
that marks a directory

[spearhead]

Cron job is what i am doing today, but the problem is - how to recognize if URD is done with the directory (or set in tmp ) ?
It is often shows 100% and on GUI it is in "finished" group yet directory stays in tmp/

I recognize by running the full cycle (par2repair, unrar etc which is time , CPU, memory consuming).

can you at least set sort of a semaphore (file with certain name)
that marks a directory

If the download is finished, it starts a new task to unrar/par2/etc it. So you'd have to wait for that too. I'll see if I can change the logic around... to change the dl to done after the unrar.

Ok I changed it so that if it is in done/ it is really done and won't touch the files any more

[dvb3eak]

I noticed that when urd picks an article to download (sometimes my provider kicks me out of nntp then I have to restart), it takes sets to download randomly. While it would be really nice to finish those that have only a few % to go. Let's say today I had an article that was 92% downloaded yesterday but was not downloading lately for some reason, but when It picked the next article to load it took a fresh one (100% to go). My disk space is limited so is the lifetime of articles on the server, it would be nice if articles have priorities depending on percentage of downloaded part. So 92 % downloaded should be higher priority than 10% or 0. Of course if a user hits ""start" that shall override any automatic decisions made be the program.

[dvb3eak]

If there is any way to recognize language the article is on, it would be nice to be able to filter by language. For example it happens to me sometimes I download an article for a day or two just to find out it is on German which I do not quite understand :-(

[Styck]

I feel your pain

URD doesn't know the language a set is in, you could try previewing an included .nfo file for the cases the set HAS a .nfo file.

With extended setinfo, there's a language field that should make it clear what language the file is in; however it needs to be filled in by someone first.

[dvb3eak]

Option to supress all articles with less than XXX% percent availability would be nice to have.
I am lazy  and I am not going to search all of these missing pieces for articles with bullets (balls) other than green or white.

I would certainly appreciate ability to just hide sets unless % - percent is equal or above 100%

URD rocks!!!


 

[spearhead]

I would certainly appreciate ability to just hide sets unless % - percent is equal or above 100%

That will be in the next version. I was only contemplating to have that a global value or per newsgroup.

Note orange sets should be ok to download too, if they have par files.

[spearhead]

Option to supress all articles with less than XXX% percent availability would be nice to have.
I am lazy  and I am not going to search all of these missing pieces for articles with bullets (balls) other than green or white.

I would certainly appreciate ability to just hide sets unless % - percent is equal or above 100%

This is in the latest svn revision. I added it as a global setting per user.