Hashing in DC++ Options

[Grey]

I downloaded DC++ today to poke around in it in preparation for the next respawn and holy shit hashing takes a long time. So this post is a warning, do not leave hashing until the last minute! I'll be sharing roughly 1.1 terabytes of 'legit backups' and it's predicting hashing with be done in...75 hours!

Just a word of caution

-Grey

[Alioth]

So I was going to make some smart-arse reply to this thread regarding the topic of hashing as it has been thoroughly covered in another thread when to my surprise, said thread has seemed to disappeared. I'm talking about the big long one with the debate on which version of DC++ to use and the whole hashing thing. Something about DC++ and hashing should probably go in Tank's awesome Jumbo Q & A thread if the lords and ladies of Respawn do wish it

TL;DR Yes we know about hashing. Use version 0.674 or higher plz.

[deano]

Alioth: We all do not know about hashing.

If I see one person hashing at the event, you owe me pizza. Deal?

[Alioth]

Alioth: We all do not know about hashing.

If I see one person hashing at the event, you owe me pizza. Deal?

I owe you a kick in the nuts!
You didn't read my all my post, did you?...

----------------
Now playing: Bluejuice - Hunnamunnafeeb

[R4N]

The DC++ thread dissapeared for a reason, feel free to discuss the various methods of hashing, but if this thread delves IN ANY WAY into illegal file sharing, i delete it

fair warning

[Grey]

Well well, it seems poor predictive computing has struck again. After coming home from a night out, I found the hashing complete, only a few hours after I posted. I looked around for a DC++ thread and there are quite a few links to dead forum posts but I couldn't for the life of me find an active post. All in all, hashing takes only a few, about 5ish hours, to do for ~1tb across multiple drives and gogo sticky DC++ topic

-Grey

[Chopz]

Hashing all depends on the speed of your computer. for instance my 1.4tb took about 2 hours, Please use .674 - it hashes, .;750 doesnt and therefore it cant be taken from multiple computers. with hashed files, if someone else has the same file and they are both hashed, dc will download it from both places and give you a faster speed.

[Redback]

I usually just leave dc++ running, so my shares are always hashed

[Chopz]

BTW dont share your folder that you are receiving files on, it makes your computer run slow as you will always be hashing.

[Andrewzor]

Theres always bound to be someone who forgot to hash at the lan I always do it a week or two before the lan, I just leave it on overnight and its usually done.

Yay

[Redback]

BTW dont share your folder that you are receiving files on, it makes your computer run slow as you will always be hashing.

dc is supposed to remember the hash of the files it downloads

[priorax]

Please use .674 - it hashes, .;750 doesnt

Umm, from my experience with .75 it hashes, it's the builds before .674 that don't

[oohms]

673, which is 674 without an annoying CPU usage bug, is the best version to use, and can get files off people with both 306 era and 0.7xx clients.

If you don't have time to hash, or something breaks, you can use 306, and you only miss out on people with the new clients (But can still leech off people with 673/674)

(0.7xx users can't leech or share with 0.306 and vice versa, but 673/674 does both)

[Maelstrom]

I upgraded to 0.761

I figure most people have worked out by now to pre-hash

Plus it has some nice features, like being able to grab parts of the same linux iso from multiple people

[oohms]

I upgraded to 0.761

I figure most people have worked out by now to pre-hash

Plus it has some nice features, like being able to grab parts of the same linux iso from multiple people

There are still a lot of people on 306.. so i guess you'll have to outweigh the new features vs. what variety of linux ISOs you want

[Redback]

Should ban non hashing clients from the hub imo.

Hash or die.

[wolfmother]

673, which is 674 without an annoying CPU usage bug, is the best version to use, and can get files off people with both 306 era and 0.7xx clients.

If you don't have time to hash, or something breaks, you can use 306, and you only miss out on people with the new clients (But can still leech off people with 673/674)

(0.7xx users can't leech or share with 0.306 and vice versa, but 673/674 does both)

0.401 will also read both styles of file list but won't hash, but I believe it has that CPU usage bug.

[wolfmother]

Should ban non hashing clients from the hub imo.

Hash or die.

To be honest, if anything, we should ban hashing clients. None of the advantages that it gives on the internet are applicable to LANs. I haven't yet seen a single hardcore proponent of hashed-only clients that fully understands why you want to hash in the first place.

[Redback]

If you are trying to get something off a busy sharer, you can often get some of it from someone else. End result is the busy sharers are less busy, spreading the load somewhat.

[crenn]

If you are trying to get something off a busy sharer, you can often get some of it from someone else. End result is the busy sharers are less busy, spreading the load somewhat.

*selects users using 0.304-0.678, right clicks and selects match queue* Don't know what you mean

[wolfmother]

Wolfmother's big DC++ hashing writeup
Background - What is hashing?
Hashing is a way of uniquely identifying the contents of any file. After processing the entirety of the file's contents, DC++ will spit out a small piece of text that can be used as a "fingerprint" for that file. This allows a lot of neat things like being able to verify the integrity of a file or being able to find other computers on the network that also have that file. However, it is not without its downsides.

The primary problem is the fact that it takes a long time to process large amounts of files, often up to days for large shares. The other is that since hashing is very important on the internet, the developers of DC++ intentionally removed compatibility with non-hashing clients in newer versions. This becomes a problem, as often someone comes to a LAN without hashing or knowing that they should hash, and cannot share their files with people using the newer versions until they spend a few days hashing their files (by which time the LAN is often over).

About TTH
DC++ uses a method of hashing called TTH - Tiger Tree Hashing. Tiger is the name of the hashing algorithm used; it's optimized for high performance on desktop systems and while it doesn't have as large a margin of security as many other popular systems, it has not been broken and there is no reason to believe it'll be broken in the near future. Instead of hashing the whole file in one go, DC++ will hash small (64kb) pieces of the file and then hash the hashes together until it comes up with a final result. This takes about 10% longer than just hashing the file outright, but gives the advantage that it's possible to identify an incomplete part of a file.

However, keeping track of all of those hashes (there can be tens of thousands for a high definition video file) uses a substantial amount of memory, and every hash lookup request means looking through this. This file is saved as HashData.dat and will generally grow at a rate of about 50-60mb per terabyte shared. It must be loaded into memory when DC++ is launched and stay there, and a lot of important operations like searching, downloading or uploading will require it to be accessed frequently. This can hamper performance somewhat on systems with slower CPUs or small amounts of RAM.

How long will it take?
The amount of time it takes to hash files depends on the amount to be shared and the hardware your system is running. The read speed (from the hard disk) depends on the speed of your hard disk(s), hard disk controller(s), and motherboard. The hashing speed depends on the speed of your CPU and RAM. A rule of thumb I use for my server is about 6 hours per terabyte, but it can vary wildly depending on your setup.

DC++ has been designed to hash on a single core at a time, so it is not uncommon to see one core being used 100% while hashing in the Task Manager. This is normal and you can't really improve performance much more if this is what you're seeing.

Why it's good on the internet
Hashing has a lot of advantages on the internet:

• Fake files are much less of a problem.
• You can download from multiple users at once without the risk of one of their copies being corrupt (even if they have different filenames).
• Resuming incomplete downloads, particularly from different users

Why LANs are very different to the internet

• Fake/dodgey files are extremely uncommon at a LAN (compared to the total share size) and you can generally download a different copy in seconds anyway.
• Bandwidth is much higher, so there is nowhere near as much unique data on the network (meaning that there are probably only a handful of unique copies of a given file but a few dozen mirrors of each with filenames intact, meaning you can just Match Queue and it'll probably work better than a hash request)
• At any given time, a substantial number of people will be hitting the network's transfer limit*. Unless the person you're downloading off has a crapton of slots you'll be hitting that bottleneck long before they max out their link.
• Total sharesizes and data throughput are much, much higher, meaning that those tiny little CPU and memory hits that hashing causes suddenly become much, much bigger.
• Bandwidth is so high that verifying and resuming incomplete files is usually a waste of time; it's just about quicker to just download it again from scratch.
• And of course, there's the processing period at the start.

That said, those advantages are not completely moot at a lan; they just become no longer worth the downsides. DC's approach to hashing is quite hamfisted and not suited to the environment at all.

*This is part of how the network is laid out; each table only gets a few gigabits to the core.
Version compatibilities
There are two types of file list; hashed, and non-hashed (they're actually different formats). However, actually choosing a version of DC++ is not that simple: there are some versions that can read one but not the other, and there is no version which will generate both. If someone wants to download a hashed file list from you and your version only generates a non-hashed file list, they won't be able to see your files and thus can't download from you. However, it's possible to be running a version which will readtheir hashed file list even if it doesn't generate one, in which case you can download from them.

DC++ is open source, so there are a lot of variations of it and variations of those variations available. Without getting into the complexities of ApexDC, StrongDC, IceDC etc etc what really matters is the version of DC++ they're based on. Here's a list of the major versions of DC++ with the versions worth using in bold:

• Before .306 - generates only non-hashed, reads only non-hashed.
• .306 - generates only non-hashed, reads only non-hashed.
• .401 - generates only non-hashed, reads both hashed and non-hashed. Has a bug involving large file sizes.
• .673 - generates only hashed, reads both hashed and non-hashed.
• .674 - generates only hashed, reads both hashed and non-hashed. Has a CPU usage bug.
• After .674 - generates only hashed, reads only hashed.

Also, there's a version called LANDC++ which I believe is based on a fairly recent version of IceDC++. It generates a file list in the same format as a hashed file list, but without the hashes. Since no version of DC++ will read such a list, it's not compatible with any of the above versions. Avoid.


Also
No, hash requests do not put any extra load on the network. Also you should have a minimum of one upload slot per hard disk plus an extra; any less and you'll make baby jesus cry.

[R4N]

I think thats enough of an explaination to keep everyone informed

case closed