qBittorrent unable to handle very large torrents · Issue #8449 · qbittorrent/qBittorrent · GitHub
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

qBittorrent unable to handle very large torrents #8449

Closed
MrEldritch opened this issue Feb 19, 2018 · 5 comments
Closed

qBittorrent unable to handle very large torrents #8449

MrEldritch opened this issue Feb 19, 2018 · 5 comments
Milestone

Comments

@MrEldritch
Copy link

MrEldritch commented Feb 19, 2018

qBittorrent version and Operating System

qBittorrent v4.0.4 (64 bit); Windows 10 Version 10.0.16299 Build 16299

What is the problem

When attempting to download a very large torrent (Danbooru2017, a extremely-large-scale tagged-image dataset intended for machine learning, found at https://www.gwern.net/docs/anime/danbooru2017-torrent.tar.xz), I encounter an error: Failed to load the torrent: (path to torrent file). Error: bencoded item count limit exceeded.

What is the expected behavior

The .torrent file itself is not defective; I have successfully added and downloaded the torrent in another client (Transmission). I would expect to be able to add and download this .torrent in qBittorrent, as I would normally do with any other .torrent file.

Steps to reproduce

-Download and extract the .torrent files from the linked archive
-In qBittorrent, File>Add Torrent File and select one of the included .torrents
-Error displays.

Extra info(if any)

Danbooru2017 is really unusually large, as far as datasets go - each of the .torrents contains ~300k images weighing in at around ~200GB. It is entirely possible that this might not be a "bug" at all, but that it's simply some deliberate size-check exceeding a bound set by someone who assumed nobody would ever intend to download that many items. However, since the .torrent is clearly well-formed and compatible with other clients (such as Transmission), and since clearly torrents large enough to exceed that limit do exist in practice, it seems like qBittorrent really ought to be able to handle it.

Presumably any other sufficiently large torrent would trigger this error; this is just the first time I have ever needed to download a collection this large.

(screenshot of the Torrent Properties window from Transmission, to give specific numbers on size)
image

@Chocobo1
Copy link
Member

@arvidn
I skimmed through code yet didn't find anything, do you mind to take a look at it?
qbt checked the torrent via:

TorrentInfo info(NativePtr(new libt::torrent_info(Utils::Fs::toNativePath(path).toStdString(), ec)));

@Chocobo1 Chocobo1 added this to the 4.0.5 milestone Feb 19, 2018
@arvidn
Copy link
Contributor

arvidn commented Feb 19, 2018

This is due to default limits on number of tokens and tree-depth of bencoded structures when parsing it. It's to prevent an old exploit where a recursive-decent parser could cause a stack overflow by just sending lots of "l" (for list) or "d" (for dictionary).

These default limits can be specified when calling bdecode(). However, if you're using the torrent_info constructor that takes a file, these options cannot be controlled, but you're stuck with the default.

If that's the case, I would suggest separating the torrent_info construction into two steps, first to parse the bdecode and then construct the torrent_info with the resulting bdecode_node.

@arvidn
Copy link
Contributor

arvidn commented Feb 19, 2018

See depth_limit and token_limit here: https://libtorrent.org/reference-Bdecoding.html#bdecode

Chocobo1 added a commit to Chocobo1/qBittorrent that referenced this issue Feb 19, 2018
`torrent_info` constructor has default limits that can't be changed via
parameters, so we handle the loading process manually and explicitly
specifiy the limits to `bdecode()`
Chocobo1 added a commit to Chocobo1/qBittorrent that referenced this issue Feb 19, 2018
`torrent_info` constructor has default limits that can't be changed via
parameters, so we handle the loading process manually and explicitly
specifiy the limits to `bdecode()`.
The token_limit is changed to 10000000.
Chocobo1 added a commit to Chocobo1/qBittorrent that referenced this issue Feb 19, 2018
`torrent_info` constructor has default limits that can't be changed via
parameters, so we handle the loading process manually and explicitly
specifiy the limits to `bdecode()`.
The token_limit is changed to 10000000.
@Chocobo1
Copy link
Member

I've submitted PR #8455.

Chocobo1 added a commit to Chocobo1/qBittorrent that referenced this issue Feb 20, 2018
`torrent_info` constructor has default limits that can't be changed via
parameters, so we handle the loading process manually and explicitly
specifiy the limits to `bdecode()`.
The token_limit is changed to 10000000.
Chocobo1 added a commit to Chocobo1/qBittorrent that referenced this issue Feb 20, 2018
`torrent_info` constructor has default limits that can't be changed via
parameters, so we handle the loading process manually and explicitly
specifiy the limits to `bdecode()`.
The token_limit is also changed to 10000000.
Chocobo1 added a commit to Chocobo1/qBittorrent that referenced this issue Feb 22, 2018
`torrent_info` constructor has default limits that can't be changed via
parameters, so we handle the loading process manually and explicitly
specifiy the limits to `bdecode()`.
The token_limit is also changed to 10000000.
Chocobo1 added a commit to Chocobo1/qBittorrent that referenced this issue Feb 22, 2018
`torrent_info` constructor has default limits that can't be changed via
parameters, so we handle the loading process manually and explicitly
specifiy the limits to `bdecode()`.
The token_limit is also changed to 10000000.
Chocobo1 added a commit that referenced this issue Mar 1, 2018
Fix loading very large torrents. Closes #8449
@Chocobo1
Copy link
Member

Chocobo1 commented Mar 1, 2018

@MrEldritch
Thanks for the report! Will be fixed in next release (v4.0.5)

@qbittorrent qbittorrent locked and limited conversation to collaborators Feb 28, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants