TumblThree - A Tumblr Backup Application

TumblThree - A Tumblr Backup Application

TumblThree is the code rewrite of TumblTwo, a free and open source Tumblr blog backup application, using C# with WPF and the MVVM pattern. It uses the Win Application Framework (WAF). It downloads photo, video, audio and text posts from a given tumblr blog.

Screenshots:

TumblThree - A Tumblr Backup Application.</a></p>
<p><a id=

Features:

  • Source code at github (Written in C# using WPF and MVVM).
  • Multiple concurrent downloads of a single blog.
  • Multiple concurrent downloads of different blogs.
  • Internationalization support (currently available: en, zh, ru, de, fr).
  • A download queue.
  • Autosave of the queuelist.
  • Save, clear and restore the queuelist.
  • A clipboard monitor that detects blogname.tumblr.com urls in the clipboard (copy and paste) and automatically adds the blog to the bloglist.
  • A settings panel (change download location, turn preview off/on, define number of concurrent downloads, set the imagesize of downloaded pictures, set download defaults, enable portable mode, etc.).
  • Uses Windows proxy settings.
  • A bandwidth throttler.
  • An option to download an url list instead of the actual files.
  • Set a start time for a automatic download (e.g. during nights).
  • An option to skip the download of a file if it has already been downloaded before in any currently added blog.
  • Uses SSL connections.
  • Preview of photos & videos.
  • Taskbar buttons and key bindings.

Blog backup/download:

  • Download of photo, video (only tumblr.com hosted), text, audio, quote, conversation, link and question posts.
  • Download meta information for photo, video and audio posts.
  • Downloads inlined photos and videos (e.g. photos embedded in question&answer posts).
  • Download of _raw image files (original/higher resolution pictures).
  • Support for downloading Imgur, Gfycat, Webmshare, Mixtape, Lolisafe, Uguu, Catbox and SafeMoe linked files in tumblr posts.
  • Download of safe mode/NSFW blogs.
  • Allows to download only original content of the blog and skip reblogged posts.
  • Can download only tagged posts.
  • Can download only specific blog pages instead of the whole blog.
  • Allows to download blog posts in a defined time span.
  • Can download hidden blogs (login required / dash board blogs).
  • Can download password protected blogs (of non-hidden blogs).

Liked/by backup/download:

  • A downloader for downloading "liked by" photos and videos instead of a tumblr blog (e.g. https://www.tumblr.com/liked/by/wallpaperfx/) (login required).
  • Download of _raw image files (original/higher resolution pictures).
  • Allows to download posts in a defined time span.

Tumblr search backup/download:

  • A downloader for downloading photos and videos from the tumblr search (e.g. http://www.tumblr.com/search/my+keywords).
  • Download of _raw image files (original/higher resolution pictures).
  • Can download only specific blog pages instead of the whole blog.

Tumblr tag search backup/download:

  • A downloader for downloading photos and videos from the tumblr tag search (e.g. http://www.tumblr.com/tagged/my+keywords) (login required).
  • Download of _raw image files (original/higher resolution pictures).
  • Allows to download posts in a defined time span.

Program Usage:

  • Extract the .zip file and run the application by double clicking TumblThree.exe.
  • Copy the url of any tumblr.com blog you want to backup from into the textbox at the bottom left. Afterwards, click on 'Add Blog' on the right side of it.
  • Alternatively, if you copy (ctrl-c) a tumblr.com blog url from the address bar/text file, the clipboard monitor from TumblThree will detect it and automatically add the blog.
  • To start the download process, click on 'Crawl'. The application will regularly check for (new) blogs in the queue and start processing them, until you stop the application by pressing 'Stop'. So, you can either add blogs to the queue via 'Add to Queue' or double click/drag'n'drop first and then click 'Crawl', or you start the download process first and add blogs to the queue afterwards.
  • A light blue bar left to the blog in the queue indicates a actively downloading blog.
  • The blog manager on the left side also indicates the state of each blog. A red background shows an offline blog, a green background an actively crawling blog and a purple background an enqueued blog.
  • You change the download location, the number of concurrent connections, the default backup settings for each newly added blog and various other settings in the 'Settings'.
  • In the Details window you can view statistics of your blog and set blog specific options. You can here what kind of post type (photo, video, audio, text, conversation, quote, link) to download.
  • For downloading only tagged posts, you'll have to do some steps:
    1. Add the blog url.
    2. Open the blog in the details tab, enter the tags in the Tags textbox in a comma separated list without the leading hash (#) sign. E.g. great big car,bears would search for images that are tagged for either a great big car or bears or both.
  • For downloading password protected blogs, you'll have to do some steps:
    1. Add the blog url.
    2. Open the blog in the details tab, enter the password in the Password textbox.
  • For downloading hidden blogs (login required blogs), you have to do some steps:
    1. Go to Settings, click on the Connection tab and fill in your tumblr email address (login) and password, then click the Authenticate button. If the login was successfully, the label will change and display your email address. The email address and password are not stored locally on disk but cookies are generated and saved in %LOCALAPPDATA%\TumblThree in json format.
    2. Add the blog url.
  • For downloading liked photos and videos, you'll have to do some steps:
    1. Go to Settings, click on the Connection tab and fill in your tumblr email address (login) and password, then click the Authenticate button. If the login was successfully, the label will change and display your email address. The email address and password are not stored locally on disk but cookies are generated and saved in %LOCALAPPDATA%\TumblThree in json format.
    2. Add the blog url including the liked/by string in the url (e.g. https://www.tumblr.com/liked/by/wallpaperfx/).
    3. For downloading your own likes, make sure you've (temporarily) enabled the following options in your blogs settings (i.e. https://www.tumblr.com/settings/blog/yourblogname):
      1. Likes -> Share posts you like (to enable the publicly visible liked/by page)
      2. Visibility -> blog is explicit (to see/download NSFW likes)
  • For downloading photos and videos from the tumblr search, you'll have to do some steps:
    1. Add the search url including your key words separated by plus signs (+) in the url (e.g. https://www.tumblr.com/search/my+special+tags).
  • For downloading photos and videos from the tumblr tag search, you'll have to do some steps:
    1. Go to Settings, click on the Connection tab and fill in your tumblr email address (login) and password, then click the Authenticate button. If the login was successfully, the label will change and display your email address. The email address and password are not stored locally on disk but cookies are generated and saved in %LOCALAPPDATA%\TumblThree in json format.
    2. Add the search url including your tags separated by plus signs (+) in the url (e.g. https://www.tumblr.com/tagged/my+special+tags).

Key Mappings:

  • Currently mapped keys:
    • double click on a blog adds it to the queue
    • drag and drop of blogs from the manager (left side) to the queue
    • space -- start crawl
    • ctrl-space -- pause crawl
    • shift-space -- stop crawl
    • del -- remove blog from queuelist
    • shift-del -- remove blog from blogmanager
    • ctrl-shift-g -- manually trigger the garbage collection

Getting Started:

The default settings should cover most users. You should only have to change the download location and the kind of posts you want to download. For this, in the Settings (click on the Settings button in the lower panel of the main user interface) you might want to change:

  • General -> Download location: Specifies where to download the files. The default is in a folder Blogs relative to the TumblThree.exe
  • Blog -> Settings applied to each blog upon addition:
    • Here you can set what posts newly added blogs will download per default. To change what each blog downloads, click on a blog in the main interface, select the Details Tab on the right and change the settings. This separation allows to download different kind of post for different blogs. You can change the download settings for multiple existing blogs by selecting them with shift+left click for a range or ctrl-a for all of them.
    • Note: You might want to always select:
      • Download Reblogged posts: Downloads reblogs, not just original content of the blog author.

Settings you might want to change if the download speed is not satisfactory:

  • Connection -> Concurrent connections: Specifies the number of connections used for downloading posts. The number is shared between all actively downloading blogs.
  • Connection -> Concurrent video connections: Specifies the number of connections used for downloading tumblr video posts. The vt.tumblr.com host regularly closes connections if the number is too high. Thus, the maximum number of vt.tumblr.com connections can be specified here independently.
  • Connection -> Concurrent blogs: Number of blogs to download in parallel.

Most likely you don't have to change any of the other connection settings. In particular, settings you should never change, unless you're sure you know what you are doing:

  • Connection -> Limit Tumblr Api Connections: Leave this checkbox checked and do not change the corresponding values of 90 connections per 60 seconds. If you still change them, you might end up with offline blogs or missing downloads.

Further Insights:

  • Note: All the following files are stored in json format and can be opened in any editor.
  • Application settings are stored in C:\Users\Username\AppData\Local\TumblThree\.
  • You can use the portable mode (settings->general) to stores the application settings in the same folder as the executable.
  • For each blog there is also a database (serialized class) file in the Index folder of the download location named after the blogname.tumblr. Here blog relative information is stored like what files have been downloaded, the url of the blog and when it was added. This allows you to move your downloaded files (photos, videos, audio files) to a different location without interfering with the download process.
  • Some settings aren't hooked up to the graphical user interface. It's possible to view all TumblThree settings by opening the settings.json in any editor located in C:\Users\Username\AppData\Local\TumblThree\. Their names should be self explainatory. Some notable settings to further fine tune the application include:
    • BufferSize: Allows to set the buffer size for downloading binary files (photos, videos) in multiples of 4KB. The default is 2MB, thus the BufferSize has a value of 512. Increasing this value reduces disk fragmentation as more of the file is kept in the memory before it gets written out to the disk but increases the memory usage.
    • MaxNumberOfRetries: Sets the maximum number of retries if a tumblr server forcefully closes the connection. This might regularly happen on the tumblr video host (vt.tumblr.com) if too many connections were opened in parallel. After the limit is exhausted, the file is left truncated, but is also not registered as a successful downloaded. Thus, the file can be resumed in the next crawl.
    • TumblrHosts: Contains a list of hosts which is tried for downloading _raw photos if the photo size is set to raw. If none of the hosts contains the _raw version, the actually scanned host is tried with the next lower resolution (1028).

Changelog:

2018-07-05:

  • Implements the Tumblr login process and cookie handling in code instead of relying on the Internet Explorer for the Tumblr login process.

2018-06-09:

  • Fixes hidden Tumblr blog download problems caused by the new Tumblr ToS.

2018-05-20:

  • Programmatically agrees to new ToS and GDPR.
  • Implements SVC authentication changes. The SVC service is used to display the dash board blogs (i.e. hidden tumblr blogs). Changes in this internal Tumblr api prohibited TumblThrees access.
  • Saves the last post id in successful hidden tumblr downloads.
  • Improves the text parser of the tumblr api and tumblr svc data models. Separated the slug from the url as the data models are inconsistent. Separated the photoset urls from the photo urls. Moved the date information into a separate column.
  • Minor text changes of some user interface elements.

2018-04-18:

  • Updates the tumblr blog crawler and the hidden tumblr datamodel to reflect tumblr api changes that break blog download of previous TumblThree versions.

2018-02-28:

  • Allows to download only specific pages of hidden Tumblr blogs and in the tumblr search.
  • Improves the proxy settings. TumblThree now uses the default Windows (Internet Explorer) settings if not overridden within TumblThree.
  • Changes the behavior of the timeout value (Settings->Connection->Timeout). The timeout value now counts file chunks of 4kb instead of the whole file download, thus it should better detect if a download is stalled or a connection dropped without canceling active downloads of larger files (e.g. videos).
  • Changes default timeout value (for new users) from 600s to 30s.
  • Fixes possible download of the same photo but with different resolutions. This happened if the _raw file download was interrupted (the timeout hit), then the same photo was queued for download with the _1280 resolution. If the blog was then subsequently queued again, the _raw file was downloaded next to the _1280 file.
  • Fixes reblog/original post detection in the tumblr hidden crawler.
  • Fixes check blog status during startup-option.
  • Fixes download of password protected tumblr blogs.
  • Adds Mixtape, Lolisafe, Uguu, Catbox and SafeMoe parser (thanks to bun-dev).

2017-12-31:

  • Fixes a bug that released the video connection semaphore too often. That means the slider in the settings for limiting the video downloads didn't work at all. It should properly limit the connections to the vt.tumblr.com host and prevent incomplete video downloads now.
  • Includes a rewrite of the blog detection during blog addition. It should reduce latency if you mass add blogs by copying urls into the clipboard (ctrl-c). Offline blogs aren't added anymore.
  • Notifies the user when a connection timeout has occurred. The message states whether the timeout has occurred during downloading or crawling. If it happened during crawling, you might want to re-queue the blog at some point to grab missing posts. A connection timeout should only happen if your connection is wonky. You can decrease/increase the timeout in the settings (settings->connection).
  • You can now specify in the Details-panel for each blog where its files should be downloaded. If the text box control is empty, the files are downloaded as in previous releases in the folder specified in the global download location (settings->general), plus the blogs name.
  • Imgur.com linked albums in tumblr posts are now entirely downloaded if enabled (details panel->external->download imgur). Previously, only directly linked images were detected.
  • Adds an option to load all blog databases into memory and compare each to-download binary file to all databases across TumblThree before downloading. If the file has already been downloaded in any blog before, the file is skipped and will not be counted as downloaded. You can enable this in the settings (settings->global).
  • Allows to add hidden tumblr blogs using the dashboard url (i.e. https://www.tumblr.com/dashboard/blog/blogtobackup).
  • Allows to add all blog types without the protocol suffix (i.e. wallpaperfx.tumblr.com, www.tumblr.com/search/cars).
  • Adds an option to enable a confirmation dialog before removing blogs (#186, #130, #98). It's off by default.

2017-11-17:

  • Adds support for downloading Imgur.com, Gfycat.com and Webmshare.com linked files in tumblr posts.
  • Improves downloading of tumblr liked/by photos and videos.

2017-10-20:

  • Restores bandwidth limiter functionality.

2017-10-13:

  • Changes the default _raw photo host.

2017-10-09:

  • Fixes crawler stop in hidden tumblr blog downloads.
  • Adds options to set the default blog settings for the download from time, download to time and tags in the settings menu.
  • Adds some (ar, el, es, fa, fi, he, hi, it, ja, ko, no, pa, pl, pt, th, tr and vi) google translate translations.

2017-09-08:

  • Can download password protected blogs of non-hidden blogs.
  • Minor UI updates.

2017-08-22:

2017-08-21:

  • French, Spanish and simplified Chinese translations.
  • Removes user interface lag during blog addition.
  • Allows to set the buffer size for downloading binary files in the settings.json in multiples of 4KB. The variable is called BufferSize. The new default is 2MB, thus the BufferSize has a value of 512. Previously it was set to 4KB, but apparently Windows does not do any useful caching on NTFS if multiple writes are concurrent and async. Thus, this should reduce disk fragmentation.
  • Uses .NET Framework 4.6 now as it should be available for all supported windows versions (Windows Vista and above).
  • Improved the selection handling in the details panel. If multiple blogs are selected, old values are now kept if they are the same for all blogs and changes are immediately reflected.
  • Audio file download support for tumblr and hidden tumblr blogs.
  • More code Refactoring.

2017-07-03:

  • Can download hidden (login required/dash board) blogs.

2017-06-30:

  • Improved performance and bugfixes.

2017-06-20:

  • Downloads high resolution (_raw) images.
  • Updated translations (German and Russian).
  • Applies changed settings immediately.

2017-06-04:

  • Sets the date modified date in the Explorer to the posts time.
  • Allows to download single or ranges of blog pages.
  • Full screen media preview.

2017-05-20:

  • Option to skip reblogged posts.
  • Improves detection of inlined photos and videos in text posts (e.g. in answer posts).

2017-05-14:

  • Portable mode.
  • Downloads liked photos and videos.

2017-04-18:

  • Code refactoring.
  • Uses async/await in most of the code instead of tasks from the threadpool.
  • Uses a consumer producer pattern for grabbing and downloading as the Tumblr api v1 is now rate limited.
  • Downloads are now resumable.
  • Data files are now saved as json instead of binary.
  • Reduced memory usage by layering off the downloaded file list and only load it if needed.
  • Improves ui responsiveness.

2017-01-08:

  • Improves the speed of the network code.
  • Adds an option to use a http proxy.
  • Downloads inline images of tumblr posts.
  • Added Russian translation.

2016-12-13:

  • Improves the ui scaling of the main window for smaller resolutions.
  • Prevents crawling of offline blogs.
  • If the same blog is multiple times in the queue and already once active, any other free crawler task will skip and remove any already active blog and proceed to the next inactive blog in the queue.
  • Improved german translation.

2016-12-10:

  • The check for already downloaded files is now independent from the actual host and based entirely on the filename. It look likes the host/mirror does actual vary which would result in a reload of the file since its url changed.
  • Add scrollbars to the settings window if the controls do not fit.
  • Safely replaces blog indexes. If there is an error (e.g. no disk space left) during the update of the index file, the old state should not be corrupted anymore.
  • Changes some color and adds an alternate color for the blog manager.

2016-11-23:

  • Fixes application crashes which occurred by adding tumblr blogs without title or description.
  • Decreases determination time of already downloaded files for large blogs (>100,000 posts) by at least three orders of magnitude.

2016-11-22:

  • Creates more meta information (post id, reblog key, timestamp, tags, slug, title) of the posts, including image, video and audio types.
  • Fixes the progress calculation by adding the found duplicates to the progress. Also states them in the details window.
  • Fixes a locking issue for the meta files (*.txt) which resulted in incomplete downloads.
  • Updates the details and settings view for a better understanding on how to use the application.

2016-11-20:

  • Fixes proper counting of downloaded files.
  • Fully implements the details window (context menus, etc.).

2016-11-18:

  • Fixes the initial automatic queue restore function.
  • Fixes the autodownload function.

2016-11-16:

  • Picture- and videopreview in the details window.
  • Allows the download of text, audio, quote, conversation, link type posts.
  • Download of text, audio, quote, conversation, link and .gif images are based on each blog instead of a global setting and can be turned on/off in the details view. The settings in the settings window are used as template for newly added blogs.
  • Modified .tumblr index files get now always saved upon application exit regardless of the crawlers state. Previously if the application was closed during an active crawl, the index wasn't updated.
  • Inlined the WAF code under lib for easier project setup for newcomers that want to contribute code.
  • bugfixes, UI and memory enhancements.

2016-10-15:

  • Bandwidth throttling.
  • Connection timeout settings.
  • auto queue and start download function.
  • save states of the UI (column size and order).
  • download of hidden blogs.
  • fix proper saving of the ratings and tags.

2016-06-11:

  • Added German translation.

2016-06-10:

  • Support for tumblr.com hosted videos. Check the settings window to enable video download (default: off).

2016-06-08:

  • Tag crawling now properly working. Also it's case-insensitive now.
  • Fixed crash upon blogs with zero-image count in the queue list (e.g. blog is offline, or tag search didn't evaluate any images).
  • Fixed randomly occurring crash in the clipboard monitor.
  • Changed icons (requested by the TumblOne creator).

2016-04-12:

  • Now with progress output in the Queue tab (during url crawling for imageurls -- the number of posts evaluated; during downloading -- the current image url).
  • Added missing resume button in the taskbar control.

2016-04-11:

  • Support for urls starting with https:
  • Fixes application crashed upon pressing the stop-button due to improper exception handling
  • Now saves the index file at every time. Previously the application would exit if the crawling processes was still active without properly waiting them to finish and save its state. Now there is a grace period for the tasks to finish. Same was true if the crawl was paused and then exited.

Download:

Comments

anonymous (not verified)
Sat, 08/07/2017 - 07:25

I think the raw url only works on images uploaded from 2014 and onwards. The first mention of raw urls on the tumblr api is around that time.

zab
Sat, 08/07/2017 - 17:53

Well, I've changed it now. I've checked the amount of successful hits and the media.tumblr.com is more frequent than the media68.tumblr.com host.
You can specify a list of possible hosts now in the settings.json. The default list contains, in this order:

media.tumblr.com, media68.tumblr, media66.tumblr.com

If all of these fail, then the sizes are incrementally dropped on the original detected host (most likely the media68.tumblr.com) of the crawler.

Pep (not verified)
Fri, 07/07/2017 - 08:15

I'm wondering how do I download a Tumblr page by month? I want to download only June 2017 of a certain blog. How can I do that? Thanks

zab
Mon, 10/07/2017 - 19:13

C:\Users\Username\AppData\Local\TumblThree\Settings\

If you click the "portable mode" in the Settings under General, the settings.json will be stored next to the TumblThree.exe.

anonymous (not verified)
Tue, 11/07/2017 - 06:08

Hi!, Where do I list the possible host in the "Settings.json" thankss!!!

zdevilinside (not verified)
Sun, 30/07/2017 - 15:40

Is there any way to bypass the "safe mode" being implemented by Tumblr with your software?

zab
Sun, 30/07/2017 - 18:53

You've tried it already and it doesn't work, or are you just asking?

Any blog that does in fact not work? Name?

zdevilinside (not verified)
Thu, 03/08/2017 - 15:53

I have tried it and it doesn't work. You now need to Login to Tumblr to view some blogs such as LGBT content.

zab
Thu, 03/08/2017 - 17:37

Did you actually try the latest version before posting this? Download version v1.0.8.5, remove the blogs, re-add them.

If that does not work, name at least one blog, otherwise I cannot fix it.

For private blogs you need to login. There is no other way you can do this. Show me one software that can handle private blogs without a login. Most of them can't even handle them with login.

Edit: See my answers below. Basically every version except v1.0.8.4 works with safe-mode blogs.

zab
Tue, 01/08/2017 - 13:44

How can I if you don't tell me the blogs in question? I'm not using TumblThree regularly and your post doesn't help me fix anything.

I've said the same thing to the poster right above your comment, and also yesterday on github: If you want something to be fixed, be detailed and provide instruction so that the issue can be reproduced.

So, whats the url or name of the blogs that are identified as offline even if they apparently aren't?

anonymous (not verified)
Wed, 02/08/2017 - 08:56

Ah, well I suppose that's a good point. Well, just to preface, every last one of the problem blogs I have are NSFW.

https://therealfunk.tumblr.com/
http://shugarsketch.tumblr.com/
https://magi34.tumblr.com/
https://bigdeadrevived.tumblr.com/
https://cutesexyrobutts.tumblr.com/
https://cyanknight20.tumblr.com/
http://andava.tumblr.com/
https://ehryel.tumblr.com/

I hope though it's just a common issue and not a situation where I have to keep giving you TUmblr urls. I assure you, each and every one of these are being labeled offline by TumbThree no matter what I do, even removing them and adding them back.

zab
Thu, 03/08/2017 - 17:35

Thanks for the list. Can you remove and readd them with the v1.0.8.5 version? It should be working there already.

I just tested them, they work here.

Thanks for reporting!

anonymous (not verified)
Fri, 04/08/2017 - 11:42

I'm happy to report, though rather late, that the issue is indeed resolved! Thank you for your work

zab
Fri, 04/08/2017 - 07:43

They actually even work with version 1.0.0.0.

So, it was a bug I've introduced with the v1.0.8.4 release trying to automatically distinguish private blogs from non-private blogs and offline blogs.

anonymous (not verified)
Fri, 25/08/2017 - 18:26

i am also having this issue. 8/25/17
just downloaded the newest version.
is it cause i am trying to download more then on at a time??
i put the parallel blog and connection to "27"

http://imgur.com/a/d3oyj

zab
Fri, 25/08/2017 - 19:19

Thats probably because too many connections to the Tumblr API were opened.

In the settings, do you still have the checkbox next to "Limit Tumblr API Connections" set? Are the values next to it still at 90 connections, 60 seconds?

If not, reset them and in the general tab set "Check blog online status on startup". Close TumblThree, wait a bit, then reopen it.
If they are still set, you might have to lower the connection value to 85 or 80.

There is a server defined limit of connections you can open to the Tumblr API. If you open two instances of TumblThree or untick the checkbox the online check might fail since it just opens connections regardless of any limits on the Tumblr servers. I've just answered a similar question in a bit more detail here.

anonymous (not verified)
Sat, 26/08/2017 - 01:16

I don't have "Limit Tumblr API Connections" box checked and yes the value are still at default. 90/60. Still new to this, so i don't know how to mess with the settings. Also, i tired to use the "Authentication" in the setting, a screen pops up and then disappears. Why is this so?

zab
Sat, 26/08/2017 - 04:12

Then check the checkbox, otherwise you'll have this error.
If the authenticate window closes, you're done.

Edit: I've changed the code now that is displays that you are over the rate limit instead of marking the blog offline.

anonymous (not verified)
Tue, 01/08/2017 - 16:46

Hi!

I checked one of the blogs crawled with latest TumbleThree and found quite a lot of duplicates with "_1280" and "_raw" in filenames, with one of these files usually corrupted (not completely downloaded, according to its filesize). Sometimes the images are of the same height and width in pixels, sometimes their sizes differs . Does TumblThree have option to check filesize and re-download corrupted images&

zab
Thu, 03/08/2017 - 16:35

I think I've already fixed that in the v1.0.8.2 and upwards releases. That happened with my initial _raw download implementation. The newest one shouldn't create files without content.

Did you try this with the most recent version and the error still exists?

Thanks for posting!

Abdulla Yousif (not verified)
Fri, 04/08/2017 - 13:35

Hi

Is it possible to download (Posted privately) posts?

Thanks

Larry (not verified)
Sun, 13/08/2017 - 17:59

Just found TumblThree and working through understanding how it works. One issue I've encountered is video blogs with hundreds or thousands of posts, yet TumblThree only downloads a handful of videos, or even none. In the settings I've indicated to download all images and videos, nothing else... what am I doing wrong? I've even tried checking "Download reblogged posts", but that only added a few more downloads...

Thanks for any help.

zab
Sun, 13/08/2017 - 21:05

You had the download reblogged posts checked for the blog or just in the settings? Check the details window once you've selected the blog. Otherwise it will only download original content, which might be just a few videos depending on your blog.

Otherwise, also always, state the blog name/url otherwise I cannot help you or fix anything. And the TumblThree version too.

zab
Mon, 21/08/2017 - 00:14

As you can see, it requires a login now to access https://www.tumblr.com/liked/by/wallpaperfx/, thus it doesn't work with the current code. That wasn't the case before. It was freely accessible.

Either they changed it in general for all blogs (like with the safe mode), or you can set your likes private. I don't know. Either case, it probably takes some time.

Edit: Fixed. I broke the code in the refactoring. Thanks for letting me now!

zab
Tue, 22/08/2017 - 07:02

Ah, I see now what you mean. I have forgotten to update the validation rules for the textbox in the corner since I basically never use it. I'll update it.

The best is if you just select the url and copy it into the clipboard with ctrl-c. TumblThree will then add the blog automatically.

Edit: If you redownload the latest binary it should work already.

UnknownUser (not verified)
Tue, 22/08/2017 - 08:43

Your Tumblr tool is just a brilliant, one of a kind. I didn't even expect such a great downloading speed. Thank you for your tremendous support! I wish you all the best.

James (not verified)
Sun, 20/08/2017 - 22:18

I just wanted to comment and say thank you for all of your forks and efforts! I really appreciate TumbleTwo/Three and how much they've made my life easier. As an archival nut, this really does put me at ease. So thank you again :)

anonymous (not verified)
Fri, 25/08/2017 - 18:05

this error came up, "limit exceeded.lower connection to the tumblr api." what is this and how do i fix this issue/ error??

zab
Fri, 25/08/2017 - 19:07

It's not dramatic. It just tells you that some pages of the blog couldn't be crawled because too many connections to the Tumblr API were opened. Thus, if you want to search all pages you'll have to recrawl the blog.

For downloading a "normal" blog TumblThree uses the Tumblr api as it provides a stable interface that is guaranteed to not change over the time. The data is also provided in detail and in a nice form. The downside of this interface is that it only allows a specifc amount of a connections/accesses per time.

In the settings you can lower them, but for me the default of 90 connections per 60 seconds actually always worked. Since the exact amount is not specified by Tumblr it's a guessed value.

So, either recrawl the blog, see if it happens again. if not, fine. If it happens again you might want to try 85 connections per 60 seconds or similar. It probably won't change anything (in terms of speed) since the crawling/searching isn't the speed-limiting step but downloading the photos/videos is. So there is probably no downside in doing this. Of course, it depends on your connection.

Rakesh (not verified)
Wed, 30/08/2017 - 14:45

Can i able to download post by date?

Rakesh (not verified)
Fri, 01/09/2017 - 11:10

it will be better if that same code could be used to crowl own liked posts.

zab
Mon, 04/09/2017 - 14:59

What did you not understand?

I've taken my free time to implement this for you. But it was of course not the right thing, because you didn't say exactly what you wanted in your one (!) sentence. That's not my fault.

Now you come here and demand another feature, and keep asking the same thing within hours, even though I've already said that it's not implemented (yet) for liked/by downloads.

Let my be clear:

  • Open source doesn't mean I've implement things for random people on request, right now, in my spare time with zero compensation.
  • Open source doesn't mean I've to give free support for my software, I've decided to give for free to other people for zero compensation.
  • Open source doesn't mean I owe you or anybody anything.

Right now, after all the minus points you've collected here, I certainly will not add this. Do it yourself.

anonymous (not verified)
Mon, 04/09/2017 - 00:15

why is it that when i update to the new version my settings doesnt get carried over?

for example:
in the previous version i only checked; download images/videos/audio/force re-scan and went back and decided that i didnt want to download audio anymore. So i went into the settings and unchecked that box(audio). Then the new version comes out and my old setting are not there anymore. Now i am going back to each user and un-checking the audio box one by one. Is there a way to fix this??

its kind of a pain to reset the setting for each individual user.

anonymous (not verified)
Mon, 04/09/2017 - 11:06

Hi Zab,

yes we can download any blog within a given time span, but we can't download others or own liked post with time span.
date checkbox disappeared as soon as i put any liked post links.

need a help.

zab
Mon, 04/09/2017 - 13:07

Obviously, it "disappeared" because it isn't implemented (yet) in that particular download function. No help required. It's open source, feel free to add it.

PS: I've answered the same question two posts above yours. Next time, please read before asking.

PPS: I really hope you're not the same Rakesh guy from the two posts above.

anonymous (not verified)
Tue, 12/09/2017 - 18:46

Hi Jzab,

Firstly my apologies if this issue has been raised elsewhere, I couldn't found it.
I have noticed all of the Tumblr downloaders you made came with a useful section to limit what is downloaded based on whether the item has a specific tag. However, if anything in the tag is not a letter or number the download never works. For example, adding tags like "my-music", "(part_1)" or "#life" will result in TumblThree running the evaluation on the Tumblr page, but once fully evaluated nothing downloads.
I am sure this is a simple issue with the software recognizing these symbols, is there an existing fix? If there isn't, would there be a possibility of this being included in a future update?

Many thanks

zab
Tue, 12/09/2017 - 21:26

Hi!

Yes, it's easy to fix. I just split the entered string at each "," into separate strings and use them as tags.

But I've just tested it with non alphanumeric tags and for me at least it seems to work. You must not include the leading hash (#) sign for the tags.

I've added your example tags to my testing blog (https://privtumbl.tumblr.com/) and when I enter "my-music" or "(part-1)" it downloads the right posts.

anonymous (not verified)
Tue, 12/09/2017 - 21:54

Hi Zab

I am not sure but I think my question wasn't explained well, but I don't see a way to fix my issue in your answer. Also firstly just to warn you, I am not heavily PC literature, so "split the entered string at each , into separate strings" is not clear to me.
However, I think you interpreted my question as "if I want two or more tags tags, how do I add both of them to the tag section". This is not what my concern was.

My issue relates to using a single tag that has a non-alphabetic or numeric symbol in it. My issue is if I add your blog to TumblThree and download with no tags, all the images, text, etc is successfully downloaded. Also, if I download from your blog using the tag "fox", I will get the one image of a fox and nothing else, which is also a success. However if I add just the tag "my-music", nothing is downloaded. The only reason I can see is the use of the "-" symbol causes an issue.

Is there a solution to the problem I have listed above, or was your original answer the solution to this problem? If your original answer was the solution, could you try add a screenshot or something to that effect of you inputting the solution.

Sorry for the confusion

zab
Tue, 12/09/2017 - 22:00

I did understand this. Look in my last paragraph. The other was just an explanation of how things works, in case of you were interested.

I basically just told you it works for me and I couldn't reproduce it in my test:

You can test my test blog (https://privtumbl.tumblr.com/ -- just 5 pictures), I've added your example tags there, and added them to TumblThree, it worked. I added:

  • my-music
  • (part_1)

and the proper files were downloaded.

Otherwise give me a blog where I can reproduce your stated behavior including the tags.

anonymous (not verified)
Tue, 12/09/2017 - 23:58

Thanks Zab

Apologies again for the confusion. I did also think this is what you might have meant, unfortunately my problem appears to be now be more complicated.
When I include your Tumblr blog and add the "my-music" tag for the first time I get it to download everything (aka it works). But then if i run the scan again, it suddenly doesn't work. The "Download files" column is now empty with no number at all where there was originally a 1. Also even though there is nothing in the "Queue" section the pause and stop buttons are still clickable, suggesting the program is in some sort of broken loop.

zab
Wed, 13/09/2017 - 07:56

Works for me too.

There still is a 1 in the "Downloaded Files" and a 1 in the "Number of Downloads" cell if I re-queue my testing blog (https://privtumbl.tumblr.com/) or even delete it and re-queue it twice without deleting the downloaded picture.

btw, the "Number of Downloads"-cell varies depending on your search parameter. Thus, If TumblThree doesn't find any post matching your search criteria (tags), there will be an empty "Number of downloads-"cell.

So, I don't know. I still cannot reproduce it. I've put quite some time in the scanning/downloading since it's the core functionality. I somewhat doubt it would fall apart about something simple like this ..

Pages