TumblThree - A Tumblr Backup Application

TumblThree - A Tumblr Backup Application

TumblThree is the code rewrite of TumblTwo, a free and open source Tumblr blog backup application, using C# with WPF and the MVVM pattern. It uses the Win Application Framework (WAF). It downloads photo, video, audio and text posts from a given tumblr blog.

Screenshots:

TumblThree - A Tumblr Backup Application.</a></p>
<p><a id=

Features:

  • Source code at github (Written in C# using WPF and MVVM).
  • Multiple concurrent downloads of a single blog.
  • Multiple concurrent downloads of different blogs.
  • Internationalization support (currently available: en, zh, ru, de, fr).
  • A download queue.
  • Autosave of the queuelist.
  • Save, clear and restore the queuelist.
  • A clipboard monitor that detects blogname.tumblr.com urls in the clipboard (copy and paste) and automatically adds the blog to the bloglist.
  • A settings panel (change download location, turn preview off/on, define number of concurrent downloads, set the imagesize of downloaded pictures, set download defaults, enable portable mode, etc.).
  • Uses Windows proxy settings.
  • A bandwidth throttler.
  • An option to download an url list instead of the actual files.
  • Set a start time for a automatic download (e.g. during nights).
  • An option to skip the download of a file if it has already been downloaded before in any currently added blog.
  • Uses SSL connections.
  • Preview of photos & videos.
  • Taskbar buttons and key bindings.

Blog backup/download:

  • Download of photo, video (only tumblr.com hosted), text, audio, quote, conversation, link and question posts.
  • Download meta information for photo, video and audio posts.
  • Downloads inlined photos and videos (e.g. photos embedded in question&answer posts).
  • Download of _raw image files (original/higher resolution pictures).
  • Support for downloading Imgur, Gfycat, Webmshare, Mixtape, Lolisafe, Uguu, Catbox and SafeMoe linked files in tumblr posts.
  • Download of safe mode/NSFW blogs.
  • Allows to download only original content of the blog and skip reblogged posts.
  • Can download only tagged posts.
  • Can download only specific blog pages instead of the whole blog.
  • Allows to download blog posts in a defined time span.
  • Can download hidden blogs (login required / dash board blogs).
  • Can download password protected blogs (of non-hidden blogs).

Liked/by backup/download:

  • A downloader for downloading "liked by" photos and videos instead of a tumblr blog (e.g. https://www.tumblr.com/liked/by/wallpaperfx/) (login required).
  • Download of _raw image files (original/higher resolution pictures).
  • Allows to download posts in a defined time span.

Tumblr search backup/download:

  • A downloader for downloading photos and videos from the tumblr search (e.g. http://www.tumblr.com/search/my+keywords).
  • Download of _raw image files (original/higher resolution pictures).
  • Can download only specific blog pages instead of the whole blog.

Tumblr tag search backup/download:

  • A downloader for downloading photos and videos from the tumblr tag search (e.g. http://www.tumblr.com/tagged/my+keywords) (login required).
  • Download of _raw image files (original/higher resolution pictures).
  • Allows to download posts in a defined time span.

Program Usage:

  • Extract the .zip file and run the application by double clicking TumblThree.exe.
  • Copy the url of any tumblr.com blog you want to backup from into the textbox at the bottom left. Afterwards, click on 'Add Blog' on the right side of it.
  • Alternatively, if you copy (ctrl-c) a tumblr.com blog url from the address bar/text file, the clipboard monitor from TumblThree will detect it and automatically add the blog.
  • To start the download process, click on 'Crawl'. The application will regularly check for (new) blogs in the queue and start processing them, until you stop the application by pressing 'Stop'. So, you can either add blogs to the queue via 'Add to Queue' or double click/drag'n'drop first and then click 'Crawl', or you start the download process first and add blogs to the queue afterwards.
  • A light blue bar left to the blog in the queue indicates a actively downloading blog.
  • The blog manager on the left side also indicates the state of each blog. A red background shows an offline blog, a green background an actively crawling blog and a purple background an enqueued blog.
  • You change the download location, the number of concurrent connections, the default backup settings for each newly added blog and various other settings in the 'Settings'.
  • In the Details window you can view statistics of your blog and set blog specific options. You can here what kind of post type (photo, video, audio, text, conversation, quote, link) to download.
  • For downloading only tagged posts, you'll have to do some steps:
    1. Add the blog url.
    2. Open the blog in the details tab, enter the tags in the Tags textbox in a comma separated list without the leading hash (#) sign. E.g. great big car,bears would search for images that are tagged for either a great big car or bears or both.
  • For downloading password protected blogs, you'll have to do some steps:
    1. Add the blog url.
    2. Open the blog in the details tab, enter the password in the Password textbox.
  • For downloading hidden blogs (login required blogs), you have to do some steps:
    1. Go to Settings, click on the Connection tab and fill in your tumblr email address (login) and password, then click the Authenticate button. If the login was successfully, the label will change and display your email address. The email address and password are not stored locally on disk but cookies are generated and saved in %LOCALAPPDATA%\TumblThree in json format.
    2. Add the blog url.
  • For downloading liked photos and videos, you'll have to do some steps:
    1. Go to Settings, click on the Connection tab and fill in your tumblr email address (login) and password, then click the Authenticate button. If the login was successfully, the label will change and display your email address. The email address and password are not stored locally on disk but cookies are generated and saved in %LOCALAPPDATA%\TumblThree in json format.
    2. Add the blog url including the liked/by string in the url (e.g. https://www.tumblr.com/liked/by/wallpaperfx/).
    3. For downloading your own likes, make sure you've (temporarily) enabled the following options in your blogs settings (i.e. https://www.tumblr.com/settings/blog/yourblogname):
      1. Likes -> Share posts you like (to enable the publicly visible liked/by page)
      2. Visibility -> blog is explicit (to see/download NSFW likes)
  • For downloading photos and videos from the tumblr search, you'll have to do some steps:
    1. Add the search url including your key words separated by plus signs (+) in the url (e.g. https://www.tumblr.com/search/my+special+tags).
  • For downloading photos and videos from the tumblr tag search, you'll have to do some steps:
    1. Go to Settings, click on the Connection tab and fill in your tumblr email address (login) and password, then click the Authenticate button. If the login was successfully, the label will change and display your email address. The email address and password are not stored locally on disk but cookies are generated and saved in %LOCALAPPDATA%\TumblThree in json format.
    2. Add the search url including your tags separated by plus signs (+) in the url (e.g. https://www.tumblr.com/tagged/my+special+tags).

Key Mappings:

  • Currently mapped keys:
    • double click on a blog adds it to the queue
    • drag and drop of blogs from the manager (left side) to the queue
    • space -- start crawl
    • ctrl-space -- pause crawl
    • shift-space -- stop crawl
    • del -- remove blog from queuelist
    • shift-del -- remove blog from blogmanager
    • ctrl-shift-g -- manually trigger the garbage collection

Getting Started:

The default settings should cover most users. You should only have to change the download location and the kind of posts you want to download. For this, in the Settings (click on the Settings button in the lower panel of the main user interface) you might want to change:

  • General -> Download location: Specifies where to download the files. The default is in a folder Blogs relative to the TumblThree.exe
  • Blog -> Settings applied to each blog upon addition:
    • Here you can set what posts newly added blogs will download per default. To change what each blog downloads, click on a blog in the main interface, select the Details Tab on the right and change the settings. This separation allows to download different kind of post for different blogs. You can change the download settings for multiple existing blogs by selecting them with shift+left click for a range or ctrl-a for all of them.
    • Note: You might want to always select:
      • Download Reblogged posts: Downloads reblogs, not just original content of the blog author.

Settings you might want to change if the download speed is not satisfactory:

  • Connection -> Concurrent connections: Specifies the number of connections used for downloading posts. The number is shared between all actively downloading blogs.
  • Connection -> Concurrent video connections: Specifies the number of connections used for downloading tumblr video posts. The vt.tumblr.com host regularly closes connections if the number is too high. Thus, the maximum number of vt.tumblr.com connections can be specified here independently.
  • Connection -> Concurrent blogs: Number of blogs to download in parallel.

Most likely you don't have to change any of the other connection settings. In particular, settings you should never change, unless you're sure you know what you are doing:

  • Connection -> Limit Tumblr Api Connections: Leave this checkbox checked and do not change the corresponding values of 90 connections per 60 seconds. If you still change them, you might end up with offline blogs or missing downloads.

Further Insights:

  • Note: All the following files are stored in json format and can be opened in any editor.
  • Application settings are stored in C:\Users\Username\AppData\Local\TumblThree\.
  • You can use the portable mode (settings->general) to stores the application settings in the same folder as the executable.
  • For each blog there is also a database (serialized class) file in the Index folder of the download location named after the blogname.tumblr. Here blog relative information is stored like what files have been downloaded, the url of the blog and when it was added. This allows you to move your downloaded files (photos, videos, audio files) to a different location without interfering with the download process.
  • Some settings aren't hooked up to the graphical user interface. It's possible to view all TumblThree settings by opening the settings.json in any editor located in C:\Users\Username\AppData\Local\TumblThree\. Their names should be self explainatory. Some notable settings to further fine tune the application include:
    • BufferSize: Allows to set the buffer size for downloading binary files (photos, videos) in multiples of 4KB. The default is 2MB, thus the BufferSize has a value of 512. Increasing this value reduces disk fragmentation as more of the file is kept in the memory before it gets written out to the disk but increases the memory usage.
    • MaxNumberOfRetries: Sets the maximum number of retries if a tumblr server forcefully closes the connection. This might regularly happen on the tumblr video host (vt.tumblr.com) if too many connections were opened in parallel. After the limit is exhausted, the file is left truncated, but is also not registered as a successful downloaded. Thus, the file can be resumed in the next crawl.
    • TumblrHosts: Contains a list of hosts which is tried for downloading _raw photos if the photo size is set to raw. If none of the hosts contains the _raw version, the actually scanned host is tried with the next lower resolution (1028).

Changelog:

2018-07-05:

  • Implements the Tumblr login process and cookie handling in code instead of relying on the Internet Explorer for the Tumblr login process.

2018-06-09:

  • Fixes hidden Tumblr blog download problems caused by the new Tumblr ToS.

2018-05-20:

  • Programmatically agrees to new ToS and GDPR.
  • Implements SVC authentication changes. The SVC service is used to display the dash board blogs (i.e. hidden tumblr blogs). Changes in this internal Tumblr api prohibited TumblThrees access.
  • Saves the last post id in successful hidden tumblr downloads.
  • Improves the text parser of the tumblr api and tumblr svc data models. Separated the slug from the url as the data models are inconsistent. Separated the photoset urls from the photo urls. Moved the date information into a separate column.
  • Minor text changes of some user interface elements.

2018-04-18:

  • Updates the tumblr blog crawler and the hidden tumblr datamodel to reflect tumblr api changes that break blog download of previous TumblThree versions.

2018-02-28:

  • Allows to download only specific pages of hidden Tumblr blogs and in the tumblr search.
  • Improves the proxy settings. TumblThree now uses the default Windows (Internet Explorer) settings if not overridden within TumblThree.
  • Changes the behavior of the timeout value (Settings->Connection->Timeout). The timeout value now counts file chunks of 4kb instead of the whole file download, thus it should better detect if a download is stalled or a connection dropped without canceling active downloads of larger files (e.g. videos).
  • Changes default timeout value (for new users) from 600s to 30s.
  • Fixes possible download of the same photo but with different resolutions. This happened if the _raw file download was interrupted (the timeout hit), then the same photo was queued for download with the _1280 resolution. If the blog was then subsequently queued again, the _raw file was downloaded next to the _1280 file.
  • Fixes reblog/original post detection in the tumblr hidden crawler.
  • Fixes check blog status during startup-option.
  • Fixes download of password protected tumblr blogs.
  • Adds Mixtape, Lolisafe, Uguu, Catbox and SafeMoe parser (thanks to bun-dev).

2017-12-31:

  • Fixes a bug that released the video connection semaphore too often. That means the slider in the settings for limiting the video downloads didn't work at all. It should properly limit the connections to the vt.tumblr.com host and prevent incomplete video downloads now.
  • Includes a rewrite of the blog detection during blog addition. It should reduce latency if you mass add blogs by copying urls into the clipboard (ctrl-c). Offline blogs aren't added anymore.
  • Notifies the user when a connection timeout has occurred. The message states whether the timeout has occurred during downloading or crawling. If it happened during crawling, you might want to re-queue the blog at some point to grab missing posts. A connection timeout should only happen if your connection is wonky. You can decrease/increase the timeout in the settings (settings->connection).
  • You can now specify in the Details-panel for each blog where its files should be downloaded. If the text box control is empty, the files are downloaded as in previous releases in the folder specified in the global download location (settings->general), plus the blogs name.
  • Imgur.com linked albums in tumblr posts are now entirely downloaded if enabled (details panel->external->download imgur). Previously, only directly linked images were detected.
  • Adds an option to load all blog databases into memory and compare each to-download binary file to all databases across TumblThree before downloading. If the file has already been downloaded in any blog before, the file is skipped and will not be counted as downloaded. You can enable this in the settings (settings->global).
  • Allows to add hidden tumblr blogs using the dashboard url (i.e. https://www.tumblr.com/dashboard/blog/blogtobackup).
  • Allows to add all blog types without the protocol suffix (i.e. wallpaperfx.tumblr.com, www.tumblr.com/search/cars).
  • Adds an option to enable a confirmation dialog before removing blogs (#186, #130, #98). It's off by default.

2017-11-17:

  • Adds support for downloading Imgur.com, Gfycat.com and Webmshare.com linked files in tumblr posts.
  • Improves downloading of tumblr liked/by photos and videos.

2017-10-20:

  • Restores bandwidth limiter functionality.

2017-10-13:

  • Changes the default _raw photo host.

2017-10-09:

  • Fixes crawler stop in hidden tumblr blog downloads.
  • Adds options to set the default blog settings for the download from time, download to time and tags in the settings menu.
  • Adds some (ar, el, es, fa, fi, he, hi, it, ja, ko, no, pa, pl, pt, th, tr and vi) google translate translations.

2017-09-08:

  • Can download password protected blogs of non-hidden blogs.
  • Minor UI updates.

2017-08-22:

2017-08-21:

  • French, Spanish and simplified Chinese translations.
  • Removes user interface lag during blog addition.
  • Allows to set the buffer size for downloading binary files in the settings.json in multiples of 4KB. The variable is called BufferSize. The new default is 2MB, thus the BufferSize has a value of 512. Previously it was set to 4KB, but apparently Windows does not do any useful caching on NTFS if multiple writes are concurrent and async. Thus, this should reduce disk fragmentation.
  • Uses .NET Framework 4.6 now as it should be available for all supported windows versions (Windows Vista and above).
  • Improved the selection handling in the details panel. If multiple blogs are selected, old values are now kept if they are the same for all blogs and changes are immediately reflected.
  • Audio file download support for tumblr and hidden tumblr blogs.
  • More code Refactoring.

2017-07-03:

  • Can download hidden (login required/dash board) blogs.

2017-06-30:

  • Improved performance and bugfixes.

2017-06-20:

  • Downloads high resolution (_raw) images.
  • Updated translations (German and Russian).
  • Applies changed settings immediately.

2017-06-04:

  • Sets the date modified date in the Explorer to the posts time.
  • Allows to download single or ranges of blog pages.
  • Full screen media preview.

2017-05-20:

  • Option to skip reblogged posts.
  • Improves detection of inlined photos and videos in text posts (e.g. in answer posts).

2017-05-14:

  • Portable mode.
  • Downloads liked photos and videos.

2017-04-18:

  • Code refactoring.
  • Uses async/await in most of the code instead of tasks from the threadpool.
  • Uses a consumer producer pattern for grabbing and downloading as the Tumblr api v1 is now rate limited.
  • Downloads are now resumable.
  • Data files are now saved as json instead of binary.
  • Reduced memory usage by layering off the downloaded file list and only load it if needed.
  • Improves ui responsiveness.

2017-01-08:

  • Improves the speed of the network code.
  • Adds an option to use a http proxy.
  • Downloads inline images of tumblr posts.
  • Added Russian translation.

2016-12-13:

  • Improves the ui scaling of the main window for smaller resolutions.
  • Prevents crawling of offline blogs.
  • If the same blog is multiple times in the queue and already once active, any other free crawler task will skip and remove any already active blog and proceed to the next inactive blog in the queue.
  • Improved german translation.

2016-12-10:

  • The check for already downloaded files is now independent from the actual host and based entirely on the filename. It look likes the host/mirror does actual vary which would result in a reload of the file since its url changed.
  • Add scrollbars to the settings window if the controls do not fit.
  • Safely replaces blog indexes. If there is an error (e.g. no disk space left) during the update of the index file, the old state should not be corrupted anymore.
  • Changes some color and adds an alternate color for the blog manager.

2016-11-23:

  • Fixes application crashes which occurred by adding tumblr blogs without title or description.
  • Decreases determination time of already downloaded files for large blogs (>100,000 posts) by at least three orders of magnitude.

2016-11-22:

  • Creates more meta information (post id, reblog key, timestamp, tags, slug, title) of the posts, including image, video and audio types.
  • Fixes the progress calculation by adding the found duplicates to the progress. Also states them in the details window.
  • Fixes a locking issue for the meta files (*.txt) which resulted in incomplete downloads.
  • Updates the details and settings view for a better understanding on how to use the application.

2016-11-20:

  • Fixes proper counting of downloaded files.
  • Fully implements the details window (context menus, etc.).

2016-11-18:

  • Fixes the initial automatic queue restore function.
  • Fixes the autodownload function.

2016-11-16:

  • Picture- and videopreview in the details window.
  • Allows the download of text, audio, quote, conversation, link type posts.
  • Download of text, audio, quote, conversation, link and .gif images are based on each blog instead of a global setting and can be turned on/off in the details view. The settings in the settings window are used as template for newly added blogs.
  • Modified .tumblr index files get now always saved upon application exit regardless of the crawlers state. Previously if the application was closed during an active crawl, the index wasn't updated.
  • Inlined the WAF code under lib for easier project setup for newcomers that want to contribute code.
  • bugfixes, UI and memory enhancements.

2016-10-15:

  • Bandwidth throttling.
  • Connection timeout settings.
  • auto queue and start download function.
  • save states of the UI (column size and order).
  • download of hidden blogs.
  • fix proper saving of the ratings and tags.

2016-06-11:

  • Added German translation.

2016-06-10:

  • Support for tumblr.com hosted videos. Check the settings window to enable video download (default: off).

2016-06-08:

  • Tag crawling now properly working. Also it's case-insensitive now.
  • Fixed crash upon blogs with zero-image count in the queue list (e.g. blog is offline, or tag search didn't evaluate any images).
  • Fixed randomly occurring crash in the clipboard monitor.
  • Changed icons (requested by the TumblOne creator).

2016-04-12:

  • Now with progress output in the Queue tab (during url crawling for imageurls -- the number of posts evaluated; during downloading -- the current image url).
  • Added missing resume button in the taskbar control.

2016-04-11:

  • Support for urls starting with https:
  • Fixes application crashed upon pressing the stop-button due to improper exception handling
  • Now saves the index file at every time. Previously the application would exit if the crawling processes was still active without properly waiting them to finish and save its state. Now there is a grace period for the tasks to finish. Same was true if the crawl was paused and then exited.

Download:

Comments

anonymous (not verified)
Sun, 17/09/2017 - 19:13

Apologies for delay

Managed to finally get it to work, it appears that the TumblThree was calling on old Appdata data. I hadn't updates the TumblThree over the last few changes, and it appears on of those changes related to the previously saved accounts. Despite what I thought was a clean install of the newest patch, there was some fault in the new installation. So for some reason I can't fathom, purging everything Tumblthree related (including Appdata folder) and simply re-installing and re-entering all of the accounts with tags worked.

Apologies for the delay, hopefully this issue was simply my fault during installation.

anonymous (not verified)
Tue, 12/09/2017 - 23:34

So I noticed that you have the ability to export a list of blogs that are added to TumblThree, but not the ability to import from such a list. Could this at all be implemented in the future?

zab
Wed, 13/09/2017 - 03:13

i didn't see a need for this, since you can simply open that file, select all (ctrl-a), copy (ctrl-c) and all blogs will be added if you have the Clipboard Monitor activated.

anonymous (not verified)
Thu, 14/09/2017 - 02:32

I guess that makes sense. I turned it off so it would stop adding every tumblr I copied. But if such a feature is technically already in, is there at least a way to make it more obvious?

anonymous (not verified)
Thu, 14/09/2017 - 03:27

I'm afraid I don't have a proper example, but I'm beginning to to get the feeling that conversation posts aren't properly being detected and downloaded. I have been noticing images relating to them not being downloaded at all.

zab
Thu, 14/09/2017 - 20:52

No reproducible steps means probably no fix. It would take me way to much time to dive into any non-obvious issue than I've free time right now. Besides that, I've just tried to reproduce it myself, and I cannot even add anything to an conversation post ...

No offense, but there are plenty of other real bugs/improvements to do than trying to fix something that people feel might be broken.

I'm developing alone, and cannot really keep up with all the requests, thus it means I've to prioritize somehow.

anonymous (not verified)
Thu, 14/09/2017 - 23:19

I may actually be refering to it wrong then. It's probably not a conversation post, but I guess reblog posts where the users reblog with an image.

With that realization in mind then, I do realize in most cases, I turn off the "download reblogged posts" deal for nearly everything except art blogs that reblog the art from the main blogs. Sadly in my case, I don't know what blog was having the issue to test this though. If I come across it again, I'll immediately let you know

anonymous (not verified)
Sun, 17/09/2017 - 15:02

So I'll give a series of example posts from a particular account:

https://tenkaboutholes.tumblr.com/post/165215488443/tenkaboutart-high-cl...
https://tenkaboutholes.tumblr.com/post/164907760608/tenkaboutart-charact...
https://tenkaboutholes.tumblr.com/post/164752720818/tenkaboutart-charact...

So as I was describing, the images that are being added in this persons reblogs are NOT being picked up at all.

The original image in the reblog is, but the follow up is not

zab
Sun, 17/09/2017 - 16:46

hmm ..

It's a photo added to a photo post. I've actually excluded looking for inline photos in photo posts on purpose (any only there) since that will add the same original photo again as well which i wanted to prevent. Also I wasn't aware of that you can add photos to photo posts, but since this seems to be possible, I'll have to think about how to grep all the photo urls without adding them multiple times.

I'm not sure right now when I've time for adding the code for this, but I'll add a note in the release note to let you know.

Thanks for letting me know!

Edit: I've added it already. I'll briefly test it this evening and update it afterwards.

anonymous (not verified)
Tue, 19/09/2017 - 01:28

Any possible chance you can add support of downloading tumblr blogs through http://www.gramunion.com/ ?
It saves me time from having to login for certain blogs. It also might be easier to backup a tumblr through this site as it sorts it and doesn't change based on theme.

I don't think it can access hidden tumblrs though.

zab
Tue, 19/09/2017 - 20:06

Well, it's basically the same then, right? Looks like they take tumblr.com and just put their webdesign over it. Plus you might have some comfort/better design choices.

Honestly, I don't see any reason why I should spend time in that. It would even download the same content.

What login time are you talking about exactly? You only have to login to tumblr.com/TumblThree once as well.

Feel free to add it to TumblThree. It's open source after all, but I personally won't do it. For me It makes more sense to add flickr or instagram.

anonymous (not verified)
Mon, 25/09/2017 - 14:27

I have this blog here for you to look at, it's a test blog, and it's hidden. For some reason, TumblThree isn't downloading from hidden blogs anymore. It will, however, download reblogs that blog has made. I'm not able to make it REPEAT downloading only reblogs, but I assure you that is happening.

And yes, I'm following this blog. And yes, I've made sure my authentication is going through

If you need any further information, do let me know

http://test1251541121521.tumblr.com/

zab
Mon, 25/09/2017 - 19:12

Not really my fault. The responses from the tumblr server seems to vary. Instead of sending a List of Player objects they are sending a simple string for some reason. Thus, the deserialization fails.

I don't really have time to fix this properly, and quite frankly, zero interest right now in fixing things for a software I'm personally not using.

I've changed the code and re-uploaded the releases where that part of the server response is simply not used at all. It should be working for you now. It only affects a missing caption in the the video meta data.

If someone wants to fix this, feel free to commit proper code.

anonymous (not verified)
Tue, 26/09/2017 - 08:28

No no, I get ya. I swear to you, if I was any better a programmer at all, I'd have started contacting you months back and just started doing my own work and research to fork and commit

anonymous (not verified)
Mon, 09/10/2017 - 15:23

I want to add an observation I'm making into this, though I know you said you don't have time or interest to fix this properly.

It seems that the only visibility option affecting this is the one regarding whether the blog is viewable within the tumblr dashboard or not. That is to say whether you see a full website or if you see the website loading on the side of the tumblr dashboard.

Again I know this isn't much, and I know you're busy with other things, so I'm hoping someone out there maybe who is interested can maybe take the time to take a look over and see.

zab
Mon, 09/10/2017 - 16:12

Of course thats the option that affects it, because that's exactly what toggles the blog to be hidden (== requires a login). And the TumblThree downloader for hidden blogs is a different one (completely different code) as for non-hidden blogs.

But I've already fixed it in the v1.0.8.24 release in the a way that at least the example blog works on the cost of discarding some video meta information.

anonymous (not verified)
Mon, 09/10/2017 - 20:45

Hold up, I made that post and had a better example blog to that one. I mainly download images.

http://hecchidechu.tumblr.com/

ACTUALLY WAIT! I FOUND THE ISSUE BY SHEAR LUCK AND CAN EXPLAIN IT TO YOU BETTER! Or at least a lead. For all blogs, I don't have "Download Reblogged Posts" unless I know it's like a persons dedicated art blog where they reblog from their main blog. On a weird hunch, I decided to turn it on for this one in particular and VOILA, I was downloading all of their posts and stuff again. Downside is if they're reblogging, I'm getting reblogs too.

But I do hope this helps then.

zab
Mon, 09/10/2017 - 21:25

Thanks. Found it, and it helped.

It actually was a different issue. I'll fix it tonight and upload a new version.

cedmen (not verified)
Fri, 06/10/2017 - 02:08

Hi, I just started using Tumbl with the latest version. From day one everytime I open the app I got those errors.
#Error1 Could not restore UI setting
#Error2 The queuelist couldnt be loaded

Crawl function does almost nothing, evaluates few tumblr post, stop and does nothing. The very few time it start downloading something it only downloads 50 ish photos out of 5000.
Am I missing something? Sorry if its a noob problem

zab
Fri, 06/10/2017 - 02:54

To get rid of the errors, which also prevent that the queuelist and settings will be updated, delete the settings files in C:\Users\yourusername\AppData\Local\TumblThree\Settings\. That's also written in the release notes ...
Then see if that fixes your problem without messing in the settings, except for changing the download location.

Otherwise i probably cannot help, since TumblThree downloads here and your description doesn't really say me/hint to something..

cedmen (not verified)
Fri, 06/10/2017 - 03:07

I saw it on the release notes and I tried it couple of times before posting but with no luck.

anonymous (not verified)
Sun, 08/10/2017 - 12:42

1. Maybe it is more convenient to store all translations in separate subfolder?
2. I believe you should've keep existing translations (Russian, German, French, Spanish and Chinese) instead of replacing them with GoogleTranlate results. Even if some of them were incomplete and had some strings weren't translated, they're still better and much more correct. At least the Russian one is now mostly rubbish.

zab
Sun, 08/10/2017 - 12:56

2. It's obvious, but takes time for me to go over all of them, and I simply haven't had that time. I'll eventually restore the manually translated back into their spots where I haven't had updated to original text. The point it that I depend on other people for the some of the translations, and the Russian for example wasn't updates for month by someone how can speak it.

1. Not possible to change the structure. But they are in subfolder anyways? I'll not upload 20 files separately, each for one of the translations since I'm currently packing all the releases manually. Sure, it could be done automatically, but that brings me back to point 2. Time ..

anonymous (not verified)
Sun, 08/10/2017 - 20:20

1. Thanks, will wait for new version with reverted translations. Just wanted to say that if all translation subfolders would be stored in "./Lang/" folder ("./Lang/en/...", "./Lang/de/..." etc.), it would look more nice and neat from aesthetical point of view.

2. Could help with translation, but have zero experience with GIT/SVN and all this push/commit/diff/etc stuff. Is it OK if I upload updated translated strings on PasteBin and post link here in comments section?

zab
Sun, 08/10/2017 - 20:56

Yes, of course, it's okay if you upload it somewhere. You can also send to me via email.

Thanks for your work!

Claf (not verified)
Wed, 11/10/2017 - 23:25

Its a great tool, but something is wrong with duplicates. Is there way to force downloading duplicates? When I download my "likes" it only downloads less then half of my liked Images. 200 out of 450 while the rest is marked as duplicates which is simply not the case. I checked manually and found lots of Images in my "likedby page" that are completely ignored by thumblethree because of beeing somehow recognized as duplicates. Any help would be appreciated! Thanks again for the great tool.

anonymous (not verified)
Thu, 12/10/2017 - 09:27

I'm gonna note that this isn't a problem of your fault or the program's fault, something does seem to be going on with tumblr.

For some reason the RAW files are no longer accessible, either by manual means or by means of the program. I'm investigating as to what's going on, but I'm hoping it's something easily resolveable

zab
Thu, 12/10/2017 - 09:50

Thanks for the info.

I've just opened an issue on my github site with your information. Maybe someone else can help figuring out how to solve this.

anonymous (not verified)
Thu, 12/10/2017 - 10:45

There's nothing concrete yet. I'm manually checking, and it seems to be fairly random. Some images WILL give a RAW file, some won't at all. Sometimes you can get the RAW if you prod for it using a proxy, other times it won't. It's an all over the place mess it seems. I'm keeping an eye on a forum post on e621 on the subject. Cause if SOMEONE is gonna figure it out, I place bets it's someone on there.

I just hope it's a server issue is all and not a purposeful thing

anonymous (not verified)
Thu, 12/10/2017 - 15:13

Replacing media.tumblr.com with data.tumblr.com might work for a while.

zab
Thu, 12/10/2017 - 15:23

Thanks. I'll check it tonight ..

Just for the info, you can already change the host by modifying the TumblrHost array in the Settings.json in the C:\Users\Username\AppData\Local\TumblThree\Settings\

per default it looks like this:

  "TumblrHosts": [
    "media.tumblr.com"
  ],

change it to:

  "TumblrHosts": [
    "data.tumblr.com"
  ],

and it should try to download the _raw files from data.tumblr.com instead of media.tumblr.com.

You can also add both separated by a comma, but at some point it might get quite costly since each of those hosts is tried for the raw photo, than it falls back to the scanned host with the 1280 resolution if non of them resulted in a successful download. Thus you might end up trying several hosts for a single photo which might add up to some delays.

TumblThree has to be closed for the modification as the Settings.json is only once loaded during startup.

nanopulga (not verified)
Thu, 12/10/2017 - 18:25

Oh good to know this replacemant. I was worried about not accesing to "raw" images. I hope, as someone said before, that is just a problem related with Tumblr servers. Or, if it's just a change to access to them, at least stay this change and is not Tumblr trying to forbidden the acces to "raw" images.

zab
Sat, 14/10/2017 - 21:38

I've filtered out your email address.

If you like, you can do the translation.

There is one translation file containing all the string for the user interface, and one for the underlying application. The user interface is more important however.

The best would be if you could replace all the English sentences with your translation in the linked files between the value tags and upload/send me the file by email. If that's to much work to me, I also take simple text files ..

Thanks alot for your effort!

zab
Thu, 12/10/2017 - 16:09

Well, at least I can access most of them using the data.tumblr.com host, and that I cannot access any photos with the old media.tumblr.com host anymore, that's true. But most of them aren't really _raw'ish, are they? But I'm not sure if that was ever different though ...

anonymous (not verified)
Thu, 12/10/2017 - 17:54

Well it depends on the image. In some cases, they weren't all TOO different being that they were under the resize. But in cases where the image is like some super large, 16Megepixel image, it would make a world of difference.

Speaking of megapixels, if the image in question was a photo, you'd also get the EXIF data as well.

When you're downloading the RAW image, you are downloading (and we have to hope this is still the case) the raw file that the uploader originally uploaded.

Plus, and I know I'm right, let's say you did upload an image to the exact resolution that Tumblr supports. Tumblr would still not just resize, but recompress (re-encode?) the image to their standard. A JPG to JPG compression. And I will let you know, it has shown in the weirdest way, with even whole lines of pixels moved over.

So if you're wondering if it's even worth the fuss, it very much is worth the fuss.

I am gonna note that sadly, there may need to be some tweaking to be done. While TumblThree will access the images from data.tumblr, it doesn't seem to always download the _RAW files. I can verify the _RAW files are accessible. It does acknowledge they exist, but won't download.

Hilariously, if I keep making TumblThree queue up the blog (and ensure Force Rescan is enabled), it does eventually download the RAW files. Though this is resulting in a sort of duplicates situation of _raw and _1280 images. And of course, it seems random whether the software will download the _raw or not

anonymous (not verified)
Thu, 12/10/2017 - 18:02

Actually I'm continuing to make observations as I keep running this particular blog. I think I'm noticing that somehow images are being skipped over? I am certain what I'm seeing can be reproduced in other blogs, this one I happened to decide to add today and try downloading to test.

I can't be super certain of my claim, but I am observing it download images of _1280 that i don't think it's downloaded before?

anonymous (not verified)
Thu, 12/10/2017 - 19:23

Same anon here. There's a user on the TumblrHelp subreddit who thinks this is a CDN issue, which may explain why I'm hearing of some users having the issue and others not. May also explain the proxy bit.

I do hope this is just a temporary issue though. And not anything purposeful. But I also wish you luck in this as well. I know this isn't a full time deal for you. But if I can help in providing as much information on this, and that at all can help, I will certainly provide my assistance that way

anonymous (not verified)
Fri, 13/10/2017 - 01:10

Okay I have two things to report on this:

1) any time I open TumblThree, it resets the string under "TumblrHosts" back to "media.tumblr.com"

2)

Well i was gonna report that deleting the folder and then deleting and readding the blog still bring me _1280's and not _raw's, but I can't well report this cause I don't know if this is because TumblThree isn't reading the url from data.tumblr.com right or if tumblthree is simply sort of "hard coded" (loosely) to download from media.tumblr.com

This is difficult to discern. But I hope this information helps

anonymous (not verified)
Fri, 13/10/2017 - 04:49

Okay, update on this: I made sure to set the "settings.json" file as a read-only file as a drastic measure to ensure that the "TumblrHosts" value doesn't revert to some default value. I'm sad to report that while this does keep the value from changing, I still find TumblThree downloading _1280's and not _raw's.

So that's a bummer. And that's my update on the matter

zab
Fri, 13/10/2017 - 21:20

Thanks for doing all the testing even though my information about the settings.json wasn't correct I've noticed. My intention was to be prepared for exactly this case, but I've actually never tested the feature. The property was read-only in the code, thus changing the settings.json didn't do anything.

It's fixed now however, and I've also updated the release that it defaults to the data.tumblr.com host.

Thanks again!

RawrxD (not verified)
Fri, 13/10/2017 - 12:12

Since 12/10/2017 Raw images now require you not only to change 1280 and etc in the link, but to change "media" to "data".

T3-Q.Anon (not verified)
Fri, 13/10/2017 - 20:29

Wall of Text anon from before.

So unless something comes up regarding "media.tumblr", it is to be assumed that _raw is only accessible via "data.tumblr". I'm uncertain whether the change to "data.tumblr" would stop the downloading of any other media though, doubt it. Testing might be needed.

I suppose the best thing to do is to allow for a choice between "media.tumblr" and "data.tumblr" in the options, cause (at least for me) changing the Settings.Json doesn't do anything, and TumblThree reverts the changes I make manually.

zab
Fri, 13/10/2017 - 20:34

Yes, I've noticed that changing the settings doesn't work. My intention was to make it changeable for exactly this situation (and never tested it), but I've made the property actually read only in the code, thus changing doesn't do anything.

It's fixed already, but now I've noticed that tumblr seems to host their stuff (now) on amazon S3, and I get wonderful exceptions when i try to download from the data.tumblr.com host using ssl about an invalid certificate. They seem to have forgotten to update the hosts in their ssl cert ...

I'll quickly fix this, then upload a new release.

T3-Q.Anon (not verified)
Fri, 13/10/2017 - 23:29

Thank you for your hard work in all this. It is very much appreciated, more than you can imagine

anonymous (not verified)
Sat, 14/10/2017 - 06:07

Hello, I have been attempting to use Tumblthree for quite a while but for some reason, the first five buttons and the check Clipboard buttons do not work completely, meaning I can't add anything to the application at all. The Add Blog button doesn't do anything, no errors or even indication that I even clicked it. I've tried authenticating, redownloading, even trying other versions of the program, but nothing is working. I even tried redownloading it on a laptop, still no success. What can I do to make it work? I'm operating on Win10.

On a side note, how well does it download posts? I know a blog I want to save images off of, though it has 123671 posts to go through, and while Tumbltwo works fine for most blogs, all it does when I try to download a specific tag is that it scans through the posts and stops, saying the "queue is finished" even though it didn't download any images.

Pages