This process is what I follow to upload images to the bot. Some of the steps are optional, and some steps can be performed in different ways than how I do it or in a different order. My methods are probably not the fastest possible, but if you follow them to the letter there shouldn't be any problems. All the software I link to is freeware that has worked for me. A small number of things (mostly command-line) may only work on Windows; if you use a different operating system it should not be hard to modify these to the correct version for your OS.
Before you start, I recommend reading through the whole process at least once to get an idea of what needs to get done and how much work this entails.
Step 0: Check bot status
You'll probably want to do this several times during the process (at the very least, when you start gathering and when you're ready to upload), so it's at the top. There are two commands you can post into chat to check the current status of the bot.
#status will give you statistics about the bot itself, which has more than just the image-posting module. Most of these statistics aren't very useful but you may need it occasionally.#post status gives statistics about the image-posting module. It will tell you whether automatic posting is enabled, the parameters, and how many unposted images are remaining. The number of images remaining is probably the most important of these for this process. If the number is too low (the bot posts about 160 per week) we need to upload a lot more soon. On the other hand, if you're uploading niche images, don't upload 2000 when there's only a couple hundred images remaining in the bot, since those will dominate. Basically, so long as you apply a bit of common sense, this is pretty easy to understand.
Step 1: Gather images
You'll need to download a large number of images that you like. Ideally, these should be the highest available quality for each image. Try to avoid images that are cropped, downscaled, or otherwise modified from the original.
There are several image board sites which I use for this purpose. Most of these are not work-safe. They all have slightly different purposes, but (other than pixiv) are mostly equivalent so you can pick whichever you prefer or use any combination. You may need to use other sites as well if these are not sufficient.
The use of these sites is mostly self-explanatory. You can search for tags which represent certain properties of the image. Most of them also allow for restricting your search to images considered "safe", which may not be exactly the same as what our chatroom considers safe. All of them allow for downloading original images.
I suggest staying away from more general image sharing sites like Tumblr, imgur, and Facebook. These are known to apply (sometimes severe) compression to the images. In addition, modified images (cropped, filtered, etc) are quite common. These generally do not give great results with our bot, and you can almost always find higher quality versions of the same images on other sites or with one of the various reverse image search engines.
You'll want to download all the images into a single folder with nothing else in it. The number of images you'll need to get is up to you, but keep in mind that most of the later steps aren't significantly harder for thousands of images than it would be for 10, so it may be useful to get significantly more than you'll actually need and remove some later.
As a note, I recommend avoiding animated gifs. At least some of the steps below (for instance, creating thumbnails) are much harder for them. It should be possible to get things to work with animated gifs, but they're a lot more work than other formats. If you stick to JPG, PNG, and stationary GIFs, you shouldn't have any trouble in the later steps. Also, users in the main room may be annoyed if an animated gif is particularly distracting.
While I could give a great deal of advice on how to best optimize this step and what is/isn't worth doing, this is really a topic for somewhere else. The goal of this post isn't so much to tell you how to download images as to tell you what to do once you have finished downloading them. I suggest experimenting to see what works for you.
Note: It's highly recommended that once you finish downloading, you make a backup folder by copying the existing one to a secure location, and update this with each step as necessary. It's not hard to accidentally destroy things in some of the later steps, so having a backup will potentially save many hours of work.
Step 2: Rename images
If you, like me, used a half-dozen different sites with different naming conventions to get your images, it may be somewhat annoying to have them all in the same place. This step is optional, but will rename them all randomly with a fixed format, which makes them easier to work with. If you don't do this, things should still work fine as long as none of your filenames ends with _thumb before the extension.
For this, I use this random names script. It's relatively self-explanatory. You download the script and put it in the same directory as your image collection. Then run the script. It will randomly rename each of the files in the directory. After that, move the script and the translation file it creates to a different directory. For whatever reason, it seems that site is now broken and the script isn't available there, so I've uploaded the script here if you still want it. Alternatively, you can use Advanced Renamer (see below) for this step.
Occasionally, when running this script, you'll be left with some junk files in addition to the correct ones. Simply sort the folder by extension to find the junk ones and delete them.
Step 3: Remove duplicates
If you have more than a few dozen images, there's a decent chance you have some unintentional duplication. This is very difficult and time-consuming to find by hand, since in many cases the images are not exactly the same, but are closely related.
For this, I use VisiPics. It will identify not just exact duplicates, but also similar images. You can adjust the sensitivity of the checks it uses; it may be useful to test this some. I use fairly strict sensitivity, and also check all identified duplicates by hand before deleting them. This was sufficient for anime-themed images.
For an archive of over 20000 images, this resulted in a few hundred duplicates, but your results may vary. The full scan takes a few hours with that many images.
Unfortunately, there's no practical way to check for duplicates between different users' uploads. This could be added as a feature to the bot in the future, but right now, we have to live with the fact that there will be some rare duplication when multiple users add the same image. This will probably be relatively rare, but for that reason, it's especially important to ensure your own contributions are as duplicate-free as possible.
The above steps (in principle) only need to be performed once, and can work for an arbitrarily large collection of images. The steps below need to be done each time you want to upload some of them to the bot. If you aren't uploading all of them at once, start here on subsequent uploads.
Step 4: Sort images
So you now have an archive of a lot of images without duplicates. Most likely, you won't want to upload the whole thing at once to the bot, and there may be other concerns that you have about some of the images.
At this point, I strongly recommend going through the images and hand-checking each one. There are many reasons to perform a second check, such as making sure all images are high quality and appropriate for the room. I generally sort my images into 3 folders:
- Post: There are no issues posting this image in the chat room.
- Low Quality: The image may be artistically low quality, have unusual dimensions, or otherwise be not worth posting in chat.
- Questionable: The image is not low quality, but may not be appropriate for chat.
This sorting takes a few seconds (5-10) per image. Low quality is essentially entirely subjective, and I don't think I can quantify the methods I use here, nor is it necessarily useful to do so. Questionable is aimed at being more objective. Anything with nudity, underwear, or sexually explicit is put here. Beyond that, anything I personally find to be sexually suggestive goes here, and I have a couple other minor rules that aren't really worth noting here. The goal of this is not to completely remove all inappropriate images, which is an impossible task since in many cases it's almost completely subjective, but just to do a good enough job that people can deal with the remaining subjective cases in the room after they are posted.
Depending on your interests and images, you may want to modify this step significantly. I still highly recommend hand-checking each image before they're added, and removing ones you don't think should get posted. While there aren't currently any restrictions on who can upload to the bot, such restrictions might need to be added if users don't police their own uploads to some degree.
Step 5: Make thumbnails of images
Now, you should have a folder of images to post. The problem is that they're probably mostly large images, and some people in chat are on low-bandwidth connections. So that we don't end up causing them problems, we've developed a user script which will allow the chat version of the picture to be a small 300x300 image, while the link points (for those running the script) to the original. Accommodating this means you'll need to do a bit of extra work generating the thumbnail files.
The logic of the following operation can be illustrated like this:
Imagine the blue square is 300x300 pixels. Pictures with portrait and album orientation will be downsized to fit the square keeping the original height/width ratio, so the thumbnail is not distorted and not cropped.
For this step, I use ImageMagick. First, be absolutely sure to make a copy of the Post folder called "Thumbnails" or something similar. The two folders should be the same initially. ImageMagick will overwrite all the images in the Thumbnail folder with downsized versions, so don't run it on the main folder, only on the copied one. Everything we do in this step is in the Thumbnail folder.
Next, check the thumbnail folder for any small images. Chat will downsize any images to fit in a 300x300 pixel box. If you have any images where both the length and width dimensions are under 300 pixels, move them temporarily to a different location. These images are already small enough to act as their own thumbnail versions; you don't want to resize them. After you run ImageMagick on all the other images, move these back in the thumbnail folder. In most cases you will have few (if any) of these, but if you do have some you need to follow this step to avoid the chat version getting upsized.
Now, you have a folder of a lot of images that you want to downsize so that the larger dimension is 300 pixels. Open a command prompt and navigate to the thumbnail folder. Then run the following ImageMagick command:
mogrify -resize 300x300 -quality 100 *.*
This will take some time to run if you have a lot of images. The end result will be that all images in the folder are resized to fit in a 300x300 box. The quality 100 ensures that the resizing is done as faithfully as possible. It isn't necessary, but even with this enabled, most thumbnails end up quite reasonably sized, and at lower quality levels there may be noticeable compression artifacts on some images, so I recommend it.
In rare cases, this may generate junk files the same way that the renaming did; you should delete these the same as before. In rarer cases it may fail to resize a small number of images. In these cases, you can handle the remaining ones via running it again, or by one or more convert -resize 300x300 quality 100 filename.ext filename.ext commands.
After this step, it's recommended to check that the number of files in the thumbnail directory is still the same as the post directory, and that all of them have both their width and height less than 300 pixels. If so, this step worked.
Step 6: Rename thumbnails
Now you have two folders, one with thumbnails and one with the original images. Both of them should have files with the same names. You need to get one folder in which the thumbnails have _thumb added to their filenames.
For this step, I use Advanced Renamer. It can only handle 10000 files at a time, but that's more than you should be uploading at once anyway so it's fine. In principle, Advanced Renamer could work for step 2 as well.
Click the "add" button and click "directory", then find the thumbnail folder. Add this to the renamer. It should load all the thumbnail image file names; you can check that there's the right number of files loaded. For renaming method, use the rule <Name>_thumb.<Ext>. If you look in the right column of the middle table, you should see the new filenames of the pattern name_thumb.ext.
Once you check that everything looks right, click the "Start Batch" button to do the renaming. This should finish in a matter of seconds. Recheck the thumbnail folder one last time that it has the correct number of files and that they all have _thumb attached to their names.
Step 7: Generate list of all thumbnail filenames
This isn't actually used until step 10, but generating the list now saves some extra work in the form of eliminating duplicates. We want to generate a list of all the filenames of files in the directory. This can easily be done via command line.
Open a command prompt and navigate to the thumbnail directory (if you aren't already there from step 5). Then run the following command:
dir /b /s /a:-D > names.txt
This will create a new file called names.txt in this folder. Open it with Notepad++. The contents should look like this, likely with more lines:
C:\Users\Logan\Pictures\Example\Thumbnails\12810-22653-26093_thumb.jpg
C:\Users\Logan\Pictures\Example\Thumbnails\14349-27417-26639_thumb.jpg
C:\Users\Logan\Pictures\Example\Thumbnails\17822-1683-12809_thumb.jpg
C:\Users\Logan\Pictures\Example\Thumbnails\21372-19840-12974_thumb.jpg
C:\Users\Logan\Pictures\Example\Thumbnails\23193-19768-15737_thumb.jpg
C:\Users\Logan\Pictures\Example\Thumbnails\24102-18218-2554_thumb.jpg
C:\Users\Logan\Pictures\Example\Thumbnails\2416-24375-13269_thumb.jpg
C:\Users\Logan\Pictures\Example\Thumbnails\8713-11952-23783_thumb.jpg
C:\Users\Logan\Pictures\Example\Thumbnails\899-18705-32578_thumb.jpg
C:\Users\Logan\Pictures\Example\Thumbnails\names.txt
Delete the last line with names.txt and then save the file. Then move it to a safe location which you won't forget about.
Step 8: Upload images to Google Drive
Now, you should have 2 folders. One is named "Post" which has all the full-size images, and one named "Thumbnails" which has the chat-sized images. These two folders should have the same number of files, and the names should be the same except that the Thumbnails directory will have _thumb at the end of each filename. You should also have a text file with all the thumbnail names. Once you check all of these are correct, move the thumbnails to the "Post" folder. For things to work properly, they need to be in the same folder.
Now we're ready to upload the images. There are several minimum requirements for the uploading/hosting service you choose.
- Needs to keep images available indefinitely, i.e. no expiration date or view limits
- No watermarks
- Allows bulk uploads
- Allows hotlinking
- Stable urls
There's also some highly desirable features:
- Systematic url structure
- No image compression
- Easy interface
For all of these, Google Drive is the preferred place to upload. There are others that are in principle workable, such as Dropbox or Flickr, but Google Drive is also what all the various user scripts (including the thumbnail script) assume, so you should use it. While there are a couple of rare bugs which can cause headaches, overall it's the easiest way, and the 15 GB of storage per account lasts a long time (after which you can simply make another account).
Note in particular that you can not just copy a bunch of image links from boards like Danbooru or Pixiv. These sites almost universally block hotlinking. While the image will show up for you, it invariably will not for anyone else in chat. Also, you should not use services like imgur or minus, because these don't guarantee your image will still be visible a long time in the future, and the bot may take a long time to post the link. Doing either of these will probably only result in your images getting removed because no one can see them. From here on I will assume you are using Google drive.
In the drive, click "My Drive > Upload Folder" and upload the Post folder, which should now have both the original and thumbnail images in it. This uploading will take some time. Note that if it fails for any reason, it isn't easy to restart the upload; you're usually better off just deleting the failed attempt and trying again. The uploading doesn't usually fail though unless your internet connection goes out or you close the tab.
You'll want to check that the upload worked properly. Go to the folder and check that it has the right number of files, which is twice the number of images you are uploading. You can check this by going in and clicking any image. Once the image preview loads, in the lower left corner it will tell you the number of images in the folder, e.g. "1 of 18 items". If you have more than 50 images in the folder, it only loads 50 at a time, so you'll need to repeatedly scroll down to load more. Unfortunately, after a few hundred images are loaded on screen, the folder has a chance to crash if you scroll too quickly, so you'll want to do this slowly. Other rare bugs that sometimes happen are images failing to get uploaded or getting uploaded twice; keep an eye out for these since they're quite possible if you're uploading 1000 or more files at a time.
Note that this check can not easily be automated via the Google drive API. The reason is that the API fails when more than 500 items are in a single folder, and only finds 500 of them in many cases. This bug has been known to Google for a long time, but they haven't fixed it, and there's no indication that it is likely to get fixed.
Step 9: Share Google drive folder
Now that you've uploaded the "Post" folder, you should see it in your drive. Right click and click the Share button. In the pop-up menu, click Advanced, and then click the Change button next to Access: Private. Change this to "Public on the Web", and make sure it's set to "Anyone can view".
Unfortunately, recent interface changes have made the next step a bit harder, but it's still doable. Right click the folder and click "Get link". Copy the link url. It should look something like this: https://drive.google.com/open?id=0B37SNNYCVhA_fnRQd1JUQ05SOTJPaXNZS08xQm5lNTlUdVJCVFVpNkV1WXM4SEN3Q3lZM2s&authuser=0 Copy everything after id= and before &authuser=0, i.e. 0B37SNNYCVhA_fnRQd1JUQ05SOTJPaXNZS08xQm5lNTlUdVJCVFVpNkV1WXM4SEN3Q3lZM2s in this case. Take this long folder id string and copy it into the url https://googledrive.com/host/[folderid]. This will take you to a rather bland-looking page that has a lot of links. Note that the url you get is actually a redirect page, but unlike the page it redirects to, the link itself is stable. In this case, you'd end up with https://googledrive.com/host/0B37SNNYCVhA_fnRQd1JUQ05SOTJPaXNZS08xQm5lNTlUdVJCVFVpNkV1WXM4SEN3Q3lZM2s. If you upload a lot at once, you'll notice that this page doesn't actually have all of the links to your images, only 1000 or so. I don't know exactly which 1000 it has, but if you actually parse this page you'll find that it's missing a lot of links.
If you take this folder url and add to it the filename of one of the uploaded files, the url you'll get will look like https://googledrive.com/host/0B37SNNYCVhA_fnRQd1JUQ05SOTJPaXNZS08xQm5lNTlUdVJCVFVpNkV1WXM4SEN3Q3lZM2s/24102-18218-2554.jpg. If you go there, you'll see that indeed, this link hosts the image. While this process seems a bit sketchy, it's actually completely intended, supported, and stable. It isn't particularly well-documented or easy to find though, especially in the new interface. Check a few of these from a private browsing window to make sure everything seems to be working alright.
You've finished uploading and sharing the images now. You'll want to check here that all the images are publicly visible. There are some rare bugs which will make things not work for a small number of images. Sometimes (about 1 in 1000 files) a file will fail to get shared properly. You'll see it while you're logged in, but no one else can. Hence, you should open the folder in the main Google drive interface but in a private browsing window, and check that the number of files is the same as what you expect. If any are missing, you'll need to re-upload those individually. I've had to do this every time so far, so it's safe to assume you'll need to as well if you're uploading a lot.
Step 10: Generate list of thumbnail urls
All the hard work is done at this point, and you're potentially only a couple minutes from getting all the images added. Now, we just need to get the hosting urls for all the thumbnail images. Open the names.txt file from step 7 in Notepad++. All the lines should be of the form [path]\filename.txt. In my example, [path] is C:\Users\Logan\Pictures\Example\Thumbnails. Go to Search > Replace. In the "Find what" box, put [path]\ (i.e. C:\Users\Logan\Pictures\Example\Thumbnails\) and in the "Replace with" box, put https://googledrive.com/host/[folderid]/ (in our example, https://googledrive.com/host/0B37SNNYCVhA_fnRQd1JUQ05SOTJPaXNZS08xQm5lNTlUdVJCVFVpNkV1WXM4SEN3Q3lZM2s/). Your file should now look something like this:
https://googledrive.com/host/0B37SNNYCVhA_fnRQd1JUQ05SOTJPaXNZS08xQm5lNTlUdVJCVFVpNkV1WXM4SEN3Q3lZM2s/12810-22653-26093_thumb.jpg
https://googledrive.com/host/0B37SNNYCVhA_fnRQd1JUQ05SOTJPaXNZS08xQm5lNTlUdVJCVFVpNkV1WXM4SEN3Q3lZM2s/14349-27417-26639_thumb.jpg
https://googledrive.com/host/0B37SNNYCVhA_fnRQd1JUQ05SOTJPaXNZS08xQm5lNTlUdVJCVFVpNkV1WXM4SEN3Q3lZM2s/17822-1683-12809_thumb.jpg
https://googledrive.com/host/0B37SNNYCVhA_fnRQd1JUQ05SOTJPaXNZS08xQm5lNTlUdVJCVFVpNkV1WXM4SEN3Q3lZM2s/21372-19840-12974_thumb.jpg
https://googledrive.com/host/0B37SNNYCVhA_fnRQd1JUQ05SOTJPaXNZS08xQm5lNTlUdVJCVFVpNkV1WXM4SEN3Q3lZM2s/23193-19768-15737_thumb.jpg
https://googledrive.com/host/0B37SNNYCVhA_fnRQd1JUQ05SOTJPaXNZS08xQm5lNTlUdVJCVFVpNkV1WXM4SEN3Q3lZM2s/24102-18218-2554_thumb.jpg
https://googledrive.com/host/0B37SNNYCVhA_fnRQd1JUQ05SOTJPaXNZS08xQm5lNTlUdVJCVFVpNkV1WXM4SEN3Q3lZM2s/2416-24375-13269_thumb.jpg
https://googledrive.com/host/0B37SNNYCVhA_fnRQd1JUQ05SOTJPaXNZS08xQm5lNTlUdVJCVFVpNkV1WXM4SEN3Q3lZM2s/8713-11952-23783_thumb.jpg
https://googledrive.com/host/0B37SNNYCVhA_fnRQd1JUQ05SOTJPaXNZS08xQm5lNTlUdVJCVFVpNkV1WXM4SEN3Q3lZM2s/899-18705-32578_thumb.jpg
If it does (and if the number of lines is still correct), you're ready to move on to the last part.
Step 11: Upload list to pastebin
Go to Pastebin and copy the text of the names.txt file into the text box. On the optional paste settings, it needs to be either public or unlisted, and the expiration period should be at least long enough to finish the remaining steps. Click submit, and you'll be taken to your submitted pastebin (example: http://pastebin.com/UZmNtDkp). Each line on this should be the hosting url for one of the thumbnail images. Check one last time that the pastebin has the right number of lines.
Step 12 A: Feed pastebin raw url to bot (without tags)
In the header, of the pastebin, click the "Raw" link. It will take you to a raw text file, e.g. http://pastebin.com/raw.php?i=UZmNtDkp. Copy this url, and in the main chatroom run the command
#post add [pastebinurl]
Where [pastebinurl] is the raw url you copied. If this works properly, the bot will tell you:
Links processed: 9, valid: 9, invalid: 0. Posted: 0, unposted: 0. New links added: 9.
With 9 replaced by whatever the correct number of images that you tried to add was. So long as you've followed all these steps properly, there should not be any "invalid", "posted", or "queued", and they should all be added. Congratulations! You're done uploading images (for now).
Step 12 B: Feed pastebin raw url to bot (with tags)
If you want to tag those images so they could be specifically requested from the tag pool, add tags at the end of the above command like so:
#post add [pastebinurl] [tag1] | [tag2] | [tagN]
Be sure the tags make sense and describe the images in the batch properly. Remember that people will be using those tags, not search engines, so don't use synonymous and similar words. If an image belongs to an anime or manga, use that anime/manga's name, or use the tag original.
Examples:
#post add [pastebinurl] wallpaper
#post add [pastebinurl] wallpaper | Miyazaki
#post add [pastebinurl] Madoka
#post add [pastebinurl] KonoSuba | Megumin
#post add [pastebinurl] original
#post add [pastebinurl] original | scenery
I'll be the first one to admit this method is fairly complicated. By now, I have no trouble doing it from memory, but for someone who has never done it before it may be intimidating. Some of the steps could be automated better, but I've had no incentive to do so.
For me, when uploading 1000 or more images at a time, the vast majority of the time that isn't already automated is spent in step 4, which can not reasonably be automated. Hence, there's very little actual time to save even though the process is rather complex. It would be great to have a simple method that works for a small number of images (say, less than 20) in a fraction of the time, but I don't have any such methods at this time. I'd encourage others to make simplifications to this though so that it's more accessible for people without putting a great deal of time into all the steps.
