Red Skies at Night

February 26, 2011

A Linux-Based Photography Workflow (Part 5: Scanning)

Filed under: articles, floss, linux, process, products, scanners, scanning, tools, workflow — Tags: , , , — Eric Jeschke @ 3:38 pm

This is part of a series of posts on Linux-based software tools for a photography workflow. Please read that first if you are coming to this series fresh–it will provide the necessary background information to explain the purpose of this series.

In this part I move on to the topic of scanning. If you are a new digital-based photographer you might not have any need for this information, but if you shoot and scan film, or like me, are old enough to have old stocks of slides and negatives from the pre-digital days, you may want to scan these so that you can integrate them into a digital archive or print workflow. I’m on my second film scanner. The model I’m currently using is the Epson Perfection 700, a flatbed scanner that has some inserts for batch scanning a number of slides or negative strips at once. The resolution is high enough on this scanner that it is more than sufficient for the lenses and technique that I used back in the film days. Here is a picture of the scanner all loaded up with a set of 12 slides.

Epson V700 w/Batch Slide Holder

I’m going to cheat a little bit again and point to some older posts that I wrote about my scanning workflow almost exactly two years ago. The posts are still highly relevant, since I haven’t changed my scanning workflow one iota since then and I constantly refer back to those posts as reminders when I fire up my scanning workflow. Without further ado, part 1, part 2 and addendum.

For those who don’t wish to follow up all that information in one go, here is a basic summary:
In my early days of scanning, I quickly settled on a commercial program called VueScan, by Ed Hamrick. He sells a version of the program for Linux, Mac and Windows. It looks and behaves more or less identically across all the platforms. Although the price has gone up a bit since those days, this is still a very good product at a reasonable price. I looked at the open source alternatives such as xsane, etc., but they just didn’t match up with the feature set and workflow potential of vuescan. As you can see from the posts, there are many, many settings–this is not a program for novice users. It has a steep learning curve, but once you have mastered it the reward is an efficient, powerful and flexible scanning workflow that is almost unrivaled by any other scanning program period. The posts above describe a two step workflow that results in the RAW scans being saved, and then subsequently “developed”. The key thing here is that if you perform the first pass correctly you never have to go through the tedious scanning process again–like camera RAW files, you can reprocess the scanner RAWs as many times as you like from the hard drive. My second pass is usually to process the RAWs into TIFFs (again using vuescan), and after that I can edit them using GIMP or Raw Therapee for further processing, or run a batch operation using ImageMagick to sharpen, possibly downsample and convert to JPEG for the web.

VueScan 1

Excellent product, highly recommended. According to his web site, there is now a decent book describing a workflow using vuescan, and I only wish that had been around when I was learning it. As it was I remember scouring quite a few web pages and a suffered a few false starts before I finally figured out the subtleties of the program and how to make the most of it.  Nothing like having to scan the same batch of slides again and again to encourage you to figure out how to avoid those mistakes.

VueScan 2

March 14, 2009

Scanning, again

Filed under: articles, photos, process, scanners, scanning, workflow — Tags: , , , , , , — Eric Jeschke @ 10:11 pm

Untitled

Well, sorry to go on so long about this, but I keep adjusting my work flow to fit the changing requirements of the images, and I thought I would keep you informed. If you are coming to this fresh, you might have a look at post 1, post 2 and post 3 first.

The earlier slide sets I had been scanning had a lot of water horizon shots, and because tilted water horizons bug me (unless trying for that effect), I felt the need to rotate the scanned images to level, and so I set the “Border (%)” parameter of Vuescan to 5% to give me essentially an extra bit of space for cropping around the fairly good auto crop selection, since I was rotating and cropping manually afterward. The rotations were often minor, and due to slight slide mismount or angled position in the batch slide holder as, or more often than, photographer composition errors.

However with the more recent sets (photo here a typical example from one) I feel no urgent need to rotate. Indeed, I’d be hard pressed to figure out which way to rotate and how much, without a level horizon to guide me. So I began setting the “Border (%)” setting to -1%, in order to crop off the slight black borders that were being included in the automatically selected crop (it’s fairly conservative about your borders). This yields sets in which I don’t have to manually post process every picture afterward, but only the occasional one. Again, just to be clear, this is in the second pass from scanned RAWs to baked TIFFs. I’m still including the 5% extra border in the initial scans to RAW, just to be safe. And having the RAWs on disk means I can rescan the entire set to TIFF without loading a slide, in a fraction of the time. Sweetness.

Again, using the load/save settings feature of Vuescan made it easy to save these settings under the name “raw-scans-to-cropped-tiffs”.

March 11, 2009

On Scanning (Redux)

Filed under: articles, photos, process, scanners, scanning, workflow — Tags: , , , , — Eric Jeschke @ 12:05 am

Untitled

On the scanning front, I posted a few days ago about how I’ve been cranking my large collection of old analog photography through my epson scanner to digitize it. This is merely an update, an addendum, on a couple small refinements to the process that I’ve incorporated. You may want to read those posts if you want any sort of context for what I’m about to relate next, which deals with the arcana of vuescan.

I found that due to the slides not being so perfect in the mounts I was doing an awful lot of minor rotating in post processing. So I began to religiously set the setting for “Border (%)” (in the “Crop” tab) to 5%, which adds an extra amount around the automatically selected crop. As I mentioned before, I’m kind of lazy about post processing, and since the auto crop worked so well I wasn’t going to fret a lot about a few pixels lost here and there on an edge. But if I’m ending up rotating many of them anyway, I’m going to need to crop after the rotation. In that case, no sense losing any pixels whatsoever.  So I started dialing in the 5%, which adds a small black border to all the slides where the mount is being scanned.

If you set that parameter, you are best off also setting the “Buffer (%)” parameter (also in the “Crop” tab) also to 5% to avoid reading all that black into the white balance calculation (this is during the second scan, from RAW scans to TIFFs). I’m using the “White Balance” setting for choosing color, which seems to do something more or less like an Auto Levels in an image editor. So far I’m pretty pleased with the results, although I’m sure with a good deal more trouble I could eke a little more out of the slides. But I’ve still got a lot to go, and I can always reprocess the RAWs again any old day I want to.

I also discovered this lovely feature of loading and saving the settings. So I have two settings files, one for slides to RAWs (DNGs) and the other for second pass post processing, RAWs to TIFFs.  Using the batch scan feature and just scanning to raws means I can crank through a few dozen slides in a typical work day, and lots more on the weekends.   I have two other tools for sharpening and converting to JPEG, and uploading to Flickr.  I’ll describe those in a future post.

March 1, 2009

On Scanning (part II)

Filed under: articles, photos, process, products, reviews, scanners, scanning, tools, workflow — Tags: , , , , , — Eric Jeschke @ 10:10 pm

Butte, Wyoming

Part II of II

In my previous post, I described my current process of scanning slides and negs into RAW DNG files. In this post, I describe the post processing pass to convert the DNGs to more “baked” forms of files: TIFFs and JPEGs.

Once a basic raw scan is available as a DNG, there might be several ways to “develop” it. Currently, I am using VueScan to reprocess the RAW files into 24-bit losslessly-compressed TIFFs. The TIFFs can then be post processed further using an image editor or batch converted to JPEGs, etc. In the future, I might use a different technique, and having the RAWs around means I can “redevelop” them again if I decide later that there is a better way to do it, ideally without having to scan the slides ever again.

I usually postprocess to Adobe RGB color space for TIFFs with a color profile embedded in the file. There are other color spaces possible, but from what I have read it is best not to use too large a color space for your images. Adobe RGB seems like an all around good compromise, and has the benefit of wide support, especially in DNG files. I have a color profile for my laptop display and the results are fairly accurate, to my eye, from VueScan’s post processing.

Here are the settings that I use:

Input:
- Task: Scan to file
- Source: File
- Files:
 (DON'T
FORGET TO SET THIS!!)
- Media: Slide film
- Batch scan: List
- Batch list: 1,2,3,4,5,6,7,8,9,10,11,12
- Multi page: off
- Frame number: 1
- Preview resolution: Auto
- Scan resolution: 3200 dpi
- Rotation: None
- Skew: 0
- Mirror: on (very important to prevent work in postprocessing!)
- Auto scan: None
- Auto save: Scan
- Auto print: None
- Scan from preview: None
- Lock film base color: off
- Default options: off

Crop:
- Crop size: Maximum
- Multi crop: None
- Lock aspect ratio: off
- Border (%): 0
- Buffer (%): 0
- Preview area: Default
- Default options: off

Filter:
- Infrared clean: Light
- Restore colors: off
- Restore fading: off
- Grain reduction: None
- Sharpen: on
- Default options: off

Color:
- Color balance: White balance
- Black point (%): 0
- White point (%): 1
- Curve low: 0.25
- Curve high: 0.75
- Brightness: 1
- Brightness red: 1
- Brightness green: 1
- Brightness blue: 1
- Slide vendor: GENERIC
- Slide brand: COLOR
- Slide type: SLIDE
- Scanner color space: Built-in
- Printer color space: ICC Profile
- Printer ICC profile: path-to-ICC-profile
- Printer ICC description:
- Printer IT8 data: printer.it8
- Film color space: Built-in
- Show IT8 outline: off
- Output color space: Adobe RGB
- Monitor color space: ICC Profile
- Monitor ICC profile: path-to-ICC-profile
- View color: RGB
- Pixel colors: off
- Default options: off

Output:
- Default folder: path-to-scan-folder
- Printed size: 8x12in
- Auto file name: on
- TIFF file: on
- TIFF file name: @.tif
- TIFF size reduction: 1
- TIFF multi page: off
- TIFF file type: 24 bit RGB
- TIFF compression: On
- TIFF DNG format: off
- TIFF profile: on
- JPEG file: off
- PDF file: off
- OCR text file: off
- Index file: off
- Description:
- Copyright: my-name
- Date: desired-date
- Log file: on
- Log file max size (MB): 2
- Default options: off

Prefs:
- Language: English
- Crop units: inch
- Printed units: inch
- External viewer: off
- External editor: off
- Browser: mozilla
- Graph type: Image
- Button 1 action: None
- Button 2 action: None
- Button 3 action: None
- Button 4 action: None
- Auto refresh: on
- Display raw scan: on
- Splash screen: on
- Histogram type: Linear
- Animate crop box: on
- Thick crop box: on
- Add extensions: on
- Substitute date: on
- Warn on overwrite: on
- Warn on not ready: on
- Warn on no scanner: on
- Exit when done: off
- Beep when done: off
- Beep when auto eject: off
- Use temp file name: off
- Anti alias text: on
- Anti alias image: on
- Enable density display: off
- Enable raw from disk: off
- Disable scanners: None
- Enable sliders: on
- Enable spin buttons: on
- Enable popup tips: on
- Enable sample images: on
- Startup tip: 0
- Image memory (MB): 2048
- Window maximized: off
- Window iconized: off
- Window x offset: 96
- Window y offset: 41
- Window x size: 1728
- Window y size: 1080
- Font size (pt): 9
- Option panel width: 340
- Default options: off

Click the “@” sign by the “Files” setting and point VueScan at the first file in the series that you want to postprocess (note: DON’T forget to do this or you will likely process the wrong files–VueScan defaults to the first likely RAW file it finds in the folder, and that may not be the one that you want to start with). I use the “List” setting under “Batch scan” for the post processing pass, because there is less likelihood of reprocessing images you weren’t intending to (VueScan’s interface could be better on this, to be sure). Then set the Batch list to the set of frames to process (usually the entire set, or “1,2,3,4,5,6,7,8,9,10,11,12″).

Under the Crop tab, you may want to adjust the “Buffer (%)” setting to the same figure as you used in “Border (%)” (if you upped that figure from 0 during the scans) to avoid computing too much black into the color settings calculation (auto white balance).

Make sure the “Frame number” setting is set to 1 and hit “Preview”. All the files will be read in and preview images generated. Here again I give a quick look over to make sure that everything looks OK, paying attention now to white balance, exposure, correct cropping, mirroring, etc. This is the time to make adjustments if necessary.

Once satisfied, I hit “Scan”. The post processing is much faster than the scanning of course, since it is only reading in the RAWs from disk, applying the chosen post processing effects, and spitting out the baked files. Now the RAWs are around 100MB each. On my 2 GHz Core 2 Due laptop, it takes around 2 minutes to process all 12 files in a batch. Afterward, there are 12 new TIFF files in the output folder. These I examine with an image viewer and, if necessary, do further post processing in a separate image editor, possibly after refiling the current set of slides and whilst scanning a new batch of 12.

With the slide loading, preview, scanning and the post processing, one batch of 12 takes around an hour to do. My mode of operation is to load a set when I get up first thing in the morning, another set or two when I get home from work, and a final set or two in the late evening. Weekends if I’m around I can process more sets, while puttering around the house or doing other tasks.

Using this process I am slowly working my way through years and years of negatives and slides, and getting great results.

On Scanning

Filed under: articles, photos, process, products, reviews, scanners, scanning, tools, workflow — Tags: , , , , , , — Eric Jeschke @ 2:16 am

On Top of the World

[Look at that guy--he's on top of the world...in more ways than one!]

Part I of II

I’ve finally got some traction again in my quest to scan all my old negatives and slides. I shot analog for some 20 years before my first digital camera, and I have thousands of old negs and slides (mostly slides). It’s not just a matter of convenience to have all these photos in digital versions; it’s a race against time. Where I live, mold is a fact of life. Scanning is my only hope of saving these memories for the future.

I’m scanning them all using an Epson Perfection V700 scanner. Although I find scanning tedious, I really like this scanner. Things I like about it are that it scans at a fairly high resolution, can scan up to 12 35mm slides at a time, has holders for 35mm film strips, slides, medium format film, large format film, as well as being able to scan prints on the platen. It has Digital ICE infrared dust and scratch removal, which works wonders and saves gobs of time in postprocessing. It’s fairly fast, as these scanners go, is reasonably quiet, and produces good looking results (with the right software and settings).

Finding good software to match it has been work, but I’ve settled into using VueScan (version 8.5.04, on both Linux and Mac), by Ed Hamrick. VueScan is a powerful piece of software, but the user interface is a bit arcane and it’s difficult to figure out all the settings. In all fairness, however, I’ve never found scanner software that was easy to use. Documentation is a bit sparse, although it seems to be well used by many photographers, and useful bits of information can be found on various message boards. I’m going to do my part here by describing the process I have been using. I hope it is helpful to someone else.

One of VueScan’s powerful selling points is the ability to scan to RAW files (like a RAW camera capture). My process is to first scan the slides to RAWs, and then reprocess the RAWs to TIFFs in a secondary pass, which is very fast, since it is reading the RAW files from disk. If I get the RAW scans right, then I never have to rescan the slide; VueScan makes it possible to apply all adjustments, including the ICE dust and scratch processing, from the RAWs (since the infrared channel is saved in the RAW file). This is extremely powerful and flexible and all the same arguments that apply to RAW camera images apply to RAW scanner images. Furthermore, VueScan allows you to save your RAWs as DNG files, which means there is a decent chance that your files will not be orphaned in some proprietary RAW format.

I’m scanning at 3200 dpi, 64 bits/pixel (R-G-B + infrared). This gives a RAW file of a little less than 100MB and is generally around 4280×2896 pixels. For 35mm, with the films and lenses I used back then (not to mention my technique), 3200 dpi is plenty of resolution–any more would be simply wasting disk space.

Here then, is my process. Note that this is heavy in detail on both VueScan and the Epson Perfection V700:

LOADING THE HOLDER:
In a batch scan of twelve 35mm slides, the slides are scanned top to bottom, right to left. So I load the slides top to down from the right to the left. Don’t forget when refiling the slides afterwards in the folders, if you use a different scheme, like left to right, top to bottom.

Load slides emulsion side down, and load them with the correct viewing orientation (portrait or landscape); don’t worry about keeping them all in the same orientation. This prevents you from having to do a lot of rotation afterwards in postprocessing.

Slip the holder into the designated spot on the scanner and close the lid.

SETTINGS FOR INITIAL SCANS:
Click the “More” or “Advanced” button at the bottom of the VueScan GUI to enable the advanced options, and then fill in the tabs. Here are my settings for the initial scan in (note that not all of them are relevant to the RAWs).

Input:
- Task: Scan to file
- Source: PerfectionV700
- Mode: Transparency
- Media: Slide film
- Bits per pixel: 64 RGBI
- Batch scan: All (need to set Multi Crop on "Crop" tab for this to show
    up)
- Frame number: 1 (ditto)
- Preview resolution: Auto
- Scan resolution: 3200 dpi
- Rotation: None
- Skew: 0
- Mirror: on
- Auto scan: None
- Auto save: Scan
- Auto print: None
- Number of samples: 1
- Scan from preview: None
- Multi exposure: off
- Lock exposure: off
- Default options: off

Crop:
- Crop size: 35mm slide
- Auto offset: on
- Multi crop: 35mm slide
- Auto rotate: on
- Lock aspect ratio: off
- Border (%): 0
- Buffer (%): 0
- Preview area: Default
- Default options: off

Filter:
- Infrared clean: Light
- Restore colors: off
- Restore fading: off
- Grain reduction: None
- Sharpen: on
- Default options: off

Color:
- Color balance: White balance
- Black point (%): 0
- White point (%): 1
- Curve low: 0.25
- Curve high: 0.75
- Brightness: 1
- Brightness red: 1
- Brightness green: 1
- Brightness blue: 1
- Slide vendor: GENERIC
- Slide brand: COLOR
- Slide type: SLIDE
- Scanner color space: Built-in
- Printer color space: ICC Profile
- Printer ICC profile: path-to-the-ICC-profile
- Printer ICC description:
- Printer IT8 data: printer.it8
- Film color space: Built-in
- Show IT8 outline: off
- Output color space: Adobe RGB
- Monitor color space: ICC Profile
- Monitor ICC profile: path-to-the-ICC-profile
- View color: RGB
- Pixel colors: off
- Default options: off

Output:
- Default folder: path-to-scan-folder
- Printed size: 8x12in
- Auto file name: on
- TIFF file: off
- JPEG file: off
- PDF file: off
- OCR text file: off
- Index file: off
- Raw file: on
- Raw file name: scan0001+.dng
- Raw size reduction: 1
- Raw file type: 64 bit RGBI
- Raw output with: Scan
- Raw DNG format: on
- Description:
- Copyright:
- Date:
- Log file: on
- Log file max size (MB): 2
- Default options: off

Prefs:
- Language: English
- Crop units: inch
- Printed units: inch
- External viewer: off (sometimes on)
- Viewer: gimp (only if above setting on)
- External editor: off
- Browser: mozilla
- Graph type: Image
- Button 1 action: None
- Button 2 action: None
- Button 3 action: None
- Button 4 action: None
- Auto refresh: on
- Display raw scan: on
- Splash screen: on
- Histogram type: Linear
- Animate crop box: on
- Thick crop box: on
- Add extensions: on
- Substitute date: on
- Warn on overwrite: on
- Warn on not ready: on
- Warn on no scanner: on
- Exit when done: off
- Beep when done: off
- Beep when auto eject: off
- Use temp file name: off
- Anti alias text: on
- Anti alias image: on
- Enable density display: off
- Enable raw from disk: off
- Disable scanners: None
- Enable sliders: on
- Enable spin buttons: on
- Enable popup tips: on
- Enable sample images: on
- Startup tip: 0
- Image memory (MB): 2048
- Window maximized: off
- Window iconized: off
- Window x offset: 96
- Window y offset: 41
- Window x size: 1728
- Window y size: 1080
- Font size (pt): 9
- Option panel width: 340
- Default options: off

DOING THE PREVIEW SCAN:
Position the Frame number setting to 1 and press “Preview”. This will generate a preview scan with thumbnails. I give this a quick once over to see if there is anything seriously off, and to verify that the automatic crop detection is working well. Some people would recommend to not choose the automatic crop, but to set for “maximum crop”. Not me. I have thousands of slides to scan. I’m lazy when it comes to scanning. The more things I can get right from the first scan, the better. So far I find the automatic cropping to work very well. Every few sets of images there will be one that it had trouble with (usually something with lots and lots of white or black), and I’ll just rescan that one slide with maximum crop and then crop it manually in postprocessing. You may want to play around with the “Border (%)” option to add some extra space around the automatic crop, if you go that route and are having trouble with too tight cropping.

DOING THE SCAN:
If the preview looks good I hit “Scan” and go off to do something else while the scanner does its thing. With these settings it usually takes around 45 minutes to do 12 slides (1 batch). After the designated period I’ll come back to find 12 new DNG files sitting in the output folder.

In Part II I’ll describe the postprocessing workflow for generating the TIFFs (and/or JPEGs) from the RAW files.

October 20, 2007

Getting a Epson Perfection V700 Photo working under Ubuntu Feisty

Filed under: linux, scanners — Eric Jeschke @ 11:11 pm

As usual it was fairly easy in the end, and anything from obvious. I followed the directions here and was up and running in no time. (google + ubuntu forums == goodness)

Summary:

  1. With the scanner on and plugged in to the computer, run sane-find-scanner (as root) to see if the scanner is detected (I was using the USB interface)–it was: 
    $ sane-find-scanner 

  # sane-find-scanner will now attempt to detect your scanner. If the
  # result is different from what you expected, first make sure your
  # scanner is powered up and properly connected to your computer.

  # No SCSI scanners found. If you expected something different, make sure that
  # you have loaded a kernel SCSI driver for your SCSI adapter.

found USB scanner (vendor=0x04b8 [EPSON], product=0x012c [EPSON Scanner]) at libusb:005:036
found USB scanner (vendor=0x0483 [STMicroelectronics], product=0x2016 [Biometric Coprocessor]) at libusb:003:029
  # Your USB scanner was (probably) detected. It may or may not be supported by
  # SANE. Try scanimage -L and read the backend's manpage.

  # Not checking for parallel port scanners.

  # Most Scanners connected to the parallel port or other proprietary ports
  # can't be detected by this program.

  # You may want to run this program as root to find all devices. Once you
  # found the scanner devices, be sure to adjust access permissions as
  # necessary.
  2. I edited /etc/sane/epson.conf and replaced ‘usb’ with ‘usb 0x04b8 0x012c’ (from the output above).
  3. Once you do this, scanimage should be able to locate it, to wit: 
    $ scanimage -L
device `epson:libusb:005:036' is a Epson GT-X900 flatbed scanner

    It’s not identified correctly, but that turned out to be a non-issue.

  4. After this I could run ‘vuescan’ as root, but not joe user. The somewhat unsatisfying fix is as follows: 
          sudo gedit /etc/udev/rules.d/40-permissions.rules

    Scroll down the file until you find an entry that says:-

    # USB devices (usbfs replacement)
SUBSYSTEM=="usb_device", MODE="0664"

    Change the MODE=”0664″ to MODE=”0666″ and save the file, then reboot.

  5. Bob’s your uncle!

Theme: Shocking Blue Green. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.