Back in July I posted about a Python script I was working on to help with organizing photos of archival documents. I didn’t think it would all that interesting to many other people, but a comment from Chris Williams made me realize that there’s potentially quite a lot of demand for something like this. Digital photography in archives doesn’t seem to be much of a sexy buzz topic among digital historians, but it’s something that lots of researchers do even if they’re not into digital history (although Melissa Terras‘s latest book seems to cover it). As far as I know there aren’t any tools specifically designed to help with organizing large numbers of document images. The python script I’m working on is just a stopgap thing which is mostly specific to what I’m doing and how I work, and is never likely to be very user friendly. Maybe what we need is a Firefox extension that plugs into Zotero, or maybe image management features in Zotero itself. Some features that might be useful:

• Browse a directory of images in Firefox (I used to use MozImage for this, as I was reminded when I found this old post)
• Mark a page image as being the first or last in a document (this is the really crucial thing, and I’m not aware of any image browsers that can currently do it)
• Create sub-directories for documents and move images into them based on first and last markers
• Create Zotero items for marked documents, maybe with some fields pre-filled in a standard form which can be applied to all documents in a directory. For example, if I’m working through box SP 24/30 from the National Archives, set Repository to “TNA” and Loc in Archive to “SP 24/30”.
• Upload images to Flickr and create sets for them, maybe based on associated Zotero items; attach Flickr links to relevant Zotero items

I’m not in a position to do this myself right now, but I need to learn how to make Firefox extensions sooner or later. Apart from image management stuff, I also need a word count extension (I usually draft most of my writing in a private wiki instead of a word processor; having Firefox count the words for me is much easier than pasting into Open Office just to see how much I’ve written). The one I used to use isn’t compatible with Firefox 3.6 and the author hasn’t updated it for a long time. Counting words can’t be that hard can it? Or maybe it is.

So, does anyone have any thoughts on image management? If you take lots of photos in the archives, how do you deal with them once you get them home? Is there any software I don’t know about which would do what I need? What features would make your life easier?

8th March is International Women’s Day, and March is National Women’s History Month in the US (it would be nice if there were more of them in a year, otherwise we have 11 men’s history months and 364 men’s days by default, but you can’t have everything, especially if you’re a woman…). The theme of this year’s Women’s History Month is Writing Women Back Into History. I’ll probably write some posts about women’s and gender history later in the month. But right now seems like a good time to announce a new website/blog:

They Really Do Exist “aims to be a directory of women who are active in traditionally male arenas”, “for all those people who are sick of hearing ‘But there ARE no women in that sector!’ when they ask why the media or other publicity downplays the role played by women in any given area”. The site was the idea of Jennie Rigg, a female political blogger who is, in her own very apt words, “FUCKING SICK” of being told that there are no female political bloggers. What I find most striking about this situation is that many male political bloggers (even liberal ones) try to delegitimize feminism by claiming that it isn’t really politics. In contrast, anti-feminist academics are more likely to delegitimize feminist history by asserting that it is political and therefore doesn’t meet their standards of (false) neutrality. This double standard gives patriarchy the best of both worlds and makes things even more difficult for feminists. Maybe part of the problem of patriarchal equilibrium is that feminists are intellectually honest and abhor hypocrisy, whereas patriarchy thrives on it. Anyway, if you know of a woman who should be included in the list, leave a comment at the submit page.

Inspired by Jennie’s example, I’ve expanded the scope of the War and Gender Zotero group to include works on any aspect of military history written by women. There are now two sub-collections in the group library: one called “About Gender” which includes any works about the intersections of gender and sexuality with war written by anyone (which is what the group was originally limited to), and one called “By Women” which includes anything relating to wars and armed forces written by women. The new collection is still in its early stages. So far it only contains works by women that were already in the group library. There are lots more items in my personal library which need to be added. I’ve almost certainly made some embarrassingly wrong assumptions about people’s gender based only on their forenames, despite being trained by the Cambridge Population Group not to do that. [ETA 13 May 2011: I stopped doing this because it was too much trouble!]

And finally, here’s a photo of a nurse and some “munitions girls” from the First World War:

Last year computer programming was out, but now it’s back in. For me anyway. Having finished my data entry job in October I’ve got more spare computer time, which means I can be more active in digital history again. Some things are different now. Zotero has groups and syncing. The Programming Historian has moved since the last time I looked at it. I can finish my digital edition of Sandall’s history of the 5th Lincolns because Major Teall’s epilogue came out of copyright in the UK at the start of this year. But the biggest change is that I’ve switched my operating system from Windows to Linux. When I built my new desktop PC (codenamed Zen) I installed Ubuntu, and I love it. My laptop (codenamed Orac) still has Windows Vista, but I don’t use it much.

Changing to a completely different operating system might sound like a big step but it was actually really easy. This is partly because most of the applications I use are cross platform. I use Firefox more than any other application (and possibly more than all other applications put together). Don’t think that I spend all my time idly browsing the web: Firefox is vital for my historical research and writing. I use Zotero to store, sort, and access all the bibliographic data plus associated notes and PDFs for my research projects. These can all be synced between my PCs via the Zotero server and my own WebDAV server. My works in progress are now drafted on a private wiki which is also necessarily accessed through my web browser. This is much more powerful and flexible than writing in Word like I used to. Every page has an edit history so I can easily compare versions and revert to an earlier one. Wikilinks make it easy to fit sections together in different orders and link to supplementary information. Thanks to Google my e-mail and RSS feed reader are also on the web. When I’m not using Firefox, I still mostly use cross-platform applications. For the last few years I’ve used oXygen for XML editing and jEdit for find and replace operations, both of which are written in Java. Python can run on Linux, Windows and Macs, and although that doesn’t necessarily make individual scripts cross-platform it doesn’t really matter when I’m writing them for myself. The only Windows specific app that I’ve relied on in the last few years is MS Access. Even that was mainly because I was getting paid good money to put data into it for someone else. For my own research I’ve got some old databases from my PhD research, but all I ever need to do with them is export data into other formats.

Given all this, changing to Linux was not likely to be much of a problem, but that would be understating things. In fact it turned out to be a big advantage. Ubuntu is actually much quicker and easier to install and set up than Windows. It just works out of the box and comes with most of the things that most people need to get started. Open Office, Firefox, and even Python are all pre-installed. Once I’d added my favourite Firefox extensions and synced my Zotero library I was ready to do most of what I need to do. The only tricky things were manually installing a proprietary graphics driver and setting up DVD playback, but even this wasn’t too hard. If you don’t have a powerful new graphics card and don’t need 3D performance out of it, the pre-installed open source driver will be adequate for desktop stuff. Even setting up a network printer was completely painless.

Adding new applications is generally much easier than on Windows. Instead of buying a CD or downloading an executable file you can just access software repositories via a menu and tick boxes to select apps you want to be downloaded and installed. Because most of these apps are free in every sense of the word (like Ubuntu itself) you won’t have to pay money or agree to a licence that sells your soul to the devil. Via the repositories I could easily install Geany (a code editor which I now use for Python programming: I actually like it more than Komodo), gFTP (FTP client), the aforementioned jEdit, and the BeautifulSoup library for Python. It only took a few simple commands at the terminal to install and set up an Apache server with PHP and MySQL for local testing. oXygen had to be downloaded and installed manually as it’s a proprietary application, but the academic licence is cheap and cross-platform: I originally bought it for Windows but my licence automatically carries over to Linux. To get it working properly I had to install the proprietary Sun version of Java, but that was easy to do via the repository. There is a thing called WINE which lets you run some Windows programs in Linux, but so far I’ve only used it for listening to music with Spotify.

With everything set up to my liking, Ubuntu has made me fitter, happier and more productive. It’s faster, more secure, more stable, and less annoying than Windows. You can start using it as soon as the desktop appears on the screen instead of waiting for it to finish starting, or dealing with a patronising storm of pop-ups about how your anti-virus might be out of date or how you’ve got unused icons on your desktop. The Blue Screen of Death is now just an unpleasant memory. Linux users generally don’t have virus scanners or software firewalls because we don’t need them. The only major problem I’ve had so far is when an upgrade to a new version didn’t agree with my proprietary graphics driver and made it impossible to boot to the desktop from the hard disk. Even that was surprisingly easy to recover from, as being able to run the operating system from the LiveCD makes it very easy to rescue any files which aren’t already backed up before doing a clean reinstall (and the reinstall process is quicker and easier than for Windows).

So those are my reasons for preferring Ubuntu to Windows. If you haven’t tried Linux before you can download Ubuntu, burn it onto a CD, and then boot from the CD, which gives you an option to try it out without actually installing it on your PC. And it won’t cost you anything. Meanwhile I’ll be getting on with my research, writing and programming. And blogging about those things…

It didn’t take long before I decided to start a Zotero group. It’s called War and Gender and is dedicated to collecting and sharing any material relating to the intersections of these two very important things. There are no limits on period or place, membership is open, and all members can add to the group library. So if you’re interested, and if you’re using Zotero 2.0, get stuck in.

You can see from my profile that the groups I’m involved with so far are all quite specific and tend towards things that relatively little has been written about yet. I think we’re all still finding our way and sticking to things that are likely to be manageable. In the future it’ll be interesting to see if more general groups appear and whether they work out. I could start a British Civil Wars group, but it would be potentially huge. I already have over 900 items in my library tagged with “english civil war”, and these are mostly biased towards my research interests in England 1642-46. I don’t have very much on Scotland, Ireland, the Second Civil War, the Commonwealth or the Protectorate. Maybe more specialist sub-fields will be the way to go, but we’ll see eventually.

Ian MacInnes has set up a Zotero group for The horse in history and culture where we can pool references to horses. This was the incentive I needed to finally sort out my collections and upgrade to Zotero 2.0. I’ve set up my profile on the Zotero website, added a CV and shared my library so anyone can browse it. Although it looks like I’ve got an awful lot of collections, I’ve simplified the hierarchy and started making better use of tags. I’m not sharing notes at the moment, but maybe I will later. (NB: if you uncheck the “share notes” box in the privacy settings it only hides notes that are attached to items, not standalone notes.) There’s also another group for early-modern animal studies where you can find stuff about other species as well as horses. Now I’m wondering what other groups would be useful, but I’m not sure if I want the responsibility of owning a group yet.

A while ago I was wondering whether there were any good social networking sites for academics, and whether I really needed one. Now it looks like Zotero is going to fill the gap. Zotero 2.0 is now in beta, and Dan Cohen has given a rundown of the new features. Users will get a personal page on the Zotero site allowing us to network and interact with people with similar research interests, and users will be able to create and join groups to make sharing of bibliographic data easier. This all looks really exciting. I’ll probably try it once I’ve got my Zotero collections and tags in better order.

The new version of Zotero makes it very easy to insert Zotero-friendly bibliographies into blogs. You can now drag and drop items straight from Zotero and it will automatically create a nicely formatted HTML bibliography with COinS metadata so that other people can grab your citations into their Zotero library (you need to change some preferences but it’s easy).

So I’m going through my old posts and adding bibliographies where applicable. I’ve got up to February 2007, but the rest shouldn’t take too long. Take a look at the post archives for 2006 to see it in action.

The Great War Forum now has a sub-forum called Using The Technology devoted to discussions of how to apply IT to First World War research. This could be a good place for digital historians to reach out to amateur researchers and show them new tools and techniques. It should also be a valuable source of ideas about how to use computers for research, and examples of how technology can make a difference in practice. I’ve already been telling everyone there about how cool Zotero is.

Since my last post I’ve been doing some more experiments to see how Zotero can be used for cataloguing previously uncatalogued administrative records from the English Civil War. I’ve now put some more of my ideas into practice in demo form and they seem to work. Linking images to Zotero items and adding metadata went very smoothly. The idea of adding extra data by putting XML tags in notes also works, although this is just a stopgap until they implement custom fields. Once you have data in Zotero it’s very easy to export it as XML and do whatever you want with it. More details below, but it gets a bit technical and even includes some sample code (formatting code in WordPress is hard, and it’ll probably screw up the layout for some people). If you’re not A. Nerd and you’re not doing the shopping for your mum you might want to stop reading now.

First of all creating the items. I had a sample of 50 images of pay warrants from SP28/1A to use for this test run. This wasn’t quite a fair test as I knew in advance that they were all exactly the same kind of document: warrants authorized by the Earl of Essex in August 1642. In practice you don’t always get unbroken runs as there are sometimes other kinds of documents mixed in with the main series of warrants. So I pointed MozImage at the directory where the images were and went through each one, creating a new item, linking to the image file, and adding the metadata. The template hack that I mentioned in the last post worked really well here, saving a lot of time and effort. I realized that I was barking up the wrong tree with my hope that I could keep multiple templates in the same RDF file and select one by clicking on the folder. It makes much more sense to keep one template in each RDF file and attach a snapshot of it to the item that you created it from. That way you can see exactly which fields are prefilled. Otherwise if you have a lot of these templates it would be difficult to remember exactly what each one did. In this case I prefilled the author, repository, and location fields, and also set up notes with the XML tags that I used as pseudo-custom fields. These are to record the amounts of money due and paid, the date of payment (often different from the date that the warrant was authorized), and the names of the people who were paid.

It took about two and a half hours to create items, link images, and fill in all the data for these 50 documents. That’s an average of about three minutes per document, which isn’t too bad, and might improve with practice and/or more timesaving shortcuts (if I can think of any). It only took 10 minutes to photograph these 50 documents, which is faster than my more recent experiment with an account book (see here), I think partly because I didn’t need to use weights for this volume. So in a total of 3 minutes 10 seconds I have a digital image of the document plus the most important data in machine readable form. That compares quite favourably with taking notes by hand and then copying them onto file cards, which is how I dealt with these same documents 10 years ago.

Once the data was entered I exported the collection to a MODS XML file. When Zotero exports notes to XML any angle brackets in the note are converted to entity references, but it takes hardly any time to run a Find and Replace (I use jEdit for this) to turn them back. This is a sample record from the file, including the custom XML fields in the notes:

<mods>
<titleInfo>
<title>Warrant</title>
</titleInfo>
<typeOfResource>text</typeOfResource>
<genre authority=”local”>manuscript</genre>
<genre>Warrant</genre>
<name type=”personal”>
<namePart type=”family”>Essex</namePart>
<namePart type=”given”>Earl of</namePart>
<role>
<roleTerm type=”code” authority=”marcrelator”>aut</roleTerm>
</role>
</name>
<part>
<extent unit=”pages”>
<start>48</start>
<end>48</end>
</extent>
</part>
<originInfo>
<dateCreated>1642-08-11</dateCreated>
</originInfo>
<location>
<physicalLocation>SP 28/1A</physicalLocation>
</location>
<note type=”content”><amountdue>
<pounds>69</pounds>
<shillings>0</shillings>
<pence>0</pence>
</amountdue>
</note>
<note type=”content”><amountrecd>
<pounds>69</pounds>
<shillings>0</shillings>
<pence>0</pence>
</amountrecd>
</note>
<note type=”content”><datepaid>1642-08-15</datepaid></note>
<note type=”content”><payto standard=”">
<titlebefore>Sir</titlebefore>
<firstname>Philip</firstname>
<surname>Stapleton</surname>
<titleafter></titleafter>
</payto></note>
<note type=”content”><recdby standard=”">
<titlebefore></titlebefore>
<firstname>Edward</firstname>
<surname>Constable</surname>
<titleafter></titleafter>
</recdby></note>
</mods>

You could easily do almost anything with this data now (except putting it back into Zotero and synchronizing it with the original records might be problematic, but all I want to do is analyze the data outside Zotero). I just tried writing a Python script to pull some of the fields into a SQLite database, which isn’t very exciting, but everybody’s got to start somewhere. This is the first time I’ve tried writing Python code which actually does anything. If you’re dealing with SQLite there’s another useful Firefox plugin which can help you: SQLite Manager allows you to create, edit, and view SQLite databases.

XML DOM makes it very easy to deal with XML in Python. You just feed an XML file into the parser, and then you can access any of the elements using some simple functions. I couldn’t find a really quick and easy way to get at the text content of XML tags (I think there are some proprietary Javascript extensions which do this better) so I wrote this function:

from xml.dom import minidom #imports xml library

def xmltext (myelement): #returns list of text from xml element
    mychildren = myelement.childNodes #makes list of child nodes
    mytext = [] #list to store text data in
    for x in mychildren: #loop through list of child nodes
        if x.data: #if node contains text data, append data to list
            mytext.append(x.data)
    return mytext #return the list containing data of any text nodes

It takes an XML element object and returns a list of any text data in its child nodes. There are probably better ways to do this but I haven’t found them yet. It works well enough for what I want to do.

Then I wrote this script to pull the location and page number fields out of the XML and insert them into a database:

#import xml library
from xml.dom import minidom
#imports sqlite library
import sqlite3
#imports custom function to get text from xml element
from xmltext import xmltext

#parse xml document into object
xmldoc = minidom.parse('sp281a.xml')
#pull out all records and puts into list
records = xmldoc.getElementsByTagName('mods') 

#opens database connection and create cursor
conn = sqlite3.connect('wrtest.sqlite', isolation_level=None)
c = conn.cursor()

#loop through every record
for x in records:
   #get child location elements into list
    location = x.getElementsByTagName('physicalLocation')
   #call function to get text
    loc_text = xmltext(location[0])
    #repeat for the other field
    folio = x.getElementsByTagName('start')
    folio_text = xmltext(folio[0])

    mydata = loc_text[0], folio_text[0]
    #run SQL to insert data into table
    c.execute("insert into warrants values (?,?)", mydata) 

conn.close()

And that’s it for now. Maybe not very exciting in itself, but there’s a lot of potential. There are lots more things that I need to do, including writing some code to add up pre-decimal currency.

In a previous post I mentioned experimenting with taking photos at the Public Records Office/National Archives. Getting good photos is only part of the problem. The real work starts when you get them home. How do you organise them and make sense of them? It should be no surprise that Zotero is really useful for this, but I’ve discovered a few tricks to make it even better.

One of the main challenges of the Commonwealth Exhequer Papers (SP 28), is that although the class contains a huge amount of useful information about the British Civil Wars it isn’t very well sorted or catalogued. It includes lots of different kinds of documents created by different organisations at different times for different purposes. These documents range from warrants and bills on a single sheet of paper to account books with over 100 pages. Having photos of every page is a huge advantage over having barely legible handwritten notes. It’s almost like having the original document at home on your desk, but using Zotero we can make it better than the original document without losing anything. I can create a Zotero item corresponding to the document (whether it’s a pay warrant, an account book, or anything else) and attach the page images to it. Then I can add all kinds of metadata using the item fields, notes, and tags, and rearrange the items however I want, while still having instant access to images of the original, and still being able to get them back into their original order. That’s the theory. Today I’ve been trying it to see how it might work in practice, using some test shots of pay warrants that I took last year.

First I had to work out what kind of document I was looking at, then create a new item for it, then attach the correct images. I’ve found that the best way to do this is to use another Firefox plugin called MozImage. Sadly it isn’t a picture of Morrissey, but it’s almost as good: it lets you browse image files on your local hard drive through Firefox. [Edit: I've just noticed that it can also browse directories on a web server, which could be useful for collaboration] You can easily click forwards or backwards through a folder of photos taken in the archives to see how many pages there are in a document. Viewing it full size in the browser window, with the option to zoom in and out, makes it easy to identify what kind of document it is. Once you’ve created the item you can attach a snapshot or link to each image just as you would with a web page.

It would be nice if Zotero had some similar way of letting you scroll through item attachments. Being able to click forward and back buttons to view the next/previous attached file would be easier than having to click on each one.

Creating an item for each document can get quite tedious and repetitive, especially as there might be up to 1,000 pay warrants in each box of SP 28, each one of which needs to be catalogued as a separate document. Fortunately I thought of a quick and easy hack which lets you create an item from a template without having to do any actual programming:

1. Create a new item in Zotero and fill in the bits that you need to apply to every document of that type. For example I might set the Author as “Earl of Essex”, Title as “Pay Warrant”, Repository as “TNA: PRO”, and Loc In Archive as “SP28″.
2. Select the item and export it as Zotero RDF.
3. Open the RDF file in Firefox.
4. The magic icon should appear in the address bar. If you click on it you get a new item with the fields pre-filled with the data you set up.
5. You can also take a snapshot of the RDF file in Zotero so that you can easily find it and open it whenever you need it.

It would be even better if I could export several items to the same RDF file and choose one from the drop down menu on the folder icon, but it doesn’t seem to work like that – I just get a single icon and clicking it imports every item in the file. [Edit: selecting from multiple items isn't really necessary as I've thought of a better way to do it now]

The basic item fields are quite useful for adding metadata and sorting documents – especially author, date, repository, and location – but there’s a lot of data I need which won’t fit. This is where tags come in. I can use these to sort documents by type (pay warrants, quartering bills, accounts etc), or by content. For example I could pull out all warrants relating to cavalry troops, all warrants for purchase of horses etc. The number of officers and units is finite, so it might be feasible to have a tag for each unit or its commander (although I’d have to standardize the spelling for this to work properly).

There’s some data which tags can’t deal with. For example I need to extract and add up amounts of money. Once custom fields have been implemented (and this is a planned feature of Zotero, although I don’t know when it’s coming) this should be easy. Until then I’m wondering if I could use notes with the data marked up by XML tags. Using the template trick it would be easy to create an item with a note with the XML tags already in it. Then I could probably write a Python script to pull out the data and do something with it.

I’m also thinking about how to export data to the Your Archives wiki. It should be possible to use a script, or maybe just a citation style, to generate document references and summaries in wiki format. But this is all speculation. I’ll have to do some more thinking and experimenting. For now I’m just really pleased that Zotero exists and is free.