While trawling (not trolling) for more posts that I can include in the next MHC, I found something interesting via Muhlberger’s Early History:

The Soldier in Later Medieval England is a major research project directed by Anne Curry (who was my personal tutor when I was an undergraduate at Reading). They now have a pilot database online (with free access) with details of thousands of soldiers who fought in the Hundred Years War. This should be really useful for anyone interested in medieval military history, not least because the financial records that the data comes from give much more accurate figures for army sizes than the estimates in chronicles.

I’ve just finished clearing out my bookmarks folder and putting my links onto del.icio.us. I don’t know why I haven’t done this before as it’s much better than keeping over 300 links in my Firefox bookmarks. From now on bookmarks are only for the few sites that I access most regularly. Zotero is for pages that need to be dealt with in more detail, with snapshots, notes, and annotations, or which need to be kept together with bibliographies for projects that I’m working on. And everything else goes on del.icio.us. While I was rearranging everything I took the opportunity to add some of the best sites to History Nexus.

Some other cool things:

Operator is a Firefox plugin which detects and displays Microformats. Microformats are a simple way of embedding metadata in web pages using only HTML.

Firebug is another Firefox plugin, a bit like the Web Developer toolbar but much more powerful. It lets you inspect the code of a webpage with expanding and collapsing tags, highlights the current element on the page, displays all CSS styles which apply to an element, debugs Javascript, and even lets you rewrite the code on the fly! I’ve found it very useful for developing Exhibit pages, and it would also make it a lot easier to design or modify WordPress themes.

Yahoo Pipes is a set of tools for data mining and mashups. It’s kind of what I was wishing Google would do in a previous post. It looks complicated, but still easier than programming from scratch, and very powerful. I’ll be trying it out whenever I get time.

My digital edition of Sandall’s History of 1/5th Lincolnshire Regiment now has a new index of people. In my last post I described how names were marked up in the text. This post is about how I linked them together.

I’d already used a Python script to generate a unique id for every <persName> and <rs> tag, and to build a key using components of the name. Next I wrote another Python script to pull all the names out of the XML document and put them into a SQLite database so that I could standardize the key values. The SQLite clients I have are good for running queries but not so good for manually editing data, so I exported the table to CSV and opened it in Excel. That was possibly a mistake: when I came to feed the data back into the XML I ran into some nasty character encoding problems, but it was partly my own fault for letting some unnecessary non-ASCII characters creep in through copying and pasting text from web pages. Apart from that, editing the data in a spreadsheet was easy and convenient.

I decided to standardize the keys in the following form: surname, full forenames (or initials where names not known), service number (Other Ranks only; post-1917 6 digit number preferred as these were unique within regiment), rank (officers only; highest rank known to have been held during the First World War), years of birth and death (if known). Officers commissioned from the ranks can have a rank and number. Here are a few examples:

Sandall, Thomas Edward, Lt-Col (d 1930)
Stuart-Wortley, Edward James Montagu, Maj-Gen (1857-1934)
Pickard, Herbert, 240010 (d 1917)
Leadbeater, Conrad, 240252 Capt
Howard, William, 241683

In the first pass I just wanted to disambiguate names as quickly as possible. In most cases this could be done from internal evidence, particularly the original index. There were some cases where I needed to check online sources, but even so the process only took a few hours and was completed within one day. There were a few names which couldn’t be disambiguated with any certainty so I’ve had to ask the 5th Lincs experts for help.

In the second pass I tried to make the keys as full and detailed as possible by searching online sources for full names, service numbers, and dates of death. This was much more labour intensive and took several days. It isn’t helped by the fact that both Sandall’s book and the online sources contain many errors and inconsistencies, especially with names and service numbers. This led to lots of lateral thinking, deduction, trial and error, and ultimately frustration. There are still some cases where I couldn’t positively identify an individual or couldn’t find any trace of any possible candidates in the available records. Most of these are junior officers who are only mentioned briefly. All this research wasn’t strictly necessary at this stage, but I’m looking ahead to the possibility of using a wiki for biography pages, which will need page names in the above form.

Once the keys were consistent and as detailed as possible I re-imported the CSV into the database and wrote yet another Python script to loop through every name element in the XML, pull out the database record with matching id, and write the new key value from the database to the @key attribute of the XML element. This went smoothly once I’d got rid of all the non-ASCII characters! Then I added some code to the XSLT to generate the index of people by pulling out all the name elements, grouping them by key value, and creating links to each occurrence of that name. This is still looking quite basic, but it works.

The next step is to make each occurrence of a name link back to the index. There’s a complication here because I need the entries in the index to have valid unique ids so that I can link to them, but I also need the full regularized forms of the names to appear in the index list. This places conflicting demands on the @key attribute, because xml:ids can’t contain spaces, brackets, or commas. TEI used to have a @reg attribute for names which could contain a regularized form, but this was removed in P5. The way you’re supposed to do it now is put the regularized form in a <reg> element and surround them with a <choice> element. I could do that and use a modified form for the @key. But I’m also thinking about putting the name data into a JSON file so I can do cool things with Exhibit. That would mean that I wouldn’t have to embed the regularized form of the name in every occurrence – just have an abbreviated form that can be used as a key for name occurrences and as an id for the index entry. It would also allow me to store and use more data than can be accommodated in TEI XML. For example, during the research/disambiguating phase I collected a lot of links to medal cards and CWGC database entries. If these were in a JSON file I could easily link to them from the index page without much extra work. Now I need to work out exactly what I want to do with this and how to structure the data.

I’ve been playing with Google Custom Search and although it’s good it would be much better if it could recognize metadata in microformats, RDF, or any other formats that metadata might be found in. And if it could also scrape data off web pages using regular expressions (sort of like Feed43 but better). And if you could create custom search fields and define how they map to Google Base fields, Freebase fields, metadata tags, and scraped data.

And I want the moon on a stick.

I’ve just been looking into Google Base, which lets you upload structured data in XML format and make it searchable on Google (although so far Base pages don’t seem to show up in the standard web search). The data is described using item-types and attributes, and although Google provides recommended types and attributes you can also make up your own, for just about any purpose you want. This kind of semantic markup gives the potential for much more specific and accurate search results than a normal web search.

Now I’m wondering if this could be a possible solution to a big problem that I’ve been thinking about for a while: pulling together a list of all the British soldiers who served in the First World War and everything that’s known about them. This would be in the region of several million names. Many of the details are already available online in various places but they’re not linked together. The CWGC has a more or less complete database of personnel who meet their criteria of having died as a result of the war, although new names are discovered every so often (their own search engine can only search by name, not by regiment or service number). Surviving service records (only about 30 to 40% survived the Blitz!) are being put online by Ancestry, although it’s subscription only and the indexing and transcription are reputed to be really terrible. The UK National Archives has made the medal index cards available online (I’ve seen several transcription errors in the index but it’s apparently not as bad as Ancestry). This collection contains nearly 5.5 million records and should mention every soldier who qualified for a campaign medal by serving overseas (although there are unsubstantiated rumours on the Great War Forum that some cards were lost in transit). The medal cards also include men who were awarded a Silver War Badge for being discharged as unfit for service, even if they hadn’t served overseas. Officers are more problematic because if they survived the war they had to apply for their campaign medals and there are many known examples of officers who didn’t make a claim and so have no medal card. Commissions and gallantry medals are shown in the London Gazette, which is available online, but its search engine is notoriously difficult to use. Then there are various personal websites of people who are researching their families or a particular unit. And there’s the Great War Forum, which contains a huge number of posts on individual soldiers, often pulling together information from many different sources, from the most well known online databases to obscure local newspapers and family collections.

In theory something like Google Base could help to pull all this stuff together and make it easier to find information on specific people. For example, you could create an item type for soldiers and give it attributes like name, rank, regiment, battalion, service number etc. First of all a lot of thought and consultation would need to go into defining the item and attributes to make it as useful as possible to as many people as possible. This is definitely something to think about for the future.

However, there are some limitations which mean it isn’t going to happen soon. The biggest problem I can see is that you have to manually upload the records to your account, and edit them whenever they change. You can use the API to automate this but I think it would be much better if you could just embed Google Base metadata in a webpage and let Google’s spiders pull it out automatically. Another thing is that there doesn’t seem to be any scope for collaboration. Once you’ve uploaded your data no-one else can edit it. This is quite disappointing because sharing is a big part of Google Docs. In my experience many expert Great War researchers do not have advanced IT skills and so we need things to be as simple as possible, and easy ways of helping less IT literate people by being able to edit their stuff directly. The Your Archives wiki has shown that this can work really well: it doesn’t matter if people haven’t formatted their pages properly or don’t know how to insert a link. As long as you put up some relevant information, someone else can sort it out.

But these are changes that Google could make in the future, so it’s something to watch out for. There must be lots of other ways that historians could use Google Base. It’s already good enough for smaller data sets which have already been compiled by one person, so I might be able to put up some of my English Civil War data.