In my last project update I described how I used FineReader to OCR the text of Sandall’s History of 5th Lincolnshire Regiment. Since then I’ve manually proofread the text and inserted some basic XML markup. Proofing and basic tagging have given me a more detailed understanding of the text and the features in it, and I’ve been noting potential issues as I go. I’ll post more about how I’m using XML later, but this post is a more detailed description of the process of proofreading.

I used FineReader for proofreading, as its interface makes it easy to compare digitized text with the original image. I had previously stepped through the uncertain characters and unrecognised words, so wasn’t expecting major problems, but I still wanted to iron out any remaining scannos and impose consistent style before I exported the text. In the post on OCR I said I was exporting copies of the text at each stage for comparison purposes. I’ve now decided to abandon this because I was trying to do two contradictory things at once: create a useable text (the main aim of the project) and produce a perfect rendition of the sequence of characters in the original text in order to assess the accuracy of FineReader’s OCR. I now realise that it isn’t possible or desirable to do both at once. For example, where the text uses I for 1 the two objectives can’t be reconciled. If FineReader renders it as ‘I’ that’s a correct rendition but in order to create a useable text I need to change it to ’1′. If I wanted to quantify the accuracy of the OCR I’d have to do it separately from creating a useable text, but I don’t think the effort would be justified. I’ve already got enough of a feel for FineReader to know that it’s good enough for what I want to do but that some manual proofing will always be necessary to create a releasable text.

I decided not to carefully compare the text and image line by line as this would take too long, and I wasn’t expecting enough scannos to make it worthwhile. Instead I just read through the digitized text — slowly and carefully but in Distributed Proofreaders terms it was more like smooth reading than proofing. Since I had never read the whole book before it was unfamiliar enough to make this a viable approach. However, I made a point of double checking every number and every proper noun which I wasn’t familiar with. This slowed down the reading, and rapidly switching my attention from the text in the lower pane to the image in the upper pane and back again gave me eye strain! If I was scaling the project up, this is the kind of work which I could palm off onto someone else get volunteers to help with as it doesn’t require a great deal of technical skill (apart from knowledge of spelling and punctuation) and can be split into small chunks, as at PGDP. I also checked every end of line hyphen to make sure it was correctly marked as soft or hard. FineReader had already got a lot of these right, but there were some cases where it was obviously wrong, or which were ambiguous. These ambiguities could only be resolved by looking for occurrences of the same word elsewhere in the text. As I went I built up a hyphenation list, showing whether certain words should have a hard hyphen or not. For example headquarters didn’t but out-buildings did. I was aiming for consistent usage according to the original text, regardless of what might now be considered “correct”.

Before I started reading I spent about 15 minutes manually removing running heads from the tops of the pages. I could have done it slightly quicker using Find and Replace, but that wouldn’t have saved a great deal of time because the book has over 20 chapters which are all quite short. In future I’d recommend adjusting the text capture boxes to cut out all extraneous matter. This includes junk from around the edges of the pages as well as headers and footers.

It took about 8 hours to read through the whole text, including front and back matter. I found very few scannos. The most usual one was where the original text rendered ’11th’ as ‘IIth’ FineReader thought it was ‘nth’. Many of these had been flagged as uncertain but a few had slipped through. There were a few obviously wrong letters but generally everything made sense in context. Some stealth scannos could have slipped through, but I’m not prepared to put in the effort to find them. The perfect is the enemy of the good. Double checking names and numbers which were not self-evidently wrong turned out to be more trouble than it was worth as errors here were negligible. Most of the changes I made during proofing were actually formatting. As well as checking hyphens I had to make sure that abbreviations conformed to my style guidelines. In future I must make sure that FineReader isn’t automatically inserting spaces after full points as I spent a lot of time correcting this, especially with all the corps and rank abbreviations in a book like this. Obvious typographic errors in the original text were marked with [sic], but I didn’t find many.

Once proofing was finished I exported each batch to a utf-8 text file. Although FineReader marks soft hyphens as ¬ to distinguish them from hard hyphens, it converts them all to – when exporting them to text, which is slightly annoying. Because of this I ran Find and Replace over each batch to replace every ¬ with the entity reference &amp;shy; For my purposes I could leave it like this, but if I was creating individual page files for other people to proof through the DP interface I’d want to convert them back to ¬ after the export, which is easily done. FineReader has an option to insert a page break character at the end of every page when exporting a batch to a single file. This would be more useful if you could choose the character to insert, but as it is the extra step needed to replace the page break character with the TEI <pb/> element isn’t too much trouble.

With each batch exported to a single file I just had to tidy them up a bit and then I could start marking them up with XML tags. More about that in the next post.

Now that I’ve got all the theoretical agonising out of the way, I can actually do something about digitizing the text. This week I’m carrying out OCR and proofreading on the text of Sandall’s History of 5th Battalion the Lincolnshire Regiment. As soon as I got to work I encountered issues that I hadn’t thought of, and found that subjective decisions had to be made even earlier than I’d anticipated. This just shows that the only way to learn how to do something is to do it.

Yesterday I started by preparing the image files for OCR. I found it convenient to scan two facing pages at once, but for OCR and proofing it’s more convenient to have a single page per image, and to remove all the plates. This isn’t too difficult to achieve with batch processing. I started with Microsoft Office Picture Manager. I used it to work out where to crop the images, but couldn’t use it to do the cropping because its batch processing features seem to break if you select too many images! No problem, because Irfanview is better anyway. I discovered that a few of the page images towards the end of the book had come out slightly larger than the rest, which meant they had to be done in a separate batch. I also had to run separate batches for rectos and versos as I couldn’t find a way to split one file into two.

Once I had a separate image for each page, I rearranged them, deleted the plates, and used Irfanview to do a batch rename, so that the file name of each image matches the page number. First attempt had to be redone because I forgot to delete the blank pages on the reverse of the plates resulting in the page numbers getting out of sequence! Prefaratory material was kept in a separate directory as it’s numbered with roman numerals and outside the main sequence of page numbers. I also separated Teall’s epilogue (which I still don’t know whether I’m going to publish) and the end matter (medal lists and index which are formatted differently from the main text). Preparing the images took about an hour, but would be quicker if I did it again with the benefit of this experience.

With all the images ready, I installed the 15 day trial version of ABBYY FineReader Pro 8.0. My first impression is that it’s pretty good, and if I do any more similar projects I’ll probably buy a license for this rather than trying to find a free alternative. The Distributed Proofreaders site suggests that you usually get what you pay for with OCR software. I spent some time finding my way around the software, reading the help and tutorials, and trying a few test pages. Once I was ready to go, I created new a new batch for each of the four sections: prefatory (6 pages), main (196 pages), teall (10 pages), and end (17 pages). The main batch of 196 pages took about 3 minutes to import the images, and 20 minutes to be read by the OCR. Mostly the structure of the text was detected and converted into boxes fairly accurately. A few pages had more complicated layout (this was more true in the appendix with table of medals awarded) where I had to redraw the boxes manually. Once OCR reading was complete I saved all the pages as text files for future comparison with corrected text, so that I can see roughly how accurate uncorrected OCR is.

The next step was an automated spell check. FineReader flags all uncertain characters and non-dictionary words, then steps through them and asks for user input. I found the spellcheck window too small, but it doesn’t have to be used as you can position it so that you can see the text in the main panes. There are three main panes: an image of the whole page, showing boxes; a close-up of part of the page image; and the digitized text. These can all be resized and zoomed to suit the user. As I stepped through the checks I was able to spot various problems with the OCR and had to make decisions about how to correct things. I wrote style guidelines as I went, adding each new feature or issue as I encountered it. I already had a good idea of how to handle things &mdash; ideally keep the sequence of characters the same as in the book &mdash; but no plan survives contact with the enemy. Even at the stage of trying to match a sequence of characters to a finite set of characters there are subjective decisions to be made. This is a summary of the guidelines I came up with:

Delete running heads and page numbers. This is how PGDP does it. I don’t necessarily agree with everything they do, but in this case I don’t see the headers and footers being any use to the target audience and they are more likely to get in the way. Pages will be marked with the TEI pb element anyway.

Anyone who wants to see the original layout can look at the images. So far I haven’t been deleting heads or numbers in the spell-check phase. I’m hoping it can be done automatically (using tools from PGDP) but if it can’t it will have to be done during manual proofing. Keep line breaks. They will eventually be marked up with TEI lb element, provided that it can be done automatically. If it can’t then I won’t bother. The main exception is the index, where I’ve decided to keep each entry on one line. This will probably be marked up as a list or table, so line breaks will be irrelevant and possibly counter-productive.

Keep soft hyphens. FineReader marks possible soft hyphens as ¬ and often flags them as uncertain characters. It should be easy to replace the character with an entity reference. Sometimes it might not be clear whether a hyphen is soft or hard (eg Head-quarters). In these cases I can look for the same word elsewhere in the text. If the word only occurs once, then I can at least make an arbitrary decision without worrying about consistency.

Dashes will be replaced by entity references, except in the index where a hyphen stands for “ditto”, in which case it will be replaced by [do] to indicate how tables or lists should be structured. During the spell-check I’m leaving them as they are.

Accented characters aren’t being picked up by FineReader as it doesn’t seem to have foreign dictionaries installed. This is something I can fix for future projects. As it is, most accented characters are being flagged as uncertain, so I can add them manually using keyboard shortcuts or character map. I’m still undecided about whether to leave them as Unicode characters or convert them to entity references (I’ve read conflicting advice on this) but conversion should be trivial using Find and Replace.

Mistakes in the text are to be preserved. I’ve marked them with [sic]. This wouldn’t be appropriate for an edited text which already has its own [sic]s but I can get away with it here. The important principle is to flag the mistakes in a distinctive way which can be separated from the original text.

Roman numerals are mostly to be transcribed as they are, but there’s one exception. Sandall consistently uses I instead of 1. This means that my ideal of preserving the original sequence of characters is completely untenable. It would be very inconvenient for me and for users to preserve this idiosyncrasy. Since the emphasis is on accessible and useful empirical data, it had to go. Fortunately most occurrences are flagged as uncertain characters since FineReader is commendably cautious about l and 1. The necessary correction was obvious where the roman I was mixed with Arabic numerals (eg I9I7). Where I is on its own, it could often be decided from context (eg 1st, 1/5th, but I Corps). So far I haven’t encountered a truly intractable case, but there’s at least the possibility of checking against usage elsewhere in the text.

Abbreviations are another headache. Again I wanted to preserve the original sequence characters but am going to have to compromise over hyphens and points in abbreviations. I’ve decided to keep points but lose some hyphens.

Where the abbreviation is a series of single initial letters they can be separated by points but with no spaces or hyphens between them. So Q.M.S. not Q.-M.-S. or QMS or Q. M. S. Sandall sometimes has hyphens but sometimes doesn’t. It seems to depend on the rank in question (Q.-M.-S. but R.S.M.). I’ve decided to remove hyphens where they occur.

Where the abbreviation has one or more groups of letters from the same word, all points and hyphens are to be kept. So Lt.-Col., L.-Cpl. Maj.-Gen.

Some abbreviations don’t have points after every part, or use a slash instead of a hyphen (eg L/Cpl). In these cases they’re to be kept as they are. There are other abbreviations which have a space but no point or hyphen (A Coy, B Coy). These are also kept as they are (there is room for confusion here: a company or A company? I prefer to leave it as it is).

This sounds more confident than it actually is. I’m still compromising between preservation of original text and ease of use. Having tried some experiments with Google, it doesn’t seem to worry about points in abbreviations. A search for “ramc” seems to return the same results as a search for “r.a.m.c.”, but “r.-a.-m.-c.” confuses it. The Find function in Firefox isn’t so forgiving and will consider an abbreviation with points to be a different string from the same letters without points. If I want the best of both worlds, I could try marking the points up in XML and using AJAX to enable users to turn them on and off, but that might well be more trouble than it’s worth. Most of these abbreviations are ranks which will be marked up as part of personal names, or corps which will be marked up as organisation names, both with regularised forms in the attributes (I’m going to have to draw up a list of regularised ranks, corps, and formations). I haven’t been formatting abbreviations during the spell-check as my aim there is just to resolve issues flagged by FineReader and get the text closer to the original. I’ll be checking abbreviations later during manual proofreading, or using Find to track them down.

Fractions have caused problems for FineReader and usually come out as junk, but as they’re flagged as uncertain it’s easy to spot them. At this stage I’ve rendered them in PGDP style (1-1/2). During markup they’ll be marked up as numbers.

Another unexpected problem was the use of a large brace (ie } but bigger) to combine two rows of tabular data. My solution so far is to flag them with }} at the end of each row. Later they’ll be marked up as tables, with reference to the images to make sure the right hand column spans the correct rows.

Small capitals are another thing which I didn’t consider worth keeping. In the medal table in the appendix all names and ranks are in small caps. FineReader renders them as small caps in Rich Text but converts them to mixed letters when it exports to plain text. I think this is satisfactory and I can’t see much point marking them up as small caps.

Columns are to be marked up with the TEI element cb.

There was also some junk to be removed caused by shadows around the edges of the page images, but this wasn’t a major problem. It could be eliminated by cropping out the edges of the images.
Stepping through the automated checks for the main batch (196 pages) took 2 hours 50 minutes (1.1 pages per minute). In contrast, the end matter (only 17 pages) took 1 hour 30 minutes (0.2 pages per minute). This was mainly because of the smaller type, more complicated structure of the appendices and index, and more numbers (every occurrence of “1″ was flagged as uncertain). At the end of this stage I saved another plain text copy of the pages in a different directory, again for comparison purposes. Once I get some software that can automatically calculate the differences between the two sets of files I’ll have a clearer idea of the accuracy of the basic OCR and the improvements that can be made by automated checks.

My impression so far is that FineReader is reasonably accurate, and that it consistently flags uncertain letters. I spotted a few unflagged scannos as I was stepping through and might find more during manual proofing, but I’m generally pleased with what the software can do. So far I’ve only been using standard settings and haven’t explored the training features. If I was only making a digital text for my own personal use rather than for publication I think I’d be happy enough to run FineReader over it and rely on the automated checks without any manual proofing. It wouldn’t be perfectly accurate or consistent, but it would give me a useable text with many advantages over a printed book.

The next stage is manual proofreading to try to pick up anything that FineReader missed and to enforce consistent style where my style guidelines demand changes from the original text. Before I do that I’ll be thinking more carefully about my style choices and about any other potential issues that need to be resolved. I’ll also be looking at TEI in more detail to make sure I don’t do anything at the proofing stage which makes markup more difficult. I was hoping to break the process down into independent stages, but I’m starting to realise things are more interrelated than abstract models allow.

In my New Year post, I mentioned that I’m thinking about carrying out a couple of digital history projects in connection with my First World War research. These projects are very small and should be relatively easy to carry out on my own, but there will almost certainly be challenges. Overcoming these will give me more experience of carrying out a digital history project (this is starting to sound like a job application again!), and produce useful resources. After that, I can move on to consider some more advanced issues, such as collaborating with other people, and dealing with seventeenth-century manuscripts. To make the experience even more useful, I’m trying to blog it as I go. This post is an outline of my plans so far. Now that I’ve published my plans I’ll have to carry them out!

Background:

Before starting any digital history project, you should read Digital History by Dan Cohen and Roy Rosenzweig which is available online free of charge. It’s an easy to understand introduction which will tell you about different approaches to digital history, costs, benefits, potential problems, and generally help you to think clearly about what you want to achieve and how you can achieve it. The authors ask aspiring digital historians to get the right balance between caution and risk-taking. While a certain amount of planning is always necessary, there is also a risk that academic historians will spend too much time thinking rather than doing.

Having read the book and thought about the potential problems to be overcome, I think I’m in a good position to put my IT experience to use. I’ve been running my band’s website since April 2000, so I’m a veteran HTML coder. There have been many changes in web technology in that time, and I’ve always kept up with new developments while being careful not to jump on the latest bandwagon too soon. I have a good knowledge of (X)HTML, CSS, PHP, MySQL, image editing, Apache server management, and web accessibility issues.

I’ve studied the Old Bailey Proceedings, a large collection of digitized source material marked up with XML, got acquainted with the Text Encoding Initiative standards, and have experimented with XML myself. Recently I volunteered my services to Distributed Proofreaders, and have already proofread over 100 pages of text. I’ll be continuing with this, and once I’ve done 300 pages I can get experience of the next stages of the process. I’ll also be reading A Companion to Digital Humanities, another free online book.

I’m breaking my own rule that you should never rely on just one person when creating digital texts, but this is an experiment to see how much I can do on my own. I also want to avoid the added complications of managing a collaborative project. If things go well, I can start to look at ways of working as a team on unofficial and unfunded projects.

Project Outlines:

Project Sandall: This project will produce a digital edition of T. E. Sandall, A History of 5th Battalion Lincolnshire Regiment (Oxford, Blackwell, 1922). The text will be marked up with TEI compliant XML and published on the web. It will include a hyperlinked index of people, places, and organizations.

Project Wenham: This project will digitize and publish the correspondence of my great-grandfather, William A. Wenham, relating to his experiences as a prisoner of war during the First World War. The collection consists of a number of letters and postcards sent to his family from prison camps in Germany, with some with photographs. The text will be transcribed and marked up with TEI compliant XML, and published on the web, along with background information written by me. There will be an index of people, and possibly places. Another optional extra will be selections of relevant documents from other sources, such as battalion war diaries.

Theory:

One of the great things about digitization projects is that you can avoid some of the more difficult theoretical controversies. The most relevant theory is information theory, and Shannon pointed out that meaning is irrelevant to information. The aim of text digitization and basic markup is to represent the information in the original document as accurately as possible, minimising the noise which can be introduced as part of the process (see my post on Historical Information and Noise). Once the text is published, individual users can decide what it means to them and how far, if at all, it relates to the reality of the past.

However, record linkage brings problems of meaning and epistemology into play. This is likely to be worse in early modern manuscripts which are more noisy to begin with. For example, the same name might be spelt several different ways. Dealing with 20th century sources is easier because of standardized spelling, but linkage still involves taking an epistemological position. One advantage of TEI XML is that it has a built in mechanism for representing uncertainty and quantifying epistemic probabilities using the <certainty> element. Furthermore, XML preserves the original text. Editorial decisions made during the tagging process have little or no impact on the integrity of the text itself. If the XML code is made freely available to users under GPL, they can download the source files and edit them according to their own needs. Therefore, users who don’t agree with an editorial decision can easily change it in their own local copy of the text.

Copyright:

I believe that T. E. Sandall’s work is out of copyright in most of the world, since the book was published in 1922 and the author died in 1931, but I will be looking for more definite proof before I proceed with the project. The copyright status of the epilogue written by G. H. Teall is unknown, but it can be omitted if I can’t determine that it’s definitely out of copyright. The plates are more problematic and will probably be omitted.

William Wenham’s letters are more straightforward. There is no possibility of anyone outside the family having a claim on the copyright and no possibility of anyone being able to make money out of their publication. The status of the photographs taken at Cottbus is unknown. They might be out of copyright or they might be orphan works which are still under copyright. In this case I’m prepared to risk publishing them, but will remove them if a genuine copyright holder objects to their publication.

Image Capture:

I have a cheap A4 size flatbed scanner which gives adequate quality. The main limitation is that it’s slow, but the quantity of images to be captured is relatively small. A digital camera can capture images faster, but the quality of text is lower than scanning and would most likely lead to more OCR errors. Sandall will be scanned as 300dpi monochrome JPEGs. There is no need to create a high quality archival copy for preservation purposes. The main aim of the project is to make the text more accessible and add value in the form of XML markup and indexing. The family papers will be scanned as full colour 300dpi TIFFs which will be kept as archival copies. These masters can be used to create 100dpi JPEGs for web publication. It is intended to make images of the documents freely available in addition to the digital text.

Digitizing Text:

The scanned images of Sandall will be converted to digital text using OCR software. I have two free OCR packages: IRIS (which came free with a scanner) and Microsoft Office Document Imaging, but both have limited features and would be inadequate for this project. Abbyy Finereader Pro has more advanced features and is recommended by Distributed Proofreaders. For this project I can use the free trial version, and if I want to do more projects in the future I can buy a licence for under £100. PC Pro magazine considers Omnipage to be superior, but it costs around £400! Since no OCR is perfect, it will be necessary to proofread and correct the entire text. Finereader asks for user input when it encounters a doubtful character, and its interface should allow manual proofreading alongside that.

There is no possibility of using OCR for handwritten text, so the letters will have to be transcribed. Although we have the originals to work from, there might be some need to work from digital images. Apart from saving wear and tear on the documents, digital images are more flexible. Difficult text can be enlarged on screen, and contrast can be adjusted to bring out faded text. However, there is the added problem of viewing an image and typing at the same time, which might require specialised software. I’ve found that Zotero notes can be very useful for transcribing text from images or PDFs and might be adequate to start with. I could also use HTML/PHP/MySQL to cobble together something like the Distributed Proofreaders interface for my own use. The front end is a simple web based interface, and although I don’t know how their back end works, a local version just for my own use could be much simpler.

XML Markup:

XML can be daunting precisely because it’s so powerful and flexible: where do you start and how far do you go? Using TEI will give me a basic framework which covers the needs of the project and guarantees a certain amount of cross-compatibility. However, there are many different ways to implement TEI compliant XML and many subjective decisions to be made. At its most basic TEI markup need not be much more complex than HTML, but at the other extreme there are provisions for extremely detailed and complex markup, such as stylistic analysis of the structure of sentences. I’ve decided to split the markup into three phases:

First: mark up the basic structure of the text, such as paragraphs, headings, and page numbers. It might be possible to do some of this automatically, but manual checking and correction will be necessary.

Second: tag names of people, places, and organizations. For the printed text of Sandall it might be possible to automate some of this using regular expressions and Find and Replace functions. Proper nouns can be expected to have capital letters, and ranks and common referents (he, officer, man etc) should be easy to detect. Again manual checking and correction will be necessary. At this stage names will only be tagged to indicate that they are names. No assumptions about identity will be made until the third phase.

Third: link/index records by assigning ID numbers and/or regularized forms to the name tag attributes. At this point some subjective decisions might be necessary, and certainty will have to be evaluated and assigned.

XML tagging can be done in any text editor, but a dedicated XML editor might be useful. I have previously used a free version of Altova XML Spy for experimenting with XML, but some more advanced features might be necessary so I’ve downloaded a trial version of oXygen. If it proves useful I can buy an academic licence for $48.

Web Publishing:

The documents and supporting material will be published in HTML or XHTML. All authoring will be done in XML, and XSLT will be used to transform the source code into other formats. For this reason there is no immediate need to decide whether to use HTML or XHTML, and in practice there is very little difference between XHTML and good HTML (anyone who has problems with the transition to XHTML has written bad HTML to start with!). The XML source files will also be made available so that users can download and manipulate them. Ideally the site would be published in XML only, with XSLT style sheets transforming the code to (X)HTML on the client side, but this would exclude older browsers which can’t handle XML.

There will be little need for a database back-end as the sites will be small and the content will not be changed or added to frequently. A search engine which recognises XML fields (as at the OBP) would be useful, but as names will be linked to an index page, Google might be adequate for full text searches.

Publicity:

This is not much of an issue as the content will only appeal to a niche audience and the projects do not have to justify themselves to funding bodies. The main aim is to develop and demonstrate my digital history skills, so if I get it right it will be a useful demonstration of how to do it. The main avenues for promotion will be my blog, other people’s blogs, and the Great War forum.

Bibliography

1. Daniel J. Cohen and Roy Rosenzweig, Digital History: A Guide to Gathering, Preserving, And Presenting the Past on the Web (University of Pennsylvania Press, November 2005).
2. Thomas Edward Sandall, A History of the 5th Batt. the Lincolnshire Regiment. by Colonel T. E. Sandall, Etc (pp. vi. 221. Basil Blackwell: Oxford, 1922).
3. Ray Siemens, John Unsworth, and Susan Schreibman, Companion to Digital Humanities (Blackwell Companions to Literature and Culture) (Blackwell Publishing Professional: Oxford, December 2004).