In a few previous posts I’ve stressed the difference between information and meaning (which I picked up from Claude Shannon, the father of information theory) and some of its implications. For example, in this post I pointed out that Shannon’s separation of meaning and information is compatible with structuralist and post-structuralist theories which maintain that there is no inherent meaning in the text. (I’ve also had to deal with it in the course of digitizing a book – see here). Work on Artificial Intelligence has tended to reinforce this distinction: computers are very good at processing information but not very good at understanding meaning.

But last week Bill Turkel wrote a post which turned my understanding of the meaning/information dichotomy on its head. This isn’t such a new development as it’s following on from a post he wrote in March 2006, and that was inspired by an article by Rudi Cilibrasi and Paul Vitányi published in 2005. There’s a lot of mathematical stuff about compression algorithms which I can’t claim to understand, but the schwerpunkt is that without understanding anything about meaning, computers can compare similarities in the information content of texts and cluster them accordingly. The result is patterns that make sense to humans who can understand the meaning of the text. Bill’s example used entries from the Canadian Dictionary of National Biography, finding geographical and chronological clusters of entries.

Despite the attention grabbing title of my post, the distinction between information and meaning isn’t a false one. However, these experiments show that in practice the relationship between information and meaning within the context of a particular linguistic/cultural system is not as arbitrary and unpredictable as theorizing might suggest. Does this mean that structuralism could make a comeback against post-structuralism? Or do we need to move beyond both of those things and find a new way to think about text? Whatever the implications for theory, this is an exciting development which promises to be very useful in practice.

Last week I posted some thoughts in response to the discussions at A Historian’s Craft and Civil War Memory about history and philosophy. In that post I took some of the philosophical problems that affect history and tried to restate them in scientific terms. As Brett pointed out, this really amounted to stating the obvious in fairly uncontroversial terms, but I think that was worth doing in order to bypass the unproductive hostility between both extremes in the postmodernism wars (although the extent to which those extremes even exist is debatable). Whether the major problems we face as historians are philosophical, scientific, or a bit of both, the question remains: how much time should we spend thinking about these problems? In this post I’ll be discussing that question, but I have to warn you in advance that I can’t answer it. So there might not be much point reading any further…

First of all I have to say that I agree completely with Kevin Levin that as historians we can’t solve these problems ourselves. If cognitive scientists can’t work out what meaning is and where it comes from, then we have no chance. But while it would be naive and arrogant to attempt to find solutions to these problems, it would also be naive and arrogant to ignore them completely. I think we need to know something about the nature and extent of problems such as meaning so that we know how they affect our work. But how much is enough? Getting good at history is hard enough without also having to know about philosophy, linguistics, cognitive science or whatever. Maybe blogs can help here, because they provide an easy way to find out about unfamiliar areas. For example, I’ve relied quite heavily on Mixing Memory and Babel’s Dawn for science, and The Valve for philosophy and literary criticism. But how do we know if we can trust them if we don’t already know what they’re telling us? Could this also be a problem with peer reviewed publications which are outside our field?

Is it really worth trying to find out about problems which haven’t been solved yet and which we can’t possibly solve ourselves? The problem of meaning has major implications for history, but the jury is still out. It might stay out forever, or maybe just a lifetime. Right now we don’t even know where the answer is going to come from let alone what it will be. (If I had to make a wild speculation I’d guess that sooner or later the cognitive sciences will crack the meaning problem and that the answer will be equally uncomfortable for both empirical historians and post-structuralist theorists, but anyway…) Under these circumstances, we can’t confidently take any position, whether empirical or theoretical. We might all be wrong. Post-structuralist thought is valuable in that it reminds us that meaning is not straightforward, but that is hardly the last word. I found Elizabeth Clark’s History, Theory, Text quite disappointing because she promised that post-structuralism offered exciting new opportunities for medieval historians, but failed to deliver. Most of the book is a teleological triumphal progress towards post-structuralism in which she sneers at various historians and philosophers for not being post-structuralist enough. There’s far too little discussion of what post-structuralism can actually do for the historian. The way I see it, post-structuralism is a problem not a solution. I don’t want to ignore that problem, but I don’t want to admit defeat and stop writing on the grounds that people could just as easily find interesting meaning in words randomly generated by a computer.

(Somewhere in that paragraph I changed from first person plural to first person singular. Even I’m not sure what the significance of that is!)

So I believe that there are major problems confronting history, I realise that I can’t solve those problems, but I don’t want to ignore them either. Can I work around them in order to minimize their impact on my work? If the central problem is meaning then probably not. I used to think that digitization offered a way out of this dilemma because you could concentrate on transcribing documents without having to make any assumptions about what they mean: concentrate on information and exclude meaning. Now that I’ve tried digitizing text for myself I can see that it’s not as simple as that (see my theoretical agonizing here and here). Although meaning intrudes into every stage of the digitization process the problem is perhaps more manageable than it would be in literary criticism. Identifying the character string “Lt. R. E. W. Sandall” as a name and rank seems less problematic than interpreting the meaning of a poem (unless it’s by Jessie Pope maybe…). Once I’d established that my editing decisions were arbitrary, I had no problem getting on with it. I have a lot of sympathy for Kevin’s point that theory doesn’t seem to matter so much when you’re actually doing history. But am I deluding myself there? Or am I wasting time on unproductive thinking when I could be doing?

If you’re not convinced that what you’re doing is right, how do you motivate yourself to do the work? Historical research involves a lot of difficult and tedious work. You need a strong commitment to get through it. The possibility that my work could be proved completely worthless by new developments in a different discipline isn’t stopping me from doing history. There isn’t really anything new here. It’s always been accepted by most historians that future research could prove their own work wrong. New sources or new interpretations could easily overturn your conclusions. Historians have usually been able to carry on doing what they do rather than giving up in despair at the thought that they might be wrong.

The hypothetical extreme empiricists would be offended at the hypothetical extreme postmodernist’s suggestion that empiricism is just an arbitrary culturally constructed paradigm. I don’t have any problem with that suggestion. My “proper” work (ie my Phd thesis, my forthcoming article, and other projects that I’m working on) is mostly within the empirical paradigm. The rules and values of that paradigm are arbitrary, but that doesn’t automatically make it worthless. As Brett pointed out in response to my post last week, science is an arbitrary system constructed by human language and culture, but it’s a useful one which can predict or change the future. Empirical history can’t do that, but I still like it. Maybe that’s a lame justification, but it’s honest. I can’t make a strong case for any kind of history being really important, but I know that history is what I want to do. If I like it, and if there’s a paradigm that values my work, is that all I need? I also think that empirical research teaches valuable skills. Some of these are transferable to other careers (eg using databases, analytical thinking, project management) while others are more specialised (eg palaeography, latin). All of them are more valuable than ever in the age of digital history. We need people with these skills and familiarity with historical documents to work on digitization projects. You can only get good at these things through years of practical experience, not by reading Derrida. However, digitization projects also require familiarity with theories of text – structuralism, post-structuralism, and information theory are all highly relevant here.

Although I like working in the empirical paradigm, I also like to look outside it. Right now there still seems to be a big gap between my empirical and theoretical interests. Will I ever be able to bring them together, or are they incommensurable? If they are incommensurable, is being able to think about them both at the same time a strength or a weakness? I don’t want to get too attached to one way of doing things. I want to be as versatile as possible, but will that just make me a jack of all trades and master of none? There’s a serious danger of creating the appearance of being theoretically aware by lazily dropping the right buzzwords but not really understanding the ideas behind them. The phrase “truth effect” can create its own truth effect. In a comment to my previous post I mentioned this experiment which provides empirical proof of the truth effect: non-experts are more likely to accept bad explanations of psychological phenomena if they include irrelevant neuroscience terminology. Now I’m wondering how terminology affects perceptions of historical writing. I’d like to see more experiments here, but do psychologists consider history interesting and important enough to be the object of their study?

Thinking too much is bad, but so is not thinking enough. I don’t think there’s a single point in the middle that’s exactly right. Different amounts of thinking suit different people. Ultimately everyone needs to make their own decisions. And so I’ve written nearly 1,500 words without really saying anything. Does that make me a philosopher? No, just a blogger.

1. Elizabeth A. Clark, History, Theory, Text (Harvard UP: Cambridge, MA, 2004).

In my previous post about theories of digital text, I used Shannon’s communication theory to divide text into information and meaning, and then talked exclusively about text as information: a sequence of characters selected from a finite set. That allowed me to concentrate on one part of the problem, while excluding the more difficult problems associated with meaning. In this post, I’ll be trying to tackle some of the problems of meaning, while still trying to avoid as many as I can. I will also continue to avoid offering concrete definitions of “text” and “a text”, mainly because I haven’t found any satisfactory definitions yet, but I won’t be able to avoid using the word “text”.

When scanning, OCR, and proofreading are complete (meaning you’ve gone as far as you can/will go with it — last time I suggested that proofing can never be truly complete) you are left with one or more plain text files which contain reasonably accurate information. There is likely to be some noise in the form of wrongly transcribed characters, but it should be expected that you have selected methods which result in a level of accuracy that is acceptable to your project, and that you have assurance checks in place to be able to determine that the work does meet your minimum requirements. The information you have in the text file is a sequence of characters which more or less matches the sequence of characters in the book. What, if anything, should you do next?

You could just put the text file on the internet as it is and let users worry about what it means. This is the approach taken by Project Gutenburg. Their texts are all made available as plain text files, with some also available as HTML. This approach is based on the assumption that digitized texts should conform to the lowest common denominator, and that any additional markup might reduce cross compatibility and make the files inaccessible in the future. I don’t entirely agree with this view. TEI XML is a widespread standard and looks like it will remain so for a long time. XML files are not a proprietary format. In terms of file systems they are no different from plain text files: both can be read and edited by any text editing software on any platform. The XML Document Object Model should make it easy to update tags in the future, and if XML ever turns out to be totally useless, then you can at least use find and replace to strip it out automatically, leaving you with the original text and no markup. This is not meant to be criticism of Project Gutenburg. They are doing valuable work in making public domain works more widely accessible, and in developing tools and procedures for collaborative work. Digitizing and proofreading text is necessary before any markup can be added. Project Gutenburg stops before the markup stage, but there’s nothing to stop other people from taking PG text files and adding advanced markup.

Adding markup necessarily involves meaning to a certain extent. Even Project Gutenburg, which aims only at producing plain text editions, isn’t just transmitting the sequence of characters from printed book to ASCII codes. Some characters, such as page numbers and running heads, are omitted. This is a subjective decision about which information to include and exclude in the digital edition, based on what is most likely to be useful to readers. Therefore, there has to be some kind of judgement about what the information means.

While digital text offers more flexibility than printed text, the role of the editor is just as crucial as ever. As Edward Vanhoutte says: “The editor is always present in the organization of the material and the transcription of source documents”. Marking up the basic structure of a document according to established standards like TEI might seem unproblematic, but Susan Hockey points out that even this is an act of interpretation (p. 48). For example, replacing line break characters with paragraph tags makes an assumption about the meaning of line breaks. Hockey cites Huitfeldt’s observation that there are no objective facts about a text (p. 47). Adding TEI XML tags to a text file is imposing an arbitrary taxonomy. Nevertheless, all language is an arbitrary taxonomy. As long as we recognise that nothing actually is what it’s called, those taxonomies can be useful. The taxonomy you choose has to be relevant to your objectives, and therefore you have to know why you are digitizing a text, who the target audience is, and what they are likely to want from it. This is crucial, because there is no perfect way of digitizing a text. It also helps if your taxonomy ties into a system which is widely used and understood, and which does not vary unpredictably. TEI is a good starting point because it’s widely used, and while flexible enough to accommodate many different purposes is fixed enough to prevent too much random slippage.

In the interests of preventing slippage and maintaining cross-compatibility, it is vital to apply tags consistently. Julia Flanders points out that this is easier said than done because of the complexity and flexibility of TEI: the same feature could be marked up several different ways. Again this is a problem of meaning: how do you interpret the meaning of a sequence of characters, and how do you fit that interpretation into your arbitrary taxonomy? Flanders emphasises the importance of documenting procedures and modifying documentation in the light of management decisions on difficult interpretations or previously unknown features. The Old Bailey project has taken an innovative approach to this, using a wiki to co-ordinate XML tagging. In a collaborative project, regular assurance checks are necessary to make sure that all team members are following the documentation consistently, and that the documentation is adequate.

While marking up the basic structure of a text (paragraphs, chapters, headings) must be recognised as an act of interpretation and arbitrary classification, it should be relatively unproblematic in practice. This is particularly true of the book I’ll be working on first: a very conventional regimental history published in Britain in the 20th century. Colonel Sandall is hardly Ezra Pound or Jack Kerouac! The structure of a normal book can be seen in structuralist terms: although it’s an arbitrary system which doesn’t necessarily have a fixed relationship with reality, it’s fixed in relation to itself. Most people understand this system, which is much less complex than the whole of a language system, and publishers help to enforce conformity in printed works. Manuscripts are more problematic as they don’t necessarily have such rigid conventions. Interpreting the structure of William Wenham’s letters will be more difficult than interpreting the structure of Sandall’s book, but at least there are some established conventions of letter writing (again we’re not dealing with a modernist stream-of-consciousness here), and field postcards have their own basic structure.

The next stage of markup involves picking out dates, and names of people, places, and organizations, and therefore more subjective interpretation of meanings. At this stage no claims will be made about who or what the names signify. It will only be necessary to decide whether or not a sequence of characters represents a proper noun. Fortunately there is an established convention in English printed books that proper nouns are distinguished by a capital letter. This might even allow a certain amount of automation of picking out names, although the potential for confusion with capital letters at the beginnings of sentences will probably make a lot of human intervention necessary. John Lavagnino points out that names are not always easy to define and delimit. In Sandall’s book, names will often be accompanied by ranks, which makes them easier to spot.

The third stage of markup is potentially the most contentious because record linkage involves making epistemological claims about the identities of the people referred to by the names. The first question is whether the same name refers to the same person when it occurs in different places within the text. In doubtful cases the book’s index might help to disambiguate two people with the same name. A different rank doesn’t necessarily indicate a different person, because ranks can change. At this stage I will be attempting to reconstruct the author’s understanding of who is who, which means confronting the major problem of author’s intentions. This doesn’t mean that I can remain neutral or that my assumptions won’t influence the record linkage. Linkage at this level will be determined by my own subjective interpretation of what I think the author meant. I will have to assume that there is some consistent logic to what he wrote, but that can’t necessarily be proved from within the text.

What about outside the text? Linking the text to other records would add value for users. If identifications can be corroborated from other sources, then my judgements might be more secure. However, this also involves making more ambitious claims about meaning. How do I know that the same sequence of characters in two different texts means the same thing? Ultimately I don’t. Record linkage is an empirical technique which can’t necessarily be justified to post-structuralists, but I don’t necessarily have to justify it to post-structuralists.

Once again the important thing is the purpose of the project and the needs and expectations of its target audience. The main value of Sandall’s book is to amateur researchers who want to know more about specific individual soldiers or officers, or about what the battalion was doing at a particular time. These people are unlikely to be impressed by agonising about meaning, intentions, and epistemology. Their methodology will most likely be traditional empiricism. This is not to say that they will be naive — they can recognise that some sources are more reliable than others and that different people have different interpretations of what happened — but ultimately what they really care about will probably be “the facts” of what really happened in the past. I don’t intend to challenge those beliefs, but conversely my project doesn’t depend on them either. While record linkage requires claims about the meaning of information and the relationship between different texts, it does not necessarily involve any claims about the relationship between text and reality.

To a certain extent I hope that I can let users make up their own minds about the meaning of the text, and that if they disagree with an editorial decision they can either ignore it or save their own personal copy which they can edit to their own specifications. TEI XML adds a layer of meaning to the text, but doesn’t change the underlying information, unlike a database where the information has to be cut up and rearranged to fit into an arbitrary taxonomy. Allen Renear: “One might say that the TEI is an agreement about how to express disagreement”. Julia Flanders reminds us that editorial responsibility should not be offloaded onto the reader. The problems of digital text make the editor more, not less, important.

I hope I’ve demonstrated that nothing about editing digital texts is simple. Over the last two weeks I’ve become aware of more problems than I imagined when I set out, but it’s been very useful to think about these issues more clearly. Even if I can’t solve every problem, I can at least avoid some of them, and minimise the impact of others. Above all, these projects are intended to be educational, and I’m certainly learning a lot from them. Now I’m nearly ready to start creating the digital texts themselves.

Bibliography

1. Lou Burnard, John Unsworth, and Katherine O’Brien O’Keeffe, Electronic Textual Editing with CDROM (Modern Language Association of America, September 2006).
2. Susan M. Hockey, Electronic Texts in the Humanities (Oxford University Press, November 2000).
3. Ray Siemens, John Unsworth, and Susan Schreibman, Companion to Digital Humanities (Blackwell Companions to Literature and Culture) (Blackwell Publishing Professional: Oxford, December 2004).

As the next stage of my Digital History Projects I’ve been doing background reading and thinking about the theory of text. This week I’ve read Schreibman, Siemens, and Unsworth A Companion To Digital Humanities (2004); Burnard, O’Brien, O’Keeffe, and Unsworth Electronic Textual Editing (2006); Susan Hockey Electronic Texts in the Humanities (2000); and C. E. Shannon ‘A Mathematical Theory of Communication’ (1948). I can’t say that I understood everything (especially Shannon’s equations and Jerome McGann’s pretentious jargon) but it’s given me a lot to think about, and things are nowhere near as simple as I first assumed.

What is text? What is a text? It turns out that there are no easy answers to these questions. While I was right to think that digitization avoids some of the epistemological problems of history, allowing readers to make their own decisions about the relationship between text and reality, digital text presents plenty of new problems which could be equally intractable. A text is not necessarily the same thing as a book or an article or a play. Things get really complicated when there are differing versions of a text, as is often the case with medieval manuscripts. Should we classify them as the same text with some differences, or different texts with some similarities? The separation of information and meaning is an important concept which can allow us to think more clearly about what we’re doing, but in practice, the separation is not necessarily easy to make. This is how Shannon introduced the idea:

The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages.

By Shannon’s definition, the sequence of characters contained in a book can be considered to be information. We can select this message and attempt to reproduce it exactly without having to worry about meaning. At the very least, we can fall back on structuralism, since an alphabet is a fixed arbitrary system in which the characters are identified by the differences between them. There is no fixed relationship between a character and the sound it represents (for example, characters in the latin alphabet can be pronounced differently in English, French and German). The same character might be represented in different ways. In modern print there are different typefaces which can be used to represent the same characters. In early-modern handwriting letter forms are often very different from modern forms, and the same character might have different forms in the same word (especially s). This fits in with Saussure’s distinction between langue and parole: the size, weight, font, and even form of a character might vary, but it can still be identified as the same character in relation to the system it comes from. [EDIT: the proper words for this are substantives and accidentals] This is not to say that parole is unimportant. Typography can have a significant effect on how text is perceived and understood, just like regional accents can signify group identities and influence how speech is understood. However, it is useful to be aware of distinctions. Susan Hockey points out that computers force us to concentrate on what we are doing and why (p. 3). They also force us to analyse everything more systematically rather than assume that anything “just is”. I’ve already had to break text down into information and meaning, with meaning further broken down into equivalents of langue and parole, and I’m still a long way from having any idea of what text “is”.

In theory it should be easy to transmit a sequence of characters, even a long sequence such as a book. In practice, getting an accurate electronic transcript of printed text is one of the biggest problems for digital humanities projects. Whether using OCR or human double-keying, getting acceptable accuracy is difficult and expensive, and perfection seems unattainable. This is surprising when you consider that printed characters and ASCII codes seem to meet Shannon’s definition of a discrete channel: “Generally, a discrete channel will mean a system whereby a sequence of choices from a finite set of elementary symbols can be transmitted from one point to another.” So what’s the problem?

Shannon’s model gives us five parts of the communication system:

1. Information source
2. Transmitter
3. Channel
4. Receiver
5. Destination

The transmitter converts the message from the information source into a signal, and the receiver converts it back into a message which can be understood by the destination (usually a person). Shannon’s theory is mainly concerned with maintaining the integrity of a signal in the channel by calculating how much redundancy is required for a given level of noise. In terms of digitization projects, this is all about the electronic working of the computer and its peripherals. Thanks to the application of Shannon’s theory, we can usually be sure that when we press the “a” key on the keyboard, the “a” character will appear on the screen (the keyboard can be seen as the transmitter, and the screen as the receiver).

With double keying, the real problem is what happens between the source and the transmitter. Shannon wasn’t too worried about this, implicitly assuming that the person at the source selected the message they wanted to select. Even if they didn’t, he points out that the redundancy of the English language is about 50%, meaning that even if half of the characters are wrong the message will probably still be intelligible to the recipient. Academic projects demand much more than 50% accuracy, and also need to preserve mistakes from the original text, which makes things more complicated.

We could perhaps see the keyer as another communication system, which introduces its own noise by misreading or mistyping the characters. According to Shannon, any system can transmit a message perfectly provided that it’s transmitted slowly enough and with sufficient redundancy. This applies to typists as well as it applies to telegraph wires. Typing very slowly and carefully will reduce the number of mistakes you make. Having more people rekeying the same text will reduce the overall number of errors. In practice there will always be a probability, however small, that a mistake can be missed. It’s also likely that people will make similar mistakes in reading and typing, rather than introducing completely random errors (I don’t know if any cognitive psychologists have done any experiments on this, but it would be interesting to see if there’s any empirical proof to back up this suspicion).

If time and money are unlimited it should be possible to make transcription errors negligible by employing large numbers of typists and making them type very slowly and carefully. However, we all know that major digital humanities projects don’t have unlimited time and money. Getting the right balance is important, as is having realistic expectations. Are the demands of digitization projects too high for the available techniques? Are time and budget considerations pushing text keying beyond its limits and making errors inevitable?

OCR is attractive because it offers the possibility of automating text capture, bypassing the expense and unreliability of humans. However, Cohen and Rosenzweig cite studies which show that when the time and cost of proofreading and correcting OCR text are taken into account, double keying works out more cost effective as well as more accurate. This is because computers are much worse at recognizing characters than humans are. You can scan a document at 300dpi, and those dots will appear in the same sequence on the screen. Perfect transmission, or near enough. But when the computer tries to select a message from those dots as a “sequence of choices from a finite set of elementary symbols” things often go wrong. This is immensely frustrating, because to a human it seems like such a simple task. We can hope that advances in Artificial Intelligence will eventually lead to reliable OCR, but it’s not going to be an easy problem to solve. (The ultimate proof of the unreliability of OCR is that the online version of the Companion To Digital Humanities is full of scannos!)

As it is, OCR text needs to be proofread by at least one human. Distributed Proofreaders now use three rounds of proofing (followed by two rounds of formatting). Because of the “open source” nature of the project, which is run by unpaid volunteers, time and cost don’t need to be considered at all. A text is ready when it’s ready, and nobody has to pay for it. This makes triple proofing more feasible than in a funded project. However, it might also be the case that more proofing is required because the proofreaders themselves are an unknown quantity. As I haven’t qualified for round two yet, I don’t yet know how much time round two proofers spend correcting errors introduced by less experienced proofers in round one. Radical trust is feasible provided you get a critical mass of responsible users (which DP appears to have attained) and offers some interesting possibilities. Large numbers of unpaid volunteers doing small amounts of work very carefully might overcome some of the problems of big digitization projects, although it might also bring problems of its own.

Allowing users to supply corrections after publication could also help to increase the accuracy of transcriptions. This is even more radical than the DP model, and gives some traditional minded people the fear. Berrie, Eggert, Tiffin, and Barwell take a very traditional view of authentication which is based on the assumption that editors can make a text perfect and that it will deteriorate if not controlled (even more depressing is their emphasis on defending copyright). In the light of everything I’ve discussed so far, I suggest that the opposite might be true: it is impossible for any individual editor or team of editors to produce a perfect text, and that the more people who are involved in correcting errors, the more accurate the transcription is likely to be. Wikipedia shows that with a critical mass of committed and responsible users, even deliberate vandalism can be overcome. This is not to say that every electronic text has to be, or can be, as open as Wikipedia. The most obvious problem is getting that critical mass of users, and this will be more difficult for more esoteric projects in which fewer people are likely to take an interest. At the very least, there should be some mechanism for users to suggest corrections, even if these have to be reviewed before being implemented. For example, the Old Bailey Proceedings has a form for submitting errors.

My current projects are small enough that I can take a lot of time and care over them, but I also want to develop techniques that will scale up, otherwise the experience will be of limited value. The relatively small amount of text to be dealt with means that in absolute terms there are likely to be few errors. It’s when you scale things up to millions of words that a small probability of errors can lead to a huge number of errors.

So far I’ve only considered characters as information, and haven’t got any closer to defining what text “is”. For the purposes of digitizing a book, I can avoid that question by setting out my aim as transcribing all of the characters in a particular book. Even though the definition of a book is at least slightly less problematic than the definition of a text, there’s more to a book than a sequence of characters. I’m choosing to represent one aspect of the book while discarding others, such as ink, paper, and binding. This is an arbitrary choice. Partly it’s because of the impossibility of representing the book as a complete physical object within a digital computer. Perfect information of that kind would need to go down to the level of atoms, and we would need some mechanism for reconstructing objects from the information contained in the computer. This is getting into the realms of alchemy, and clearly isn’t possible with any current technology.

But above all, any digitization project needs to look to the requirements of its intended users. From this point of view, it’s the information contained in a book, the sequence of characters making up the message, which potentially has the most value for readers. “Frequently the messages have meaning”. In the next part I’ll be going beyond information and looking at the even greater problems associated with meaning.

Bibliography

1. Lou Burnard, John Unsworth, and Katherine O’Brien O’Keeffe, Electronic Textual Editing with CDROM (Modern Language Association of America, September 2006).
2. Susan M. Hockey, Electronic Texts in the Humanities (Oxford University Press, November 2000).
3. C E Shannon, ‘A mathematical theory of communication’, Bell System Technical Journal, 27 (1948), pp. 379-423, 623-656.
4. Ray Siemens, John Unsworth, and Susan Schreibman, Companion to Digital Humanities (Blackwell Companions to Literature and Culture) (Blackwell Publishing Professional: Oxford, December 2004).