Sunday, January 20, 2008

Relevance Feedback






Tags: | |

Monday, January 14, 2008

The Programming Historian

My colleague Alan MacEachern and I have decided to write a book to teach practicing historians how to use programming to augment their ability to do research online. The Programming Historian will be provided as an open access work via the website of NiCHE: Network in Canadian History & Environment. We'll announce the details soon. In the meantime, here are a few things that will make this work different from existing books about programming...

1. We think that you should be able to put what you learn to work in your research practice immediately. Many beginning programmers lose patience because they can't see why they're learning what they're learning.

2. Digital history requires working with sources on the web. This means that you're going to be spending most of your research time working in a browser, so you should be able to use your programming skills in the browser.

3. Our examples will build on real historical sources online and on open source projects in the digital humanities. In particular, the programs that you create will be tightly integrated with Zotero.

4. We'll draw on a wide range of techniques from information retrieval; text, data and web mining; statistical natural language processing; machine learning; and other disciplines.

If you'd like to contact us with questions or comments, there is contact information on our faculty web pages: Turkel & MacEachern.

Tags: | | | | |

Tuesday, January 08, 2008

Results When and Where You Need Them








Tags: | | | | | | |

Saturday, January 05, 2008

All is Flux

If you wanted a motto for digital history, it's hard to imagine finding anything better than the one that Heraclitus is supposed to have come up with around 500 BCE, when he said something to the effect that 'all is flux' or 'everything flows' or 'you can't step into the same river twice'.

I think that many historians have a research model which looks a bit like this:
  1. Formulate question
  2. Do research
    1. Collect a bunch of sources
    2. Decide which look most promising and skim through those
    3. Read the most relevant ones carefully
    4. Take good notes
  3. Write
  4. Publish

We all agree that the stages of the research process are indistinct and blend into one another. We all agree that there is a lot of movement to-and-fro and back-and-forth, and time for visions and revisions. Nevertheless, this research model--what the heck, let's call it Parmenidean--is widely enough understood that many professors ask their graduate students questions like "Have you done your research yet?" or "When are you going to start writing?" The students, in turn, reply with answers that may please or displease their advisors, but which are understood to be felicitous in the pragmatic sense.

Digital historians, on the other hand, have to be thoroughgoing Heracliteans and reject questions like "Have you done your research yet?" The only sensible way to do research online is to be doing everything all at once all the time. The research model looks like this:
  • Until your interpretation stabilizes...
    • You keep refining your ensemble of questions
    • Your spiders and feeds provide a constant stream of potential sources
    • Unsupervised learning methods reveal clusters which help to direct your attention
    • Adaptive filters track your interests as they fluctuate
    • You create or contribute to open source software as needed
    • You write/publish incrementally in an open access venue
    • Your research process is subject to continual peer review
    • Your reputation develops

Do we have what we need to fully implement this strategy? A lot of the pieces are already in place, including massive textual databases, search engines with APIs, XML, RSS feeds and feed readers, high-level programming languages, and tools for online scholarship like Zotero. The combined literature of statistical natural language processing, text and data mining, machine learning, and information retrieval provide a cornucopia of useful techniques. If you know how to program you're already most of the way there; if not, now is as good a time as any to begin learning how.

Tags: |

Wednesday, January 02, 2008

What's the Opposite of Big History?

A couple of times in this blog, I've mentioned big history, an ambitious attempt to narrate history from the big bang to the present. Like microhistory, the Annales school, environmental history, and a few other thematic approaches to the discipline, one of the things that big history teaches us is that we can learn something different by judiciously manipulating the scale of our inquiry.

By providing us with access to completely new kinds of sources, digital history opens up some additional possibilities for manipulating scale. Consider, for example, the cached data provided by Google and other search engines. When you do a search you have the option of following the provided link, or of seeing what the page looked like when Google's spiders last visited. The date and time that the copy was cached is also provided, and it is straightforward to write a program to retrieve the current page and cached copy and compare them to see what has changed. As a test, I did a Google search for "digital history" on 2 Jan 2008 at 15:05 GMT and recorded the times that the cached copy had been created for each page on the first page of hits. Sorted by duration, the results were: 3 days 8 hours 14 minutes, 3d 10h 24m, 3d 15h 14m, 3d 15h 36m, 4d 14h 50m, 5d 3h 12m, 5d 5h 20m, 5d 8h 24m, and 257d 15h 52m.

Now suppose you wanted to write the history of a very brief interval, say a few hours, minutes or even seconds. In the past, this kind of history--I'm not sure what to call it--would only have been possible for an event like 9/11, the JFK assassination or D-Day. But with access to Google's cache data and some sophisticated data mining tools, it becomes possible to imagine creating rich snapshots of web activity over very short intervals. And to the extent that web activity tracks real world activity and can be used to make inferences about it, it becomes possible to imagine writing the history of one second on earth, or one millisecond, or one microsecond.

Tags: | | |

Tuesday, January 01, 2008

The Search Comes First

In December, I had a chance to visit humanists at a couple of universities in the Boston area and talk about digital history. One kind of question that came up repeatedly was foundational: What do humanists really need to know in order to be more effective online researchers? What should they learn first? What constitutes a baseline literacy? How can digital humanists be introduced into existing departments, or the techniques of digital humanities be added to existing curricula?

Since I started this blog two years ago and began teaching digital history classes, I've had the chance to revisit these questions a number of times. My original answer was that it all begins with search, and I think that still holds. For me, the essence of digital history is the shift to what Roy Rosenzweig called a "culture of abundance." The internet is unimaginably large and growing exponentially. Individual researchers, on the other hand, have a sharply bounded capacity to absorb or make sense of new material.

I think that a lot of historians are resistant to the idea of processing documents computationally, because they think of it as a challenge to, or supplement for, reading. Instead, computation should be seen as a way to augment human abilities. We still need human beings to read and interpret sources, and we must still train our students in traditional philological techniques. There's no getting around the fact, however, that the way that we find sources has drastically changed in the last ten or fifteen years.

According to Search Engine Watch, as of this past summer search engines worldwide were handling about 61 billion searches per month. More than half of these were handled by Google, making its ranking algorithms the most pervasive source of bias in the history of research. It's clear that humanists need to understand how search engines work, and need to be able to parameterize their searches to get the best results. Your ability to do a virtuoso close reading is irrelevant if you can't find the sources to read in the first place. Humanists who wish to place their own material online also need to understand search engine technology, because it is the deciding factor in whether a work can be found, read and cited.

In my conversations last month, the follow-up question was usually whether or not historians and other humanists will need to be able to program computers. I'm not sure about the answer to that. I'm certain that some of them will. The discipline of history is in for some interesting times, as interpretations backed by intensive research in a few archives will be confronted with those backed by machine learning or text mining of massive datasets. My hope is that we'll find a rapprochement... but then I'm an optimist.

Tags: | | | |