Tag Archives: digital resources

How to retrieve ancient text data from Perseus

10 Wednesday Apr 2013

Posted by in Classics resources, Digital Classics, Greek Classics, Latin Classics, Software & Tools

≈ 1 Comment

Tags

, , , , ,

In my last post I was describing problems with the URL schema not being entirely predictable, and therefore computable from body of text to body of text (e.g. from Livy to Caesar). That is the way the URLs are formed, what constitutes a ‘body of text’, and what you might expect to see returned in a request and how that varies with each textual work.

Update: Schema will now include a ‘urn’ attribute

Warning: this is a long and somewhat technical post about using the Perseus CTS API to fetch classical texts as XML data

This stuff is important for software developers and “digital classicists” (that is, classicists who work with computer-information systems for analysing information about the classical world).

On the Digital Classics mailing list, some helpful hints managed to emerge to my queries. The first is, the Perseus XML interface I was using (it’s the one that’s behind the helpful “XML” button at the bottom of each passage in the HTML version that you typically use with your web browser) is probably on its last legs.

CTS Overview

The more up-to-date (but still in beta) version is Perseus CTS; where “CTS” stands for Canonical Text Services. CTS is built on work done by the Homer Multitext Project.

CTS appears to have three main functional components:

  • A catalogue service (actually called “getCapabilities”)
  • A reference validation and exploration service
  • A service that retrieves text

Some commentary on its limitations

What it is missing, is a search service. The catalogue is huge. It has listed in it every available Greek and Roman text in the Perseus database and includes details of all editions and translation of each text. It’s available here http://www.perseus.tufts.edu/hopper/CTS?request=GetCapabilities and I’m not actually linking that URL because don’t click on it just yet. It’s 2.1 MB of XML. Your browser may not like especially like it. Mine only manages to load it properly half the time.

When you do manage to download it and save it on your local disk (highly recommended), you’ll see it’s a pretty comprehensive catalogue of the data. Unordered. With no links to the texts in either the reference validation or text retrieval services, and nothing obvious as a field that gives you the unique identifier needed.

What the references are constructed from

The reference validation service assumes you know the reference you want to validate (and discover the sub-components of). But you need that first-level peek into the initial reference. Perseus uses Thesaurus Linguae Graecae referencing system for Greek texts, and the Packard Humanities Institute PHI Latin Texts system for Latin texts. These both principally organise their respective corpora around authors, assigning each their own index number. Thus, Homer is ‘tlg0012′ and Livy ‘phi0914′.

The references are formatted into a type of reference called a URN.

How to create the references

Now I’m going to tell you how to construct a functional reference ID for the CTS system.

First thing, load the catalogue URL into your browser. I’m not going to link it but cut and paste this one into your browser: http://www.perseus.tufts.edu/hopper/CTS?request=GetCapabilities – if you know how to use Wget or Curl use that instead.

Save the file to a convenient location on your disk. I called mine “CTS.xml”.

Open the file in a text editor. Notepad won’t cut it. Word most certainly will not (it’s not even a text editor!). One the Mac, I recommend BBEdit. [Update: it's been pointed out on the mailing list that Oxygen XML editor is an ideal tool. I use this tool at work and have it on my Mac at home. An Academic licence is $99, a full one nearly $500. Unless you do extensive work in XML I would not recommend to buy it. Probably on Windows by default Internet Explorer is the default program for an XML file. It, or Safari on the Mac, will suffice to read the document. Google's Chrome also works pretty well. Browsers will also "pretty print" the XML to make it easier to view.]

Use your editor’s search capability to find the author you want.

The ‘textgroup’ (normally the author) identifies the first level

You’ll find that the author’s work is contained in an XML element called “textgroup”. Here’s the text group for Livy, along with the groupname element identifying it:

<textgroup projid="latinLit:phi0914">
  <groupname xml:lang="en">Titus Livius (Livy)</groupname>
  ... (thousands of lines omitted)
</textgroup>

Pay careful attention to the ‘projid’ attribute of the textgroup. This helps form the root of the URN used to identify the text in Perseus. The URN always starts with ‘urn:cts:’. Add the projid to that, like this:

urn:cts:latinLit:phi0914

Check it in the reference validation service

That’s all texts/editions/translations by/of Livy in the Perseus database. Here’s a link to the reference validation service: http://www.perseus.tufts.edu/hopper/CTS?request=GetValidReff&urn=urn:cts:latinLit:phi0914. If you open that link, you’ll see, in XML, a list of all the available URNs for every version and edition and translation of Livy in the database. But unfortunately, no descriptive information what each version edition or translation is!

We still need the catalogue file. Go back to the catalogue file.

The ‘work’ identifies the next level of reference

Search for a book. In my case, let’s look for “Book 1″ of Livy. You’ll see the catalogue file is unordered. The version I looked at, Livy books started at Book 11 (what? The one of the missing books is miraculously in the Perseus database I hear you say? Unfortunately, it’s just the periocha of book 11). The unordered nature of the database make it especially annoying: you have to search, and you can’t browse.

Anyway the entry for Book 1 looks something like this:

<textgroup projid="latinLit:phi0914">
  <groupname xml:lang="en">Titus Livius (Livy)</groupname>
  <!-- ... (thousands of lines omitted) -->
  <work projid="latinLit:phi0011" xml:lang="lat">
    <title xmlns="http://purl.org/dc/elements/1.1/" xml:lang="en">
    The History of Rome, Book 1</title>
 <!-- ... (thousands of lines omitted) -->
</work></textgroup>

See how the Book is contained in an XML element called “work”? Note the “projid” element of the work. In this case, we don’t need the “latinLit:” part, the interesting part of the id is the “phi0011″: that’s the ID for Book 1 of Livy. We add it to the URN we’ve been constructing as follows:

urn:cts:latinLit:phi0914.phi0011

The ‘edition’ and/or ‘translation’ identifies a specific version of the work

While that’s supposed to be valid reference to Livy’s book 1, Perseus contains at least two Latin editions of the text and three English translations. These are listed inside the “work” element in either “edition” or “translation” elements, like so (for brevity I have omitted some lines that give data about the citation system of the edition):

<work projid="latinLit:phi0011" xml:lang="lat">
  <title xmlns="http://purl.org/dc/elements/1.1/" xml:lang="en">
   The History of Rome, Book 1</title>
  <edition projid="latinLit:perseus-lat1">
    <label xml:lang="en">The History of Rome, Book 1</label>
    <description xmlns="" xml:lang="en">Titi Livi ab urbe condita libri 
     editionem priman curavit Guilelmus Weissenborn editio altera auam
     curavit Mauritius Mueller Pars I. Libri I-X. Editio Stereotypica.
     Titus Livius. W. Weissenborn. H. J. M&amp;#252;ller. Leipzig. 
     Teubner. 1898. 1.
    </description>
    <!-- some lines omitted -->         
  </edition>
  <translation projid="latinLit:perseus-eng1">
    <label xml:lang="en">The History of Rome, Book 1</label>
    <description xmlns="" xml:lang="en">Livy. Books I and II With An
     English Translation. Cambridge. Cambridge, Mass., Harvard 
     University Press; London, William Heinemann, Ltd. 1919.
    </description>
    <!-- some lines omitted -->         
  </translation>
  <edition projid="latinLit:perseus-lat2">
    <label xml:lang="en">The History of Rome, Book 1</label>
    <description xmlns="" xml:lang="en">Livy. Books I and II With An
     English Translation. Cambridge. Cambridge, Mass., Harvard 
     University Press; London, William Heinemann, Ltd. 1919.
    </description>
    <!-- some lines omitted -->         
  </edition>
  <edition projid="latinLit:perseus-lat3">
    <label xml:lang="en">The History of Rome, Book 1</label>
    <description xmlns="" xml:lang="en">Livy. Ab urbe condita. Robert
     Seymour Conway. Charles Flamstead Walters. Oxford. Oxford 
     University Press. 1914. 1.</description>
    <!-- some lines omitted -->         
    <memberof collection="Perseus:collection:Greco-Roman"></memberof>
  </edition>
  <translation projid="latinLit:perseus-eng2">
    <label xml:lang="en">The History of Rome, Book 1</label>
    <description xmlns="" xml:lang="en">Livy. History of Rome by Titus
     Livius, the first eight Books. literally translated, with notes 
     and illustrations, by. D. Spillan. York Street, Covent Garden,
     London. Henry G. Bohn. John Child and son, printers. 1857. 1.
    </description>
    <!-- some lines omitted -->         
  </translation>
  <translation projid="latinLit:perseus-eng3">
    <label xml:lang="en">The History of Rome, Book 1</label>
    <description xmlns="" xml:lang="en">Perseus:bib:oclc,2311635, Livy.
     History of Rome. English. Translation by. Rev. Canon Roberts. New
     York, New York. E. P. Dutton and Co. 1912. 1. Livy. History of 
     Rome. English Translation. Rev. Canon Roberts. New York, New York.
     E.P. Dutton and Co. 1912. 2.</description>
    <!-- some lines omitted -->         
  </translation>
</work>

Now, assuming we’re after the Teubner edition of the text (the first one), we can use that edition’s ‘projId’ attribute as before, and stripping the ‘latinLit’ from it and adding it to the URN we’ve been building up, we get:

urn:cts:latinLit:phi0914.phi0011.perseus-lat1

This is the complete reference to the Weissenborn & Mueller edition of Livy’s Book 1 published by Teubner.

Check it in the reference service

We can hit up the reference validation service with that URN as follows: http://www.perseus.tufts.edu/hopper/CTS?request=GetValidReff&urn=urn:cts:latinLit:phi0914.phi0011.perseus-lat1 – you will see a complete collection of URNs for the distinct parts of Book 1 in the Teubner edition of the text.

URNs for specific passages

This URN is all of the preface that’s found at the start of Book 1:

urn:cts:latinLit:phi0914.phi0011.perseus-lat1:pr

This URN is all of Chapter 1 of Book 1 (not including the preface):

urn:cts:latinLit:phi0914.phi0011.perseus-lat1:1

You can also get parts of chapters, here is 1.4.2:

urn:cts:latinLit:phi0914.phi0011.perseus-lat1:4.2

Fetch the text chunk you want

These arguments are passed to the ‘urn’ parameter of text retrieval service of Perseus like this: http://www.perseus.tufts.edu/hopper/CTS?request=GetPassage&urn=urn:cts:latinLit:phi0914.phi0011.perseus-lat1:pr (that’s the preface).

Anatomy of the URN format used by Perseus

    urn:cts:latinLit:phi0914.phi0011.perseus-lat1:4.2
    {1}:{2}:   {3}  :   {4} . {5}   .  {6}       :{7}
  • {1} It’s a urn. This part is fixed.
  • {2} The urn is part of the ‘cts’ namespace. This part is fixed.
  • {3} The Latin Literature namespace. Would be ‘greekLit’ for Greek texts, and possibly other values.
  • {4} The textgroup’s identifier. It’s normally either the TLG or PHI author index value. In the catalogue it’s contained in the ‘projid’ attribute of the ‘textgroup’ element, stripped of the namespace.
  • {5} The work’s identifier. This may map to an author’s title or to an individual book in a larger collection of texts. This also apparently comes from either TLG or PHI indices (I’ve not verified this fact for sure). In the catalogue it’s contained in the ‘projid’ attribute of the ‘work’ element, stripped of the namespace.
  • {6} The edition of the work. This may also be a translation. This is a Perseus-specific value. In the catalogue it’s contained in the ‘projid’ attribute of the ‘edition’ or ‘translation’ element, stripped of the namespace.
  • {7} The text reference. This will be specific to the work and edition you are referencing. You can find out a simple unadorned list of what’s available by querying the reference validation service with the URN up to this point at the argument.

Note how the textgroup, work and edition use dots for separators but otherwise the data element delimiter is a colon.

Commentary

There are still problems:

  • You cannot get all of book 1 in a single hit (at least for Livy).
  • If you want book 2, you have to repeat this process (it’s phi0012)
    • So, Chapter 1 of book 2 of the Teubner text looks like this URN: urn:cts:latinLit:phi0914.phi0012.perseus-lat1:1
    • Repeat and rinse for other books/editions
  • Entirely different authors and works may have different results or slightly different algorithms for building URNs.
  • The catalogue elements ‘textgroup’, ‘work’, ‘edition’ and ‘translation’ should each have a child element, ‘urn’, that builds this URN for you, so that such explanations as I’ve attempted are unnecessary.
  • The reference checking service needs to include a modicum of descriptive information about the URNs that are returned.
  • There needs to be a search service that stitches all this together.

I hope someone can find this of use.

Inconsistencies in Perseus and unpredictable URL formation

08 Monday Apr 2013

Posted by in Classics resources, Digital Classics, Latin Classics, Personal

≈ 1 Comment

Tags

, , , ,

Some of you may be aware I’ve started programming for the iOS system in recent weeks. The stuff I’ve been doing, a native iOS interface into the Perseus Online Latin Word Tool, has really been a warm-up (to build up my Objective-C and Cocoa Touch chops) for the thing that I really want to build. I won’t go too far into that because it would be too boring to explain it in detail. Let’s just call it, “The Livy Electronic Reader”. Think of it as an iPad app that allows you to build your own translation and commentary of Livy (or some subsection of it), and as an ancillary, publish the data out to a shared Dropbox directory (or, maybe iCloud, or possibly a shared publishing mechanism, perhaps something like, a “Livy wiki”). My plan, once I’ve done enough for Livy, is to perhaps extend it to other authors, e.g. Caesar and Tacitus. I picked Livy because that’s my research interest. I’m really building a tool for myself to use for my PhD.

However, if anyone can answer the following questions about Perseus, its data format and URL scheme, or know where I can find answers, I’d be much obliged.

The first question I have, is why does the XML interface behave inconsistently in the data it returns? Can it be made consistent? More importantly, can it be made predictable and therefore computable?

Here are some examples of what I mean.

First, something that behaves reasonably predictably. These first links are to Caesar’s de Bellico Gallico

Clearly, the “document name” is “1999.02.0002″ and by adding arguments :book=n :chapter=n and :section=n you can select more or less of the content as you wish. Perfect! Give me a reference to Caesar and I can retrieve the text in an easily transformable XML format.

In contrast to to the former logical behaviour, consider these following links to Weissenborn and Muller’s 1898 edition of Livy’s text.

However if you look at those two links, you’ll see the text is not the same text. That’s because what I really labelled above a “1.1.1″ is really book 1 praefectus 1. You can access it directly with this URL:

OK, so maybe the Livy text is thrown by the presence of the special “preface” chapter.

Additionally, if you want, say, book 22 chapter 1, you might predict that this could work:

But sadly, no. That’s not even a valid document. I guess the later books are in different editions, and thus to get to book 22, it’s an entirely different document URL.

Let’s try to get the whole book contents:

Predictability is one of the greatest virtues for URI schema, and this seems to break it. The data inside the documents suggest it is broken into book, chapter, section, and the URL retrieval scheme suggests it can be retrieved as such, but there is different behaviour depending on the document content (Livy or Caesar).

So it looks like that in my app I’ve have to build a static tree of the document URLs, rather than being able to compute them on the fly, which is a much better way of doing things, usually.

Does anyone have any insight to this behaviour of Perseus? Suggestions? Comments?

More reasons to dislike electronic format books

29 Sunday Apr 2012

Posted by in Academia, Classics resources, Digital Classics, Literature, Personal, Software & Tools

≈ Comments Off

Tags

, , , , , ,

Continuing on from previously:

Another problem with electronic formats is the ease of “skimming” for research purposes. But this is probably more a matter of interface design that needs to catch up. For those who don’t know: to traverse a lot of literature quickly, you need to be able to, non-linearly but nonetheless highly methodically, work your way through a potential target text, often by just reading say, the first and last paragraph of every chapter, or the first and last sentence of every paragraph. Electronic books kind of assume that you will read the entire thing sequentially; flipping through them can be hard.

Something else related to this is the internal references in the text themselves. I mean god knows why book publishers insist on end notes but their idiocy is massively amplified in electronic editions. It’s even worse in multi-author compilations, when you just don’t want to check the end notes, but the bibliography for each article is off with the fairies somewhere down the back the (750-page) book, and the publisher has provided no internal hyperlinking. This is just stupid, and actually makes using the text for what it’s intended, i.e. scholarly research, almost impossible. Why do they do this, even think that’s a good idea? Publishers, if you’re not going to hyperlink the references then every reference ought to be a footnote, and rendered right on the same page with the text it appears in. Got that?

On the dreaded DRM see this excellent post by the author Charlie Stross.

Citing texts from electronic editions

25 Wednesday Apr 2012

Posted by in Academia, Classics resources, Digital Classics, Literature, Personal, Software & Tools

≈ 1 Comment

Tags

, , , , , , ,

Once again I’m confronted with the practice of the “page” number in modern books when I’m now reading many of the secondary texts in electronic formats (our library prefers to buy the texts in that form, as I’ve ranted about here before, because they use a really stupid Adobe DRM’d format). My school’s PhD referencing guide needs to be updated; it is silent on the issue of e-books. Kindle uses a “Location” number which can be utilised, but it’s specific to the device. If you don’t have a Kindle (lets say you got the same book from Apple iTunes/iBooks, or from another e-book vendor, or you have the paper version), the Kindle “Location” is irrelevant to you.

The real problem here descends from the idea of “page” numbers as the ideal format. It’s tied to a very particular presentation scheme that plainly is about to become out of date. The whole idea of “pages” or whatever proprietary display format you’re looking at ought to be dropped and we should concentrate on the texts themselves.

I’m a classicist. We have standard editions of each text in which each is given a canonical numbering scheme, usually along the lines of book/chapter/sentence. Some examples; Livy 5.51.5 is Intuemini enim horum deinceps annorum vel secundas res vel adversas; invenietis omnia propoera evenisse sequentibus deos, adversa spernentibus. Epic poetry is typically quoted as book and line number, e.g. Vergil Aenid 1.278-9: His ego nec metas rerum nec tempora pono; imperium sine fine dedi, or for collections of shorter poems, book/poem/line, e.g. Horace Carm 2.3.1 Aequam memento rebus in arduis. These citation schema are assigned to each text when a scholar has painstakingly compiled a ‘standard’ edition of the text from the surviving manuscripts (often published by Teubner, so sometimes referred as the ‘Teubner edition’ of the text).

For modern texts, publishers should be now assigning each text a ‘standard edition’ way of citing the text and if necessary, embedding that information into the text itself if the ebook formats won’t support it. This way it won’t matter if the reader uses a Kindle, or a Kobo, or iBooks or even a printed version; the text references are constant, and apparent to all readers.

Of course, ideally the ebook publishers ought to all agree that they will each have a way to embed the publishers’ citation system into each of their own individual formats such that every edition of a text can be referred to in a standardised way by all users of any format. But of course as we are stuck in the “format wars” period and have a vicious, anti-scholarship intellectual property regime imposed on everyone by vested corporate interests, I’d be surprised if that happens.

Opening up an ancient Egyptian library « British Museum blog

05 Thursday Apr 2012

Posted by in Archaeology, Classics resources, Digital Classics, Egyptian History

≈ Comments Off

Tags

, , ,

Via Lindsay Powell on Twitter, an informative and fascinating post about the Ramesseum papyri at the British Museum. Making more of this sort of stuff available online is really exiting, I think.

Opening up an ancient Egyptian library « British Museum blog:

The British Museum’s Online research catalogue format offered a marvellous tool for this visual presentation, especially as it is linked to the collections database with its descriptions and bibliographies. Unlike a print catalogue it is continually updatable (and it needs to be: in May I am in Geneva to examine a new doctoral thesis by Pierre Meyrat on the previously untranslated magical texts in the library). Many of the fragments have not been fully published, some have never been published in photographs before, so this format will open up the library for study – as a whole and for the first time in its modern history.

Greek Inscriptions Online Database

04 Wednesday Jan 2012

Posted by in Archaeology, Classics resources, Digital Classics, Greek Classics, Greek history

≈ Comments Off

Tags

, , ,

An online, searchable database of Greek Inscriptions from The Packard Humanities Institute:

Searchable Greek Inscriptions: A Scholarly Tool in Progress. The Packard Humanities Institute.

Description [From the digitalclassicist wiki]: The Packard Humanities Institute, in conjunction with Cornell University and The Ohio State University are making available online an extensive corpus of Greek inscriptions, intended to supersede the CD-ROMs PHI originally distributed. Access is free to all who accept their terms of use.

(Via AWOL – The Ancient World Online)

Humanities use of ‘fuzzy’ data

01 Thursday Dec 2011

Posted by in Archaeology, Digital Classics, Humanties, Science & Tech, Software & Tools

≈ Comments Off

Tags

, , , , , , , ,

A neat summary of the sort of problem that often arises in the interface between the sciences and the humanities.

How do we balance supporting novice spatial users alongside experts? Or, is geospatial analysis necessarily GIS?:

By focusing simply on technology training, there is the danger that, as well as being seen as irrelevant, too difficult or simply just boring for users (academics or students), the data gets overlooked or is made to fit a ‘system’ of analysis. For example, one problem of using GIS in humanities is the issue of ‘fuzzy’ data. This isn’t just a case of the system failing to cope with fuzziness: it also betrays an underlying assumption that data can, and should be, disambiguated and clear. For humanists, however, the questions driving research are often precisely those that look to nuance or complicate the material. We like messy results. Humanists need worry less about producing an accurate and/or truthful representation and more about how maps can be used as entry points to explore the data—this is seeing maps as   part of the investigative process rather than as an end in and of themselves.

(Via PELAGIOS)

New Approaches to Greek and Roman Myth (Camden, Jan 21)

16 Wednesday Nov 2011

Posted by in Ancient Religion, Greek Classics, Latin Classics, Reception

≈ Comments Off

Tags

, , , ,

From the CLASSICISTS mailing list:

Conference: New Approaches to Greek and Roman Myth

The Department of Classical Studies at The Open University warmly invites you to a one-day conference on “New Approaches to Greek and Roman Myth”. The event will be held at The Open University Regional Centre in London (Camden) on Saturday 21st January, 10am-4.30pm. Details of speakers and paper titles appear below. There will be £7 charge to cover the cost of lunch and refreshments. Booking forms and further details are available from Jessica Hughes (jessica.hughes@open.ac.uk).

Colleagues may also be interested in a new myth-related resource that the OU has recently produced – a collection of video animations and audio discussions on Greek Heroes and their representation in popular culture from ancient times to the modern day.
http://www8.open.ac.uk/platform/news-and-features/new-itunes-u-addition-looks-greek-heroes-in-popular-culture

all best wishes,
Jessica Hughes

  • Anastasia Bakogianni (OU) – Electra Ancient and Modern: The Reception of the Myth in Greek Tragedy and in the Modern World
  • Joanna Paul (OU) – The Half-Blood Hero: Percy Jackson and Mythmaking in the 21st Century
  • Annie Ravenshill-Johnson – The Metamorphoses of Hephaistos/Vulcan
  • Frances Eley (OU) – Staging Sacred Drama: Reception in Process
  • Sam Newington (OU) – Greece and the Near East: Creation Myths
  • Tori McKee (OU) – Reconstructing the Hippolytus myth: Euripidean Echoes in Modern Adaptations of Racine’s Phedre
  • Susan Deacy (Roehampton) – How to Write a Classical Mythology Textbook

A map showing the location of The Open University London Regional Centre can be found here: http://www3.open.ac.uk/contact/maps.aspx?contactid=1

-----------------------------------------------------------
Jessica Hughes
Lecturer in Classical Studies
The Open University
Walton Hall
Milton Keynes MK7 6AA
Website www.classicsconfidential.co.uk

The Pelagios Graph Explorer: An information superhighway for the ancient world

13 Sunday Nov 2011

Posted by in Archaeology, Classical history, Classics resources, Digital Classics, Science & Tech, Software & Tools

≈ Comments Off

Tags

, ,

The Pelagios Graph Explorer: An information superhighway for the ancient world:

Just as the settlements around the Ancient Mediterranean would seem disconnected without the sea to join them, so online ancient world resources have been separated, until now. Meaning “of the sea”, Pelagios has brought this world together using the principles of Linked Open Geodata. The Pelagios Graph Explorer allows students, researchers and the general public to discover the cities of antiquity and explore the rich interconnections between them.

(Via PELAGIOS)