ALEG

What I've done

• Implemented the rendering of work summary and work details (including reviews/criticisms and holdings information) as plain text to the browser and as email back to the user. This entailed reworking some of the output generation screens because it turned out that given the complex and "deep" nature of our formatting, XSL stylesheets were not such a good tool for generating the plain text output. Basically, white space is too hard to control. So, a more successful approach was to perform string manipulation on the generated HTML, changing para breaks to 2 new lines, end-of-table-rows to 1 new line etc.

  This was a bit suprising, and the output modules were designed with a view to plugging in different stylesheets to perform the various formatting, so a bit of refactoring of the code was required.

  Multiple work details can now be listed on one page and selected for plain text or email formatting. Hence, now we have check boxes against work titles almost everywhere!

  I'm sceptical whether plain text and email will be used very much, especially as modern browsers support cut and paste as text and emailing pages as HTML and lovely print formatting, but hopefully some people will find this functionality useful.

• The system received many requests as part of the public beta this week, so I spent some time monitoring performance and tuning. Tests with the Opera 5 browser were successful, adding it to IE4+ and Netscape 6.1+ in the list of browsers which render the site OK.

  We found some minor problems with our rendering to IE4 and Netscape 6.1 which have been corrected.

• The Java Virtual Machine hung again for about 10 minutes until Marie-Louise rang me and I restarted it. Once again, the JVM did not respond to the "QUIT" debugging signal, so something is seriously wrong.

  As there's been no news on a fix or new version, I went back through the logs trying to find a correlation between the hangs, and I think I've found something: the hangs have always occured either soon after the server has been restarted, or after all the stylesheets have been flushed.

  [Side note: the system makes heavy use of the Xalan XSL stylesheet processor. XSL stylesheets are instructions marked up in XML. The Xalan processor reads the XML and computes an internal form of the stylesheet - sort of like compiling a program. This process takes a while, so we keep a "cache" of "compiled" stylesheets, and reuse them rather than reading them as XML from disk every time. But I've noticed that if we keep them forever, the memory used by the JVM climbs without bound (bad!), so we periodically flush the compiled stylesheets which frees the memory, but requires us to re-read and re-compile them.]

  As well, there was an unresolved problem a few months ago were occasionally the stylesheet processor would crash reading a stylesheet - this was not reproducible, and seemed to just disappear one day...

  OK, so maybe something flakey is going on during stylesheet compilation. Specifically, maybe there is a synchronization problem deep in Xalan or Xerces (the XML parser used by Xalan). This is all mere speculation, but in an attempt to rule it out, I've implemented our own synchronization around the Xalan compilation code to ensure that only one stylesheet can be compiled at a time. This change is now running, so we'll see if it "works around" these hangs. Of course, the whole JVM hanging (and not responsing to QUIT signals) should never happen, so this isn't ever going to fix the root cause, but it may avoid the problem.

• Each night at 9:30, we restart the server (for luck). On Wednesday evening the server stopped but did not restart. The output was confusing - the script that performs the restart thought the server was still going, and rather than starting it again, issued a "restart" signal to it, but then found that the server had indeed stopped. Maybe this was a flukey timing problem. To make things more robust, I've inserted a 3 second wait between the stop and the start, and then another restart is issued "for luck"...

• Minor maintenance of edit screens, stylesheets and data. One of the more noticeable changes is that now no criticsm or reviews are displayed by default when a work is rendered.

Next Week

• Implement Advanced and keyword anywhere searching.

• Revise EAD production for Lu Rees Finding Aids (following changes devised by Megan and Marlene).

• Report of most criticised agents to help drive Roger's work

• Merge subjects

• William noticed that some of the place-of-publication data for loaded records is wrong where the name of the place of publication (town or city) occurs in more than one state (eg Richmond, Glebe). I'll investigate when we map the spatial thesaurus. Dan, Chris and Terry have also noticed similar incidents, so I think a very careful look at all place assignments to places occuring in multiple states/countries is warranted. (And Joan noticed Aberdeen NSW -v- Aberdeen Scotland!)

• Multiple creation events for a work as a mechanism for allowing date ranges to be associated with agents responsible for works, eg editors of a periodical.

Links of the week

• Page by Page History of the Web - John Schwartz, New York Times

  An article about Brewster Kahle's "WayBack Machine", part of The Internet Archive. The implications of this work are amazing:

  The project has spurred a kind of enthusiasm that hasn't been seen in a while in the downhearted tech world. Lawrence Lessig, a Stanford University law professor who seeks to explain the interplay of technology and society, was uncharacteristically ebullient. "My brand is pessimism," he said. "This is not something to be pessimistic about. Brewster is my hero."

  Mr. Lessig not only admires the project for the knowledge it will preserve. He said he also thinks it can shift the balance in the debate over copyrights and access to intellectual property like books, music and movies.

  Holders of copyrights will eventually drag Mr. Kahle into court, Mr. Lessig predicted. So far the battle over copyright has been fought chiefly by copyright owners and their lawyers on the one side, and college professors and computer technicians on the other. Mr. Lessig says that will change if people use the Wayback Machine. "We finally have a clear and tangible example of what's at stake," he said. "Brewster is defining the public domain." Users will see "how easy and important this technology would be in keeping us sane and honest about where we've been and where we're going."

  So, it may be "WayBack", but hopefully it propells us "WayAway" from 1984.

  Update, 5 Nov 2001: Dumpster diving on the web - Katharine Mieszkowski, salon.com

  .... For instance, a wire service, such as the Associated Press, might balk when it discovers that thousands of its stories, published on other sites, can be freely visited in the Internet Archive Wayback Machine. The testy members of the National Writers Union may also view the archive as an unauthorized and uncompensated republishing of their work. There's also the tricky question of what happens if a settlement in a lawsuit requires that libelous material be removed from a Web site, yet the original lives on in the archive?