In the handbook we wrote the following:
“Does an institution have permission to archive the content of blogs (and make it available elsewhere)? This might include permission not only from the blog author (which might be obtainable in the general terms and conditions of registering with the Institution), but also third-party content: embedded quotes, images, audio, video. Is it possible to excise potentially offending material, or is the risk (probably negligible) that an Institution might be sued for copyright breaches acceptable? Are institutional staff and students as well informed about the issues of online copyright as they are expected
to be about plagiarism, citation or photocopying regulations? Is it possible to include a default Creative Commons licence in the terms of use of the system?”

Update: Note we have since discovered that the licence for the blog which was initially published in the right hand side bar had disappeared during the upgrade. We have now reinstated the licence information. More information is given in the Archived blog page and in the Making any Upgrades to your Blog Sir? post.

Keep a copy of the software and a mysqldump of the database.

Keep a copy of the site in a structure format, such as XML or Atom.

Capture the site as plain HTML using a recursive wget or similar. This can be useful if the site software gets bitrot (nobody to maintain it) and you want to preserve the articles at their original URLs.

But this is the plan we use for bespoke installs of blogging software, not a blog hosted on a supported blog service.

More usefully, in your case, you might want to include the copyright holder(s) and the license of the entire blog so that it’s archived in the entires, not just the surrounding template. Don’t forget the comments may require a difference license, or remain copyright of the individual contributors. I don’t see any clear instructions right now telling me my comment is submitted under license XYZ. You couldn’t include this comment in a book without my permission!

Also the last post could start with “purpose/theme of blog:”, “intended audience:”, and “active from/to dates”.

Included with the contributors should be hints of how to get hold of them, long term. eg. twitter ids, and other non-institutional comms methods. Unless interested interneters asking you about it years later is the last thing you want, and just want to put the project in the past!

A last but one post could contain the more detailed info and a wrap up comment from the authors. This may include any notable incidents in the life of the blog, such as it being quoted somewhere notable.

Also, any key URLs from major posts or links to other bits of the project that’ll be preserved for sometime.

(This comment Copyright 2010, Christopher Gutteridge and contributed under a cc-by (must attribute) license)

http://www.sciam.com/0397issue/0379kahle.html

Kahle, Brewster, Preserving the Internet, Scientific American, March 1997

I am a librarian and writing an article for InfoToday on the UK Web Archive that the British Library has just announced:

You all seem to be the perfect people to give me perspective and context. Does the webarchive.ork.uk site have any particularly good or bad aspects? Is there something new and cool, or something important that is missing?

One of the interesting aspects is that IBM will be using open-source Hadoop software for the data store, and many proprietary and open-source technologies to try and get a handle on the content. Their plan is to use a spreadsheet interface and dynamically generate various rows and columns.

If this is a good forum to discuss it, please comment here, otherwise direct me to the right place or use the contact form on my web site.

Yours,

Avi

As far as Geocities rescue projects goes, another one worth a look is Reocities – http://reocities.com/

Maureen.

http://www.jiscdigitalmedia.ac.uk/training/digital-media-seminars/

If you are interested in keeping in touch with what JISC Digital Media are doing as a service then subscribing to our mailing list is the easiest way to do this. If you would like to join the mailing list then please follow the URL below to sign up:

http://www.jiscdigitalmedia.ac.uk/mailing-list/>

And following that, at the Preservation and Archiving SIG http://sun-pasig.ning.com/events/pasig-san-francisco-oct-79, also in San Francisco, Dave will present Ten Years of Digital Preservation with EPrints and, joint with Neil Jefferies and Ben O’Steen from Oxford University, on a new initiative, The Preservation Storage Network http://eprints.ecs.soton.ac.uk/17833/

http://www.resourceshelf.com/2009/08/08/web-archiving-service-preserves-data-for-the-future/

That said, as you imply, these types of figures can hide a great deal of complexity, in that certain types of Web content are likely to be more stable than others. Moreover, in practice, it probably only makes sense to think about link decay within particular contexts. For example, the article by Lawrence et al. cited above was specifically concerned with Web references in scientific publications, with a case study based on computer science journal and conference papers and technical reports. Their study found that the percentage of invalid links varied over time, e.g. from 23% in 1999 to 53% in 1994. Even then, much of the content (around 80%) was actually still available, and could be found using search engines or other methods.

Over the past few years, similar studies have been made of link decay in medical and general science journals [2], followed by similar analysis of various other domains. The evidence here also tends to suggest that link volatility is higher than that of the content itself [3]. Looking that these kinds of detailed studies might actually be more helpful than providing general link decay figures that appear to mean very little. I’m NOT saying that Web link volatility isn’t a problem, just that it is difficult to understand what link decay percentages mean without an understanding of exactly what they were trying to measure.

References:

[1] Steve Lawrence, David M. Pennock, Gary William Flake, Robert Krovetz, Frans M. Coetzee, Eric Glover, Finn Arup Nielsen, Andries Kruger, and C. Lee Giles, “Persistence of Web References in Scientific Research,” Computer 34(2), February 2001, DOI: 26-3110.1109/2.901164

[2] Robert P. Dellavalle, Eric J. Hester, Lauren F. Heilig, Amanda L. Drake, Jeff W. Kuntzman, Marla Graber, and Lisa M. Schilling, “Going, Going, Gone: Lost Internet References,” Science, 302(5646), 787-788, 31 October 2003, DOI: 10.1126/science.1088234

[3] Jonathan D. Wren, “URL Decay in MEDLINE – a 4-year Follow-up Study,” Bioinformatics, 24(11), 1381-85, 1 June 2008, DOI: 10.1093/bioinformatics/btn127

And lighten up, no one will care 50 years from now if you tweeted you were drunk or what you were eating.

WebCite provides a way of archiving URLs. However, it doesn’t
doesn’t seem to work for tweets. It does work for the FriendFeed URLs for tweets.

Right now I don’t know how “we” would handle it, and I don’t use CIL enough to know what the options for editing are.

But (imo) the same points I made above apply, and “we” should be prepared for that contingency.

The Twitter ToS say:

“Your profile and materials uploaded remain yours. You can remove your profile at any time by deleting your account. This will also remove any text and images you have stored in the system.”

Given this, I think, strictly speaking, you probably need to make it clear to anyone using an IWMW hashtag on Twitter that their tagged posts will be stored outside of the Twitter system, and even if they delete their Twitter account, you will retain copies of their content.

I suppose you may also need to consider the possibility that someone may request the deletion of stored posts? Given the clause above, it seems to me they would be quite within their rights to do so. Using a hashtag on Twitter provides a means of retrieving content, but it shouldn’t change people’s rights over their content.

I think this is an example of where making use of an existing service like Twitter with its own ToS differs from the case where you set up your own service for the event and you get to set the terms and conditions, and people get to accept them (or not) as they enrol for that service.

Surely it’s not beyond the wit of your audience to come up with a better solution (;-)?

The problem with this sort of action, similar to burning the library of congress, is that content that can no longer be updated or moved for some reason has no hope of survival.

I’ve just gone and dumped “www.geoci” on firefoxes’ awesomebar and opened just a few of the many things that turned up in serial tabs. One of them is dedicated to the page’s founding member, who died in 2000. Luckily that guy left the username and password with someone else who was part of the same club which the page services — there will be many more where the owner has passed on and left no trace. If I was to die tonight, my digital presence would go the same way. Will you burn all my old love letters and portfolios, or save at least some of them as keepsakes?

Damn this fast moving, live-in-the-now world we seem to be headed towards.

Apparently the whole of the Geocities space – the free part at least – takes up something like 15Tb. You’d think a massive corp like Yahoo would be able to keep a relatively minor concern like that running these days wouldn’t you? I’ve got £150 here that can buy another 2Tb disc like what I just purchased for my own media centre usage that can slurp up a nice 14% of all that, if 7 likeminded compadres want to chip in a similar donation.

Then we can just ask Y! to plug them into their server stack over USB / eSATA and have them sit for a day or two merrily filling up with all the old data, and we can let them go on their way.

This sort of thing really irks me because I was there pretty much at the start of it all. There are files in my oldest geo site going back to early 1995, soon after my school got an internet connection. Sure, my own stuff is crap and would only be saved for sentimental reasons (well, that and some of the photos which are still only on there in digital form, following a hard disc crash), but I’ve seen plenty of other people use it to make good quality, informative, useful (or artistically valid) sites. Which will now be swept to the four winds … either lost, or their creators (assuming they still care, never mind that a lot of people may depend on their content) having to find paid storage elsewhere, and re-link everything, and have everyone who linked to them (assuming you can even trace such things back) do similar. I find it horrifyingly ironic that a service provider such as Yahoo – whose core business is in content search, or at least once was – is so prepared to generate such a heaving mountain of dead links and lost data in one fell swoop.

It’s like burning a library. Never mind if you think 90+ percent of the stuff in there is shit, which is an opinion common to anyone who isn’t a competent polymath, think of the worthwhile stuff that will be wiped out. It’s maybe a bit like digging up an enormous time capsule, or finding an archaelogical site… with all the fabulous artefacts of a previous era, their most pertinent works of art, literature and research, and a whole heap of other everyday trinketry and what my stone hearted mother would call “trash” (yeah, thanks for throwing away that fully functioning clock radio you bought me as a 10th birthday present and had only “borrowed” from me… but not before cleaning all the stickers off… etc). Then nuking it.

And its not like there’s any other free or cheap services you can really use to recreate this any more either, certainly nothing with a sense of community. Social networking sites, most other “web 2.0″ (how i hate that word) services, and even Wikis are just no good for the kind of content collections you’d put on standard web hosts. Now if you want to make a “proper” webpage, rather than a generic Facebook profile or what-have-you (don’t even mention twitter to me), something that will hold a bit of useful static, textbook-like information, you’re going to have to pay for it. Through the nose. £5 a month is the way they’ve chosen to say it. Let’s try that another way – £60 a year. Considering how cheap all other internet things are these days, how can they defend that, and how many people can you imagine even bothering to do it.

Goodbye the idea of the internet being a free expression tool in the hands of the people any more :-/ …. yes I know GC was just an illusion of that itself, but it was a damn close one.

Incidentally, have we had anything that even resembles an explanation of *why* yahoo are doing this? I can’t imagine its a running costs problem. Maybe just trying to grub a little cash out of people who want to save their site paying for the premium one, and the little guys be damned. I’ve been a loyal user of their services (not even blocking the ads…) for a good decade and a half now between Y! mail and the absorbed GC, but recently they’ve been starting to piss me off. Forcing all kinds of stupid changes thru Yahoo Mail for one thing … deleting a secondary account without warning and with no hope of recovery (yes, it was little used, but it was my only remaining point of contact with some people who I still haven’t managed to reconnect with, and the GC site it was twinned to went simultaneously)… and now this. I think it may be boycottin’ time. Don’t care any more. Let’s have Microsoft slurp them up at a slashed price.

Incidentally, when Angelfire and the like went under … did we manage to save any of their stuff? I had a little AF account which was thankfully just an overflow for media files that didn’t fit on my main GC site, but in a similar manner there was so much content lost when it imploded.

This is all such a sad affair

It’s one of our most popular features so I’d say that archiving tweets is a pretty standard practice. It’s also much easier than setting up a FriendFeed room/group/imaginary friend .

Our premise is that blog posts are atomic events, data packets, that /might/ be more effectively captured cumulatively. Crawling is a very intensive process, typically at arbitrary intervals, often unrelated to the life-cycle of information in the target site, and frequently results in both unnecessary duplication and unwanted gaps. I’d be interested to know what the UK Web Archive is doing with any newsfeeds it harvests – but I might argue that capturing (A) HTML and (B) Newsfeeds is unnecessary duplication, if you can capture the underlying data and the queries/templates that render (A) and (B). (To say nothing of (C) the WAP view, and probably some others too). I admit this is a data-centric view – no doubt influenced by my database background, and time in a dataset archive.

As for the preservation, whether crawling or A-Pressing, that’s to some extent about what one does next, whether one has a relational database or a bunch of (W)ARC files. My guess is, also, that your average IT department will find it easier to understand the former than the latter.

FYI, Ed Pinsent, our resident archivist and Man From UKWAC, has just offered some of his preliminary thoughts relating to the crawling approach.

The use cases you describe here definitely help me understand more of what you see as the advantages – but would fall outside what I would think of as ‘preservation’ – perhaps it is agreement on what it means to ‘preserve’ a website that needs unpicking.

I think there are some distinctions to be made between how you capture the content, and how you store it, and how you can manipulate it. My instinct is that crawling html to capture the content is a more reliable way of getting the full picture for any particular blog at any particular time – and of course, if you capture the full content of the site, you get the RSS feeds as well. You could then decide to store this in a structured format if you wanted to do the kind of queries you describe.

I know Maureen promised a blog post on the project, so I’m looking forward to reading that as well.

An institutional archive of anything put out by newsfeed – tweet, blog post, wiki update… – and based on event-driven updating, rather than arbitrary crawling/backup intervals – sounds like a very worthwhile system and resource to me; and we will be looking into this for ArchivePress.

I think I’m clear on the idea that harvesting the HTML is not enough: what I believe we have in blogs are data, and data objects, and so the idea is to treat them as such. If I ask the version of this blog in the UK Web Archive to ‘show me all Brian’s posts’, or ‘show me all the posts from June 2009′, it’s dumb. With a database (relational or XML) I should be able to do that quite easily (as I can by querying the backend database to this blog; or wrapping that query in a web script.

Over an aggregated accumulation of blogs from an institution I think that if we can avoid losing the flexibility that the implicit data structure of blogs gives us, we should: all sorts of semantic, text-mining possibilities might await. To settle for screen-scraping solution is to lose that: we might as well just PDF everything.

We’ll be posting more thoughts on the data structure on the ArchivePress blog, but I don’t see it as ‘creating a database structure’ – all blog systems share common assumptions about data and metadata, in order to map to the key elements of RSS/Atom (or is it the other way round?!) and that’s kind of what we’re focusing on.

If you like you can blame Brian who told me a while back that a significant proportion of his readers consumed his blog via feed readers, not the web pages themselves; and Chris Rusbridge’s suggestion that “blogs represent an area where the content is primary and design secondary”: our premise lurks somewhere between those two ideas.

The versioning aspect I’m also interested in, as it relates to the archival issues of authenticity and reliability: blog posts are increasingly cited and yet highly susceptible to change or deletion. On balance, I’d prefer to easily cite blog posts in a stable archive rather than in the wild, and I hope the project might helps us get closer to that outcome.

Maureen will probably have more and better ideas. I’m looking forward to discussing it more over the next 6 months.

There’s a wealth of material associated with conferences worth preserving. Beyond that, and with greater impact, the history of #iranelection and potentially further public events will be more difficult to write if a Twitter archive is not preserved.

If you want to preserve the database as an artifact (which I’m not sure about as a general approach, but can see you might want to do this in some circumstances) then the only way to do this is to preserve the database – you can’t ‘recreate’ it from the feeds.

The more I think about this, I’m afraid the more problems I can see. What happens if I make a change to an old post? I can say that it is very standard practice for the web page representing that post to be updated (otherwise what’s the point) – but I don’t think you would see a consistent behaviour in the feeds from all packages – does an updated post from 3 years ago appear in the blogs RSS feed?

OK – so answering some of these questions is the point of the project – fair enough, and I don’t want to be overly negative about this – but I just can’t see the justification for doing this above harvesting the html. Could you post some more information on why you think creating a database structure (which may or may not bear some resemblance to the original database structure) for the content is important for archiving purposes?

I’ve got to say though, I’ve no interest in the preservation of my own tweets. I do have an interest in archiving Google Reader shared links and Delicious bookmarks.

here’s Thomas Crampton’s original post: http://www.thomascrampton.com/newspapers/reporter-to-ny-times-publisher-you-erased-my-career/

and here’s his latest, a chat with a search engine specialist: http://www.thomascrampton.com/uncategorized/search-expert-danny-sullivan-on-ihtcom-deletion/

BTW I’m sure the right name for a so-called “Twitteree” is a Twit. I certainly use it of myself (in this context, at least!).

Yes, and of course the British Library have done something similar in this country with their collection of 139 websites relating to the 2005 General Election in UKWAC. Unlike Library of Congress, no permission is required to access this collection, which is freely available online to the public.

Ed

Yes, as I suggested, this is just a beginning, but IMO a more satisfactory one than with BlogBackUpOnLine. The idea was to see how far you can get without hacking the code: extending the given functionality of the plugin to pull in comments too, and marry them up to the post, is probably not rocket surgery. Maybe it does that already, I’ll check!

As for look-and-feel… I still remember a certain gentleman, at Erpanet in Urbino, asking: “What if I suggested preserving look-and-feel was a load of dingo’s kidneys?” Or words to that effect.

Perhaps some kind of tiered system for classifying web archives would be useful, a la WCAG: e.g. Level 1 preserves information content, Level 2 preserves info+functionality, Level 3 preserves the info+fn+look-and-feel. Then organisations can define their objectives from the outset and not get sidetracked or scope-creep…

interesting post! Feedwordpress sounds like a very useful tool, particularly for institutions that want a central ‘repository’ of blog postings. I’ve seen it employed on other blogs and wondered how it was done.

I’d probably refer to this kind of an approach as an archive of blog posts than a blog archive – a subtle difference, but it result in quite a different end product, I think? A blog is more than just a collection of posts: it’s a platform for interaction and exchange of ideas with the community. Comments are usually enabled and the comments can become an important part of the blog, even enriching posts and expanding on ideas in the original post. Thought goes into the design and appearance of the blog. Links and widgets provide wider context. All these things make a blog more than just a series of articles, so for the sake of integrity I’d probably argue for a wider capture than the one described above. Though that said, you can probably set it to capture comments as RSS feeds… so what this really comes back to is clarity of requirements for preservation, or establishing exactly what it is you want to preserve and what it is necessary to preserve to ensure the integrity of your target is maintained and it can be re-used as per your re-use requirements.

Collecting just the posts is useful, but it restricts re-use of the archive – eg, it would be difficult to use it to illustrate how you as an institution engage with the community if you don’t keep your comments. However, I do agree that this is a useful and ‘low-cost, quick-start approach to securing data in The Cloud, and safeguarding the corporate memory.’ After all, institutions that want to do this will have to start somewhere and it’s better they capture the posts than capture nothing at all. It will be interesting to see how this particular plug-in develops further.

Maureen.

PS – What’s a blog farm? Is it like a worm farm – but with less squirming?!

Your readers may be interested to know that Hanzo is also working on a couple of open source web archiving projects.

WARC Tools are a collection of command line and web tools for the creation and manipulation of ISO 28500 web archive (WARC) files, funded by IIPC.

Search Tools adds full-text and metadata search capabilities to WARC Tools, funded by JISC. The HE angle on this is that Search Tools is a deliverable of the World Wide Web of Humanities project, a collaboration with Oxford Internet Institute in UK and Internet Archive in US.

More details and source code for these projects can be found here:
* WARC Tools: http://code.google.com/p/warc-tools/
* Search Tools: http://code.google.com/p/search-tools/

N.B. We will of course be looking at improving access to web-based resources wherever they sit. Again there is no question of existing urls changing although we will have to look at where new resources that are created collaboratively will sit (this should be seamless to the end user).

For example, at the DCC Conference last week, Amanda Spencer explained that TNA’s Web Continuity project has developed a system for government web servers that – as a last resort – can redirect to copies in the European Archive instead of presenting a 404 Not Found error.

My understanding of the article is that “no UK-based commercial online newspapers are currently being archived” by BL or UKWAC. No doubt the publishers manage archives of their own – it would be interesting to know what procedures and models they follow. Perhaps this is an example of where public archiving efforts should focus on APIs to those databases rather than (or as well as) the eventual web rendering.

In the case of the Ronnie Hazlehurst hoax, kudos to the Guardian for doing the right thing and flagging the correction. And to Wikipedia for having an auditable trail of edits. The discussion about whether the Hazlehurst “fact” was substantiated because it appeared in “reputable” sources, which themselves had sourced it from Wikipedia (without attribution) is particularly thought-provoking.

Shame the BBC article doesn’t explain the BBC’s position on its editorial integrity in this and similar situations.

The other thought is in terms of Creative Commons licences. People often don’t realise that even if they remove their CC-licensed content from the web, those that received the content while it was on the web still have a licence. As such, they are free to put it back up on the web.

Removing access doesn’t remove the licence!

A.nnotate will let you do web page capture – you can enter a URL or use a bookmarklet to take a snapshot of a web page and store a copy of the HTML in your private space on the a.nnotate.com site – which can be useful for archiving, as you get the page at a particular point in time. Currently it does a shallow copy (i.e. just the HTML) – the images etc are left on the original site, so it would need to download those too if you wanted to use it for archiving. The A.nnotate server is also available for local installation (with an API) if you want to integrate it with some other CMS.

You can also upload PDFs to A.nnotate (we use OpenOffice to convert from the various Office formats to PDF first) and these get converted to images and rendered in the browser using pure HTML / AJAX (without any dependency on Flash or Adobe reader).

We’ve certainly maintained a number of other resources well past the owner’s leaving date if there’s value to the community (i.e. the internet) and it doesn’t leave IT Services with a maintenence job it didn’t sign up for.

In Derek’s case, we were confident that he had backups of his data, so *whoosh*. In retrospect we could have done some more – for example to ensure redirection, but this I don’t really like this idea since it leaves us with an official resource potentially pointing at a sex site (if the domain name lapses).

In the case of the central wiki service provided by the University of Bath we have a clear “no deletion” policy, plus the ability to export the pages as they stand (along with attachments and comments) in HTML, PDF and XML. This is also our preferred policy for our much newer blog service (although not yet set in stone).

As Brian suggests there are a number of interesting issues relating to online educational artefacts like Auricle but for me at the core there is one issue that really matters, i.e. the value perceived by the different actors. I valued Auricle for its ability to help me organise and rehearse material and arguments for public consideration. Parts of the sector seem to have valued it as a resource, with Auricle sometimes making a small contribution to someone’s course or module. But, in common with all other HEIs, once personnel leave an institution such blogs, wikis etc associated with them are not analysed for the potential value of their content but treated more as an email account, i.e. something to be deleted. That’s sad. Why? Mainly because it represents a lost opportunity for ongoing collaboration, discourages future investment by blog, wiki authors etc in such institutionally-hosted resources, and forces a move to outside the institution (so reducing the chances of generating assets of future value to the institution). To be fair, however, it’s also necessary to recognise that supporting such resources do represent an ongoing cost for an institution (albeit a relatively tiny one in University financial terms). I now absorb such costs in sponsoring my own auricle.org domain. Nevertheless, I’m grateful to the University of Bath for allowing the original blog but, in the absence of an HE agency such as JISC offering a sustainable blog hosting service (free of editorial constraints), then the only alternative I can see is to do exactly what I have done, i.e. move into the ‘cloud’ and keep lots of copies so effort is not totally lost.

Derek Morrison, Editor of Auricle

We like to imagine that the internet is an always-on resource. But it’s not. There could be problems at your end, at the other end, or any number of points in between. Apple’s MobileMe has had teething troubles which have led to a free subscription extension of three months for its users. GMail, for all its claims of uptime and reliability, has been choking a little in the past few weeks, and has forced pro-cloud system administrators to have a bit of a rethink.

There is also the Feedwordpress plugin, which will import posts from external RSS feeds. This could be directed at one or more Twitter feeds, and tag-gregated feeds at Twemes etc, to form the beginnings of a selective institutional or personal Twitter Archive, built in WordPress.

Of course you could just keep the RSS, but this way you keep a lot of the functionality, accessibility and experience as well (with a few tweaks it could look almost exactly like Twitter).

At a time when even No 10 tweets, maybe the idea that selective Tweet archives might be valuable is not such a crazy idea.

We’ve all found that the tussle between conceptions of web resources as either “publication” or “records” can be quite vexing. On the Web so many things can fall between stools, and trying to treat everything as one or the other can feel like forcing square pegs into round holes, cracking nuts with hammers, [your favourite cliché of futile incongruence here!].

One attractive idea that we’re considering here is to make more explicit a third class of object, namely “artefact”. Mightn’t a web object, or group of objects, be selected for preservation as an “artefact”, purely on the grounds of being of demonstrable interest or use – without having to tick all the boxes associated with a “record” or a “publication”?

In the real world, preservation activities are divided between the triple pillars of Archives, Libraries and Museums, dealing respectively with records, publications and artefacts. If we can choose to keep a Chinese bell or Roman statue just because it is interesting, shouldn’t interesting web resources have the same rights?

(I was going to write a post about this, but this is quicker!

There are many instances where it may be worthwhile and, above all, feasible,to take an easily-defined web object (say, the blog of a completed project), knocking off any rough edges, and preserving it as a discrete artefact.

But equally we have to accept that the web is an endless stream of information, constantly changing, and deal intelligently with the need to maintain evidence or examples of its former states, without slowing its flow (as if we could).

Like the song says: “Dirty old river has to keep rolling….”

More seriously I think that RSS-driven automated capture sounds like a promising addition to the remote-harvesting armoury.

No technical reason we can’t layer Powerpoint-to-Flash, AV-to-Flash, etc, delivery systems on top of managed content in IRs (we’ve got 3 Eprints projects at ULCC that do just that, and another that adds commenting).

But we’ll never realistically keep up with the big boys, so if the likes of Slideshare and YouTube are going to take the hit of bandwidth, CPU, development, testing and UI design, to help us achieve publicity, dissemination, etc., why not use them as long as they are useful – but not for anything more than we can reasonably depend on them for.

Yes, you’re absolutely right. Disposal sits alongside preservation as a key stage of digital records management. I think for preservation of Web resources ‘selection’ is one of the most tricky issues as data storage capacity today might lead people into wanting to keep everything when sometimes disposal might be a more appropriate option.

Pleased to see that you’re having a good look at the blog!

Marieke

We are in the realm of what I was talking about in Aberdeen – different people using the same words to mean (subtly?) different things.

Thanks Marieke – you have just given me an idea for an exam question:-)

Unedited chat logs can make painful, occasionally hilarious reading, of course – something much riffed on, from The Guardian to Pelevin – but as the basis for edited minutes of formal meetings, they knock spots off audio transcription, shorthand systems, etc.

BTW, are all text-based, synchronous internet chat systems now subsumed in Web 2.0? Some of them predate “Web 1.0″.

I believe that if an educational institution is encouraging use of blogs to support reflection, discourse and deep-learning, it has a responsibility to make that online environment as safe as it tries to make its physical campus. Third-party hosting might be a reasonable alternative to the costs of service development and maintenance, but the institution must examine the T&C and functionality very carefully to ensure they meet standards it can recommend to those in its charge. Blogger, WordPress.com and Facebook are very general “tools”, and a particular institution might legitimately want something more tailored – like Edublogs, ELGG, Club Penguin even – or something truly bespoke.

Where an institution has insightful and innovative IT direction and policy, it could be prevented from moving at the pace of its own needs rather than those of the hosting company. (I was told just such tales about hosted VLE and assessment systems at JIF08 the other day.) The WordPress/Wordpress MU code base, for example, offers an excellent toolkit to build on: it underpins WordPress.com and Edublogs – we shouldn’t let the limitations of some of its current implementations, like JISCInvolve, blind us to its true potential.

My sense is that Warwick Blogs successfully created an online community predominantly “@ Warwick”, and I don’t think that would have been achieved if they had simply told all to go and get on with it at WordPress.com, FB or Blogger (none of them, BTW, so mature in 2004 when Warwick Blogs launched).

This is a little off the preservation track, but I think there is in common the theme that institutions be guided first and foremost by informed decisions about their own distinctive needs and responsibilities. Perhaps also that one’s data/content needs to be migratable.

I tend to agree that it’s rather hard to see what the value of a Twitter resource is, nor can I tell from this post at what point a Twitter turns into a record that requires preservation. Telling me that use of Twitter is advancing and becoming more institutionalised is worth knowing, but it does not convince me completely. Show me how its outputs relate to real-world institutional policies, and how they form a part of the corporate record. Show me what is at stake, and what are the risks of corporate memory loss if we decide to do nothing about Twitters.

I’m prepared to believe that if Twitter grows and becomes more widespread as an institutional tool, then its outputs and resources may conceivably reach a tip-over point when their value can longer be ignored. Or worse, we are forced to take action because of a disaster arising from a record-keeping gap.

I’m sure we all want to see the day when all desktop applications are so smoothly integrated with record-capturing mechanisms that a conscientious creator of information can select and add virtually any digital output into a comprehensive electronic filing system. It would be an optimistic man, however, who demands an instant solution. Institutional policies, records management practices and indeed software solutions to the problems raised by digital resources do not happen quickly; just ask anyone in the profession about the problems they’re facing with e-mail management.

UKOLN, for example, will be using twitter for the forthcoming IWMW 20098 event. In this case it is being used as a mechanism for the event organisers to be able to quickly send messages to the event participants – and if their tweets are delivered to their mobile phone, they will be able to receive alerts when they are away from a PC. As an official channel we will need to treat our (institutional) use of twitter in the same way as official emails sent out to delegates. I am conscious, for example, of the possible confusions (and perhaps, in more formal contexts, the legal ramifications) which could be cuased by inappropriate messages being sent out. And as around this time last year there were many problems caused by the floods in the southwest of England, I know that if there were serious problems associated with the event we would need a proper audit trail for tweets.

But I think your point that if a university has a twitter feed it should consider whether it wants to keep it/preserve it, is fair enough. Records of corporate activity (often minutes etc but also publicity) are something universities consider within their archiving policies and tweets could be considered part of that corporate record. The university can ask the “why” and if there is no good reason then decide not to. I then imagine that decision needs reviewed as some information/communication mediums may take over the role of others. For example you might keep a selection of paper news letters in the archives but perhaps Twitter or similar takes over the news letter role and then perhaps you need to update your policies. (Stating the obvious here

Future use is hard to predict and often we get it wrong, e.g. we keep a lot of data that does not get re-used…but maybe it will be re-used in the future.

Having said that I’d hope that twitter never went down the “scholarly record” route. If you or someone else has tweeted something of any significance, your next step should be to blog about it.

twitter is dynamic, the context is constantly changing, the language is informal – don’t go there!

The fundamental question is, “why?”, which I don’t think you either really ask or answer above??

Pandering to institutional vanity is one possible reason – just as institutions are now wanting to recall how their Web sites looked way back when it all started.

Preserving the scholarly record is another – if it can be demonstrated that tweets are beginning to form a part of the scholarly record that isn’t captured in other forms.

Meeting a legal requirement as part of QCA – where Twitter is being used to deliver learning (none of the examples above do this as far as I can tell, but presumably it is beginning to happen somewhere) then there *might* be a legal requirement to record what has been sent to whom, possibly as part of assessment?

Without know why it’s a bit hard to judge whether it is worth doing at all, or whether the potential costs and so on justify it.

Andy.

It’s not too late, though, for anyone else to contribute accounts of their own experiences, ideas, problems: please keep commenting, or contact us.

Revisiting Archive Collections is one methodology for achieving this, although it looks at a very different model of using facilitated group discussion to achieve the same ends. There’s a lot of interest in adapting this for the web but I’m not aware of anyone who has succeeded in getting funding for this yet.

It would be great to see UKWAC experiment with a facility like this for its collections, for instance.

I wonder if anyone really has suggested innovative developments be “banned” until their preservation implications are fully understood? As we know, the Web alone would be a plain and sorry place now if that were the case.

What is consistently overlooked in many discussions about preservation is the fact that it is – has to be – selective. Appraisal of what things ought to be preserved – informed but ultimately subjective – is essential to make any collection meaningful or useful, and for deciding how best to go about it.

In the SL case, we clearly can’t preserve every object, great or small, or every second of every user’s activity there, or every possible path, etc. – any more than we can in this world. Decisions need to be taken about what things in SL – from continents to conferences to individual users’ adornments – are worth recording, and why. That should help us decide how to do it, and, in turn, which DP approach best suits their preservation.

Like you say in your last para, we need to remind/help innovators to engage with all sorts of preservation possibilities, else they may regret themselves the loss of evidence of their work in 10 years time. And feasibility is a totally reasonable consideration when deciding what to do: keeping something, somehow, is better than nothing, as you and Ed have mentioned elsewhere.

But I’m also inclined to think also that, by the time an innovation is being rolled out and embedded across large institutions, it will be more mature and understood than globally novel, and at a phase where responsible institutional proponents should be ready to engage with preservation issues and include it in their planning.

Persistent URIs are not about technical solutions, they are about commitment. We must make sure we never break URIs!

(Having said all that, I need to explore more about in which circumstances, and for which kind of pages, breaking URIs is “OK”. At least partly because our hosting environment in Edinburgh is going to change some time, and we will not be able to do everything the same way. Oh hec!)

This seems a good case for public labelling of a site as ‘archived’ – though I don’t believe there is any consensus on what that actually means in the web management community. It’s perhaps also indicative of the low priority organisations sometimes give to maintenance of their websites once they are up and running, let alone once they have disbanded.

“Sometimes there may be a need for Web sites to be deleted” appears to me a rather drastic phrase, when perhaps you simply mean that the expired pages should be taken down and removed from public access. “Deletion” suggests that you would like the offending pages destroyed completely, a strategy which denies the possibility of NOC (or another agency) ever adding these pages to a web site archive. NOC should at least be keeping copies of their event activities; I agree with you that either they shouldn’t be doing it online, or they should clearly mark such pages as ‘archived events’ or referring to events in the past.

As Kevin says, many DP/WP initiatives, particularly those we’ve been involved with, have an archival angle, addressing the issue of authentically preserving digital records and information considered “essential” – significant, even. The happy medium of preserving both information and appearance, reliably and easily, still seems some way off: Adrian Brown describes some approaches in his book, but they seem very complex and resource-intensive.

I’m increasingly of the view that the idea of continuity, as described by TNA’s project for government, is an apt and valuable frame for all the key areas we’ve discussed in PoWR so far, including domain and URI persistence, design and accessibility, preservation and archiving. No doubt records management too!

We’re all generally happy to use sites that undergo radical changes as long as we can still do what we used to do without having to relearn too much. We’re not so happy when reorganisations mean that our old bookmarks stop working, and we have to hunt for the content we want all over again. And we’re even less happy when a reorganisation means that the information we want has simply disappeared from the site entirely.

Most of the time, preserving information content is sufficient, yet most current web preservation techniques tend to do better at preserving the appearance of a site (and hence the experience of using it). So there’s a gap, it seems, between what we want the technology to do for us, and what it actually does.

As Brian’s said elsewhere, we want to try and bring a user-focused approach to issues that have been covered in various forms and forums by DCC and DPC, not to mention Adrian Brown’s book. I’m approaching this as a great opportunity to take our own kind of snapshot of the state of things, and see how much of the material generated by events like the DCC’s 2006 Workshop has trickled down to practitioners. Part of the purpose of the workshops is to scope the PoWR Handbook, which we would like to make as practical and accessible as possible.

It will be good to see you at our first workshop.

In response to how does this project fit in with the Digital Curation Centre (DCC)?

The short answer is that while JISC-PoWR would hope to compliment the DCC’s work, the two are actually focussed on quite different areas of work.

The two main differences are 1) that while the DCC covers all digital data (but concentrates on scholarly and scientific materials) JISC-PoWR is focussed specifically on Web Resources (of any type). 2) The DCC is interested in maintaining and adding value to data (which preservation would be part of) and covers a wide range of topics (such as preservation of data, email preservation etc.) while JISC-PoWR is just looking at encouraging preservation by sharing of practice and transfer of knowledge between the UK HE/FE Web Manager community and the Digital Library and Digital Preservation communities.

Also JISC-PoWR is a short-term project while the DCC is a much larger project, with partners from The University of Edinburgh, The National e-Science Centre, Humanities Advanced Technology and Information Institute, Science and Technology Facilities Council (STFC) and UKOLN. As UKOLN is a member of DCC and the JISC-PoWR team we hope to be able to work together.

The different ‘preservation’ terms are defined in Digital preservation in a nutshell (Part I).

Thanks

Marieke

Plenty of food for thought … I’m looking forward to the event at Senate House, it will be interesting to see how (if?) things have moved forward since the foundation of the Digital Curation Centre. Speaking of which, how does this project fit in with the DCC?

I think the answer for complex web production technologies with a host of dependencies the only way forward is to take snapshots of published content as HTML with mirroring tools such as HTTrack.

How to take an effective automatic snapshot of a site that has complex user interaction is the real toughie – tools such as web application security scanners are beginning to become a bit cleverer at tackling interaction [is this/will this filter through to mirroring tools?] but it strikes me that crawlers and information retrieval systems with a degree of AI will be required to really crack the problem.

I am pleased to come across this blog as this is something I am working on since last few months. The National Centre for Research Methods (NCRM) has embarked upon a project called ReStore, a sustainable web resource repository. The aims of the projects are establishing a service for sustaining online resources and to come up with a long term strategy aimed at preserving & maintaining online resources. That would be quite helpful to keep intouch with people on this blog and to create awareness amongst researchers, developers and policy makers about the importance of web resource preservation.
hope to keep in touch.

Also, given that every Information Manager who does not already have web content under RM-like control has to “start here” – looking both forward and back, forging better contacts with web teams, registrars, etc. I hope at the workshops we can explore very practical examples of how they might do that.