Our premise is that blog posts are atomic events, data packets, that /might/ be more effectively captured cumulatively. Crawling is a very intensive process, typically at arbitrary intervals, often unrelated to the life-cycle of information in the target site, and frequently results in both unnecessary duplication and unwanted gaps. I’d be interested to know what the UK Web Archive is doing with any newsfeeds it harvests - but I might argue that capturing (A) HTML and (B) Newsfeeds is unnecessary duplication, if you can capture the underlying data and the queries/templates that render (A) and (B). (To say nothing of (C) the WAP view, and probably some others too). I admit this is a data-centric view - no doubt influenced by my database background, and time in a dataset archive.

As for the preservation, whether crawling or A-Pressing, that’s to some extent about what one does next, whether one has a relational database or a bunch of (W)ARC files. My guess is, also, that your average IT department will find it easier to understand the former than the latter.

FYI, Ed Pinsent, our resident archivist and Man From UKWAC, has just offered some of his preliminary thoughts relating to the crawling approach.

The use cases you describe here definitely help me understand more of what you see as the advantages - but would fall outside what I would think of as ‘preservation’ - perhaps it is agreement on what it means to ‘preserve’ a website that needs unpicking.

I think there are some distinctions to be made between how you capture the content, and how you store it, and how you can manipulate it. My instinct is that crawling html to capture the content is a more reliable way of getting the full picture for any particular blog at any particular time - and of course, if you capture the full content of the site, you get the RSS feeds as well. You could then decide to store this in a structured format if you wanted to do the kind of queries you describe.

I know Maureen promised a blog post on the project, so I’m looking forward to reading that as well.

I think I’m clear on the idea that harvesting the HTML is not enough: what I believe we have in blogs are data, and data objects, and so the idea is to treat them as such. If I ask the version of this blog in the UK Web Archive to ’show me all Brian’s posts’, or ’show me all the posts from June 2009′, it’s dumb. With a database (relational or XML) I should be able to do that quite easily (as I can by querying the backend database to this blog; or wrapping that query in a web script.

Over an aggregated accumulation of blogs from an institution I think that if we can avoid losing the flexibility that the implicit data structure of blogs gives us, we should: all sorts of semantic, text-mining possibilities might await. To settle for screen-scraping solution is to lose that: we might as well just PDF everything.

We’ll be posting more thoughts on the data structure on the ArchivePress blog, but I don’t see it as ‘creating a database structure’ - all blog systems share common assumptions about data and metadata, in order to map to the key elements of RSS/Atom (or is it the other way round?!) and that’s kind of what we’re focusing on.

If you like you can blame Brian :) who told me a while back that a significant proportion of his readers consumed his blog via feed readers, not the web pages themselves; and Chris Rusbridge’s suggestion that “blogs represent an area where the content is primary and design secondary”: our premise lurks somewhere between those two ideas.

The versioning aspect I’m also interested in, as it relates to the archival issues of authenticity and reliability: blog posts are increasingly cited and yet highly susceptible to change or deletion. On balance, I’d prefer to easily cite blog posts in a stable archive rather than in the wild, and I hope the project might helps us get closer to that outcome.

Maureen will probably have more and better ideas. I’m looking forward to discussing it more over the next 6 months.

If you want to preserve the database as an artifact (which I’m not sure about as a general approach, but can see you might want to do this in some circumstances) then the only way to do this is to preserve the database - you can’t ‘recreate’ it from the feeds.

The more I think about this, I’m afraid the more problems I can see. What happens if I make a change to an old post? I can say that it is very standard practice for the web page representing that post to be updated (otherwise what’s the point) - but I don’t think you would see a consistent behaviour in the feeds from all packages - does an updated post from 3 years ago appear in the blogs RSS feed?

OK - so answering some of these questions is the point of the project - fair enough, and I don’t want to be overly negative about this - but I just can’t see the justification for doing this above harvesting the html. Could you post some more information on why you think creating a database structure (which may or may not bear some resemblance to the original database structure) for the content is important for archiving purposes?