diglib: DCC Meeting November 21, 2005: Minutes

[Carol Hixson <chixson@darkwing.uoregon.edu>]

Below are my notes from the November 21
meeting. If anyone has any corrections or
clarifications, please send them to me or to
the list--Carol
****************************************************************
DCC Meeting November 21, 2005

Attending: Sara Brownmiller, Faye Chadwell, Nathan Georgitis, Corey Harper, 
Normandy Helmer, Carol Hixson, Erin O’Meara, Travis Ritter

The meeting focused on backup procedures and how the work we do on digital 
collections affects them. Travis described the current backup procedures. 
All servers except for oknos and libprint get backed up.
Janus has a separate backup procedure which was not described.

The full backup of all servers fits on 7 tapes. Each full backup is 1.5
terabytes. That includes some compression. There is one set of tapes
in the tape loader. Weekly backups happen on Friday. We keep the
previous two weekly backups: one here, one in the Science Library.

We keep two to three monthly backups at any given time. The first
backup of every month is retained for 90 days.

We have an annual backup starting with the beginning of a fiscal
year that is retained indefinitely. Erin noted that there may be some
data that should be discarded rather than retained indefinitely.

We have 7 sets of tapes. We just bought 30 more tapes.

The Friday backup takes a little over 24 hours. Every day we have an 
incremental backup that goes to disc. After the full backup on Friday,
the incremental backups are wiped out. The incremental backups are
affected by large-scale movement of files, deleting files, renaming of
files, or even by renaming a folder that has a lot of files in it. We have
room for 500 gigabytes of changes that can take place over an entire
week. Usually that's a lot. If large-scale changes are not coordinated,
the incremental backup pool fills up and no further incremental backups
would take place. This has happened one time that we're aware of.
This could affect the full backup , too, because the full backup process
makes use of the incremental backups.

Some things that would help: consider setting aside a large area that
never gets backed up at all, an area that could be used as a staging
area. This idea was rejected because of the difficulty of keeping things
in sync and because it moves us away from the guidelines we've
established for the mass storage unit – the work folders in the MSU
are already supposed to be the staging areas for projects.

We decided that if any of us is going to move 50 to 100 gigabytes on
any day, we should notify Systems. There are probably only a handful of 
people right now who do work on so many files in a given day: Corey,
Lesli, Normandy, Jon Jablonski. (We don't know what Media Services
does.) These people should also keep each other informed when
 they are preparing to do large-scale projects.

Travis suggested that we could look at some other things that might help.
It's possible that we could acquire a bigger incremental pool that would
give us more leeway. We would need to make sure there are more bays 
available, as well as check into the cost. We could also look at things 
like iscuzzi.

In the short term, we will investigate the feasibility of adding space to 
the incremental pool, if the cost isn’t prohibitive. We could send emails or
set up a web page to track on system activity.

In the long term, we have serious space needs. $9000 would get us 
high-speed connection to backups. Travis reported on how much our space 
currently costs us. Poison costs $4.40 a gigabyte; Ambrosia costs $1.36 a 
gigabyte. Tapes are .75 a gig. Our biggest expense is backing things up.

Corey then raised the issue of other large-scale work and its implications.
For instance, running MD5 checksums usually takes a while and no other
work should be happening in a project folder while that work is going on.
We discussed how we might better manage group membership. Corey currently 
tries to notify people with group membership when he's going
to be doing a large project that affects their ability to work on a project.
We discussed giving Corey the ability to lock members out of a folder 
temporarily. We also talked about using basecamp for managing
projects, since one of the things that basecamp does is enable
the sending of emails to a targeted group.

Carol raised the issue of investigating an off-site storage arrangement
with a nearby institution. She and Sara had talked about this before.
Identifying an Orbis/Cascade library would be easy since we could use the 
Courier service to exchange tapes. Sara noted that we have had some
earlier conversations with OSU but most people felt this wasn't far
enough away since OSU would likely be hit by the same natural disaster
that would hit UO. Sara will pursue this, perhaps with Washington.