Unexpected Errors from Metadata OAI-PMH Providers

Abstract

We study the behavior and the failure reason of metadata harvesting services. We used existing OAI services and we created our own OAI client to issue requests to them for many harvesting rounds, collecting the appropriate information. We studied 1407537 harvesting tasks from 3446 harvesting services in 552 harvesting rounds during a period of 2 years, of which 618812 (44%) failed and the remaining tasks occasionally returning fewer records. We examined the reported outcome messages, the number of records returned and the response time to discover failing patterns. While most messages indicate temporary errors, we revealed messages with specific details that indicate permanent affect or no effect to the returned metadata records.