\documentclass{elsart}
% Use the option doublespacing or reviewcopy to obtain double line spacing
% \documentclass[doublespacing]{elsart}
\usepackage{natbib}
\usepackage{epsfig}
%\usepackage{subfigure}
% The lineno packages adds line numbers. Start line numbering with
% \begin{linenumbers}, end it with \end{linenumbers}. Or switch it on
% for the whole article with \linenumbers.
% \usepackage{lineno}

\begin{document}

\begin{frontmatter}
\title{Automatic Evaluation of  Digital Libraries with 5SQual}
\author[ufmg]{B{\'a}rbara L. Moreira}
%\ead{barbara@dcc.ufmg.br}
\author[ufmg]{Marcos Andr{\'e} Gon{\c c}alves}
%\ead{mgoncalv@dcc.ufmg.br}
\author[ufmg]{Alberto H. F. Laender}
%\ead{laender@dcc.ufmg.br}
\author[vt]{Edward A. Fox}
%\ead{fox@vt.edu}
\address[ufmg]{Department of Computer Science, Federal University of Minas Gerais, 31270-901 Belo Horizonte MG Brazil}
\address[vt]{Department of Computer Science, Virginia Tech, Blacksburg, VA 24061, USA}

\begin{abstract}
Digital libraries (DLs) are complex information systems which can present changes in their structure, content, and services. These complexities and dynamics make system maintenance a non-trivial task, since it requires periodical evaluation of the different DL components. Generally, these evaluations are customized per system and are performed only when problems occur and administrator intervention is required. This work aims to change the situation. We present 5SQual, a tool which provides ways to perform automatic and configurable evaluations of some of the most important DL components, among them, digital objects, metadata, and services. The tool implements diverse numeric indicators that are associated with eight quality dimensions described in the 5S quality model. Its generic architecture was developed to be applicable to various DLs and scenarios. In sum, the main contributions of this work include: (i) the design and implementation of 5SQual, a tool that validates a theoretical DL quality model; (ii) the demonstration of the appplicabilty of the tool in several usage scenarios; and (iii) the evaluation (with usability specialists) of its graphical interface, specially designed to guide the configuration of 5SQual evaluations. We also present the results of interviews conducted with administrators of real DLs regarding their expectations and opinions about 5SQual.
\end{abstract}

\begin{keyword}
Digital Libraries, Quality Assessment, 5S, 5SQual
\end{keyword}

\end{frontmatter}

\section{Introduction} \label{sec:intro}

Digital libraries (DLs) are complex and advanced information systems that offer information through content and services designed for specific communities of users. Once the importance and applicability of this type of information system was definitely established, questions about utility, usability, and cost of DLs started to arise and greater attention was given to their evaluation.  To define what makes a DL a system of good quality can be difficult and hard to summarize, since it depends on which of the many aspects of a DL are being considered. As it has been pointed out in \citep{fuhr:01}, evaluators interested in DLs have disparate views of these systems and, as a consequence, focus on different aspects that are relevant to their specific point of view. Digital libraries also may present many differences when compared or analyzed over time. The available content can grow in size and diversity. The provided services may exhibit changes in their usage patterns, their internal organization may evolve, etc. However, in practice, most DL evaluations occur only when a problem or situation arises that requires urgent intervention. Thus, in order to improve development and to promote maintenance of such dynamic systems, periodic and recurrent quality assessments of DL components should be performed.

%Those evaluations are usually very specific, depending on the %particularities of each system. 

%, which in the case of DLs is generally very expensive and time-%consuming \citep{suleman:02},

%to promote for example, ``prevention of errors'' instead of ``immediate intervention''

%Bibliotecas digitais apresentam muitas diferenças quando comparadas %entre si ou quando analisadas ao longo do tempo. O tipo de conteúdo %disponibilizado pode aumentar e variar, os serviços providos %apresentam mudanças de padrões de uso, a organização se modifica, %etc. Além de dinâmicas e diversas, bibliotecas digitais são sistemas %cuja construção quase sempre resulta de projetos com intensivos %%ciclos de desenho, implementação e teste, de acordo com \cite%{suleman:02}. Na realidade atual, a maioria das avaliações de %bibliotecas digitais, na prática, é muito específica para cada %sistema e ocorre devido a um problema ou situação que requer %intervenção urgente da administração. Para que os ciclos de %construção de uma biblioteca digital sejam melhor sucedidos, e a %manutenção de um  sistema tão dinâmico ocorra no esquema de %``prevenção de erros'' ao invés de ``intervenção imediata'', %verificações de qualidade dos seus diversos aspectos deveriam ser %realizadas recorrente e periodicamente.

With this goal in mind, we designed, implemented, and evaluated 5SQual, a tool intended for \textit{automatic quantitative evaluation} of some of the most important components of a digital library, namely, digital objects, metadata, and services. 5SQual is grounded in a formal quality model for digital libraries \citep{goncalves:07}. The tool helps to manage and maintain digital libraries through automatic and recurrent evaluations that can diagnose problems and suggest possible improvements in the system, as well as demonstrate its evolution over time. Due to the complexity, heterogeneity, and diversity of DLs in terms of content and services, the tool has been designed to be flexible enough to be used with many different systems.

% In this sense, 5SQual is an important asset for digital librarians. 

The potential applicability and usefulness of the tool was tested by employing it for the evaluation of real DLs, such as \textit{Virginia Tech's Digital Library of Electronic Theses and Dissertations} (VT-ETD)\footnote{http://scholar.lib.vt.edu/theses/} and \textit{The Brazilian Digital Library of Computing} (BDBComp)\footnote{http://www.lbd.dcc.ufmg.br/bdbcomp/}. These evaluations generated information that, according to interviewed administrators, can be very useful to improve and maintain a DL. The tool and its evaluation serve also as a validation of the theoretical quality model for DLs presented in \citep{goncalves:07} and makes it possible for DL administrators and digital librarians to apply the model in real settings. We also performed a usability study of the 5SQual interface with usability specialists and conducted interviews with potential users (DL administrators). 


%A ferramenta 5SQual foi implementada objetivando a avaliação \textbf
%{automática} e \textbf{quantitativa} de componentes importantes de uma biblioteca digital, a saber, objetos digitais, metadados e serviços, according to the 5S quality model proposed in \citep{goncalves:07}. 5SQual visa auxiliar à administração e manutenção de bibliotecas digitais, através de avaliações automáticas e recorrentes que possam diagnosticar problemas e possíveis melhorias no sistema, assim como demonstrar a sua evolução ao longo do tempo.


In sum, the main contributions of this work are: (i) the description of  a tool that implements a quality model for  automatic quality assessment of digital library components (digital objects, metadata specifciations, and services); (ii) its application to real-world digital libraries, demonstrating its real utility and usefulness; (iii) the   evaluation (with usability specialists) of its graphical interface specially designed to guide the configuration of 5SQual evaluations; and (iv) an analysis of the results of interviews , conducted with administrators of real DLs, discussing expectations regarding 5SQual. The results of both evaluations and the opinions expressed were in general very positive.


This article is organized as follows. Section \ref{sec:context} provides background and context necessary to understand the remainder of the article. Section \ref{sec:overview} presents the quality dimensions covered by 5SQual, along with its architecture, potential benefits, and step-by-step functioning. Section \ref{sec:evaluation} deals with some examples of evaluations performed upon real DLs using 5SQual. Section \ref{sec:Eval} shows the results of a usability evaluation of the 5SQual interface and discusses the expectations and opinions of potential users about the tool, gathered through interviews. Section \ref{sec:conclusions}, which concludes the article, is followed by two appendices: Appendix A, showing how people can configure 5SQual for an evaluation; and Appendix B, showing examples of the use of some metrics.

\section{Background and Context} \label{sec:context}

Works such as \citep{goncalves:03} and \citep{klas:06} present standards for DL log formats with the goal of recording data for the evaluation of DLs. Such formats are very important for tools such as 5SQual. In \citep{goncalves:03}, an XMLLog format is described that captures detailed information about system behavior and access to its services, storing data that indicate critical aspects about user interactions with the DL, thus providing valuable information for system evaluation. \citep{klas:06} builds on that work and proposes a multilevel record scheme for DL logging.

%Trabalhos como os de \citep{goncalves:03} e \citep{klas:06} apresentam padrões para o desenvolvimento de formatos para arquivos de \textit{log} destinados a registrar dados para avaliações de bibliotecas digitais e são de grande importância para uma ferramenta como a 5SQual que se propõe avaliar diversas bibliotecas digitais. No trabalho de \citep{goncalves:03}, é descrito um formato para um \textit{log} XML que captura informações detalhadas sobre o comportamento de sistemas e o acesso aos seus serviços, armazenando dados que indicam aspectos críticos da interação dos usuários com a biblioteca digital e, ao mesmo tempo, provêem dados valiosos para uma análise do comportamento do sistema. \citep{klas:06} constróem um esquema de registro multi-nível que tem como base o trabalho de \citep{goncalves:03}.

Different approaches to evaluate the success of a DL have been studied (e.g., \citep{saracevic:00}, \citep{saracevic:00a}, \citep{fuhr:01}, \citep{saracevic:04}, \citep{tsakonas:04}, \citep{kyrillidou:05}, \citep{rao:06}, and \citep{fuhr:07}) involving users, collections, and systems, aimed at identifying generalizable metrics or context specific methods. But the literature that reports evaluations with actual data is not substantial. It seems that evaluation theorists and practitioners do not communicate well, as noticed in \citep{saracevic:04}. 5SQual addresses that challenge; it is a tool that implements and follows a theoretical quality model for DLs, and that can help administrators in the evaluation of real DLs. 

%5SQual surge como uma exceção a esta situação, consistindo em uma %ferramenta que implementa e segue um modelo teórico de qualidade, o %modelo de qualidade baseado no arcabouço 5S.

The 5S framework \citep{goncalvesetal:04} consists of a mathematical formalism that allows to precisely define a minimal digital library. According to 5S, a (minimal) digital library consists of a repository (containing collections of digital objects), catalogs (containing metadata specifications), services, and a society of users. 5S stands for Streams (related to the content), Structures (related to organization), Spaces (useful for retrieval and presentation), Scenarios (that describe the behavior of services), and Societies. 5S has been shown to be applicable to other types of DLs beyond the minimal \citep{gorton:07,YiMa07a,Murthi07,YiMa07b}.


%O 5S, descrito em \citep{goncalves:04} e \citep{goncalvesetal:04}, consiste em um arcabouço embasado em formalismo matemático que inclui uma ontologia para definir conceitos, relacionamentos e regras fundamentais que governam o domínio de bibliotecas digitais. De acordo com o modelo 5S, uma biblioteca digital consiste em um \textbf{repositório} (formado por \textbf{coleções} de \textbf{objetos digitais}), \textbf{catálogos} (constituídos por \textbf{especificações de metadados}), \textbf{serviços} e uma \textbf{sociedade} de usuários. A sigla 5S se refere a:

%\begin{itemize}
%\item \textit{Streams} and \textit{Structures} for the constrution of digital objects and collections para a construção de \textbf{objetos digitais}:
%\begin{itemize}
%\item The streams descrevem as propriedades do conteúdo da biblioteca digital, tais como a codificação e idioma para materiais textuais ou formas específicas de dados multimídia. Exemplos: texto, vídeo, áudio e imagem.
%\item The structures definem a organização do conteúdo em uma biblioteca digital. Por exemplo: coleção, catálogo, hipertexto, metadados.
%\end{itemize}
%
%\item \textit{Spaces} para a descrição de \textbf{coleções} e suas interrelações. Os espaços especificam a apresentação lógica dos vários componentes de uma biblioteca digital. Por exemplo: espaços probabilísticos, vetoriais e topológicos.

%\item \textit{Scenarios} para definir como \textbf{serviços} se comportam e mudam o estado do sistema. Exemplos: busca, navegação, recomendação.

%\item \textit{Societies} para interconectar papéis e atividades dentro de uma comunidade de usuários. As sociedades definem os atores e os serviços utilizados por eles. Exemplos: professores, aprendizes, administradores.
%\end{itemize}


In \citep{goncalves:07}, a formal model for the evaluation of digital libraries, built on top of 5S, is described. This model defines numerical indicators associated with 22 quality dimensions used to quantitatively evaluate the basic components of a digital library: digital object, metadata specification, collection, catalog, repository, and services. 

%Em \citep{goncalves:07}, um modelo formal para avaliação de bibliotecas digitais, construído sobre o arcabouço 5S, é descrito. Esse modelo define indicadores numéricos associados a 22 dimensões de qualidade para avaliar quantitativamente os componentes básicos de uma biblioteca digital - de acordo com o arcabouço 5S: objeto digital, especificação de metadados, coleção, catálogo, repositório e serviços.

The 5SQual tool implements the quantitative evaluation of eight of the quality dimensions proposed in the 5S quality model, allowing automatic evaluation of digital libraries with regard to many of their most important components. In the following section, we present an overview of 5SQual, describing its underlying quality model, architecture, operation, and interaction mode.


%A ferramenta 5SQual implementa a avaliação quantitativa de 8 das dimensões de qualidade propostas no modelo de qualidade 5S, viabilizando a avaliação automática de aspectos de objetos digitais, metadados e serviços de bibliotecas digitais. In the following section, it will be shown an overview of 5SQual, describing its underlying quality model, architecture, operation and interaction mode.

\section{5SQual Overview }  \label{sec:overview}

\subsection{The Quality Model} \label{sec:model}

The construction of 5SQual was initially based on the implementation of a subset of the quality dimensions presented in \citep{goncalves:07}. These dimensions had been chosen for a first implementation of the tool because the respective numeric indicators are user independent and objective enough to allow an automatic evaluation. Other dimensions and numeric indicators can be added to the tool in the future. In the following, we describe the dimensions included in this subset, as well as the numerical indicators used to assess them quantitatively. 

%A construção da ferramenta 5SQual inicialmente se baseou na implementação de um subconjunto das dimensões do modelo de qualidade. These dimensions have been chosen for a first implementation of 5SQual because the respective numeric indicators are user independent and objective enough to allow an automatic evaluation. Other dimensions and numeric indicators can be added to the tool in the future. Este subconjunto foi definido como um primeiro passo em relação à avaliação automática de bibliotecas digitais, possibilitando a avaliação de objetos digitais, especificações de metadados e serviços de um sistema. A seguir são descritas as dimensões que este subconjunto inclui, assim como os indicadores numéricos utilizados para avaliá-las quantitativamente.

\begin{itemize}
\item For the evaluation of digital objects
\begin{itemize}
\item \textit{Acessibility}: reflects the rights of a certain community of users to access (parts of) the digital objects of a digital library. A digital object is accessible to an actor  $x$ if: (1) the object belongs to a collection of the DL; (2) it is possible to retrieve the object from the DL repository; and (3) there is not a restrictive access policy to the object regarding the particular community to which $x$ belongs. The numerical indicator of accessibility may be obtained by calculating the percentage of the digital object's streams that  actor $x$ may access. For example, consider an electronic thesis with  five chapters, two of which are accessible by $x$; this thesis would have a accessibility value of 0.4.  

%O indicador numérico para acessibilidade pode ser obtido pela porcentagem de sequências do objeto digital que o ator X tem permissão para acessar. Por exemplo: um documento constituído por cinco capítulos, dos quais dois são acessíveis para X, apresenta acessibilidade igual 0,4.

\item \textit{Significance}: indicates the importance of digital objects regarding a specific factor, such as number of downloads, number of citations, etc. The numerical indicator gives the number of occurrences of this factor, for example, if the factor is the number of citations, a document with seven citations would have a significance value of seven.  

%\item \textit{Significância}: indica a importância dos objetos digitais em relação a determinado fator, como o número de acessos que o objeto digital recebeu, o número de vezes que foi citado, dentre outros. O indicador numérico para significância é justamente o número de ocorrências do fator em questão. Por exemplo, se o fator é o número de citações, um documento que recebeu 7 citações apresenta significância igual a 7.

\item \textit{Similarity}: estimates how related or close two digital objects are. This dimension may be evaluated based on the objects' content or based on citation relationships among objects. High values may identify replicas or quasi-replicas (indicating plagiarism).  Regarding content, the numerical indicator used by the current version of 5SQual is based on the ``bag-of-words'' measure that reflects the percentage of terms shared by two documents. Regarding citation relationships, 5SQual can calculate similarity based on bibliographic coupling \citep{kessler:63} and co-citation \citep{small:73}. 
%\item \textit{Similaridade}: estima o quão relacionados ou próximos são dois objetos digitais. Pode ser avaliada em relação ao conteúdo dos objetos ou em relação ao relacionamento de citações existente entre os objetos digitais. Considerando conteúdo, o indicador numérico utilizado por esta versão da 5SQual é o resultante do algoritmo ``bag-of-words'', que indica a porcentagem dos termos de um documento que são comuns aos termos de um segundo documento. Considerando o relacionamento de citações, 5SQual pode avaliar similaridade através dos valores obtidos com as métricas Acoplamento Bibliográfico \citep{kessler:63} e Co-citação \citep{small:73}. A aplicação da avaliação desta dimensão pode ser direcionada para a detecção de cópias indevidas dentre os documentos do sistema ou para a formação de grupos de objetos digitais relacionados (objetos que citam ou são citados por um mesmo conjunto de objetos apresentam fortes indicadores de que são relacionados);



\item \textit{Timeliness}: indicates how recent are the digital objects of the DL. This dimension may reflect dates such as the creation, publication, or storage dates, or even the date of the last citation, update, or access of the digital object. Evaluation regarding last citation or access is directly related to recentness of the content of the information carried by the object. The numeric indicator for this dimension corresponds to the difference between the chosen date and some reference date or the current date (i.e., the date of the evaluation).  
\end{itemize}

%\item \textit{Timeliness}: indica quão atuais são os objetos digitais na biblioteca digital. Esta Timeliness pode ser avaliada em relação a datas como a data de criação, de publicação, ou de armazenamento do objeto, ou ainda, data da última citação ou data de último acesso recebido pelo objeto digital. A avaliação em relação a última data de citação ou acesso recebidos indica mais do que a Timeliness do objeto em questão, indica a Timeliness da informação que o objeto carrega, mostrando se este ainda é acessado e utilizado recentemente. O indicador numérico para Timeliness corresponde ao tempo decorrido entre a data escolhida como data de referência e a data corrente, do dia da avaliação.
%\end{itemize}


\item For the evaluation of metadata specifications
\begin{itemize}
\item \textit{Completeness}: reflects how many of the attributes specified in a metadata standard have their values defined in a metadata specification. The numerical indicator for this dimension is defined as the percentage of the existing attributes in the standard for which a defined value exists in the metadata specification.


%\item Para avaliação de Especificações de Metadados:
%\begin{itemize}
%\item \textit{Completeza}: reflete quantos dos atributos especificados no padrão de metadados seguido (o que constitui o número ideal de atributos) possuem um valor definido na especificação de metadados. O indicador numérico desta dimensão é dado justamente pela porcentagem dos atributos existentes no padrão que possuem um valor associado na especificação de metadados em questão.

\item \textit{Conformance}: indicates whether the attributes and their respective values in a metadata specification follow the rules defined in a given metadata standard. The rules specify the maximum and minimum number of values an attribute may have in a metadata specification. The numerical indicator used by 5SQual calculates the percentage of the attributes of the metadata specification conformant to these rules. It is important to notice that a missing attribute may be considered conformant if the attribute is  not defined as mandatory in the standard.
\end{itemize}

%\item \textit{Conformidade}: indica se os atributos e seus valores nas especificações de metadados seguem as regras definidas pelo padrão de metadados. As regras definidas no padrão de metadados indicam o domínio e o número máximo e mínimo de vezes que um valor pode ser associado a um determinado atributo na especificação de metadados. O indicador numérico utilizado pela 5SQual consiste na porcentagem de atributos definidos pelo padrão seguido que estão consistentes com as regras de máximo e mínimo número de vezes que o atributo deve receber um valor associado na especificação. É importante destacar que mesmo ausente na especificação, o atributo ainda pode ser considerado em conformidade, contanto que não seja obrigatório de acordo com as regras do padrão seguido.
%\end{itemize}


\item For the evaluation of services
\begin{itemize}
\item \textit{Efficiency}: indicates the speed of execution of services. Its numerical indicator is given by the difference between the response and request time of the execution of a service.

%\item Para avaliação de Serviços:
%\begin{itemize}
%\item \textit{Eficiência}: indica a velocidade de execução dos serviços. %Seu indicador numérico é dado pela diferença entre os tempos de resposta %e requisição da execução de um serviço.

\item \textit{Confidence}: indicates the probability of success (no failures) of the  execution of a particular service. A failure happens when the service does not execute as expected. The numerical indicator calculates the proportion of successful executions over the total number of requests for the service.

\end{itemize}
\end{itemize}


%\item \textit{Confiança}: indica se as execuções de um determinado serviço provavelmente irão obter os resultados esperados, sem falhas. Uma falha se caracteriza quando o que deveria ocorrer não ocorre, ou não ocorre como o esperado. O indicador numérico de confiança define a proporção entre o número de execuções bem-sucedidas e o número total de execuções requisitadas para um serviço.
%\end{itemize}
%\end{itemize}


Table \ref{indicators} summarizes the discussion by showing, for each quality dimension implemented in 5SQual, its respective numerical indicators.
\begin{table}[htbp]
\small
\caption{Quality Dimensions and Respective Numerical Indicators}
\label{indicators}
\begin{center}
\begin{tabular}{||p{7cm}|p{7cm}||}
  \hline
  \textbf{Quality Dimension}&\textbf{Numerical Indicator}\\
  \hline\hline
  Accessibility & Percentage of the digital object's streams
that an actor $x$ of some community $y$ may access.\\\hline
  Significance&Number of occurences of some factors such as downloads, citations, etc.\\\hline     Similarity& Similarity measure (e.g., bag-of-words)\\\hline
  Timeliness& Difference between some chosen date and some reference date or the current date (i.e., the date of the evaluation).\\\hline
  Completeness& Percentage of the existing attributes in a metadata standard for which a defined value exists in the metadata specification\\\hline
  Conformance & Percentage of the attributes of the metadata specification conformant to the metadata standard rules\\\hline
  Efficiency& Difference between the response and request time
of the execution of a service\\\hline
  Confidence & Proportion of successful executions over the total number of requests for a service\\\hline
\end{tabular}
\end{center}
\end{table}

\subsection{The 5SQual Architecture}

The 5SQual architecture was designed with the goal of allowing the tool to be used by a large number of diverse DLs with different goals (e.g., complete periodical evaluations, diagnosis of problems). Since these systems make the information necessary for evaluation available in many distinct ways, the architecture tries to be very flexible in several aspects including:

%A arquitetura da ferramenta 5SQual foi planejada com o objetivo de possibilitar a aplicabilidade da ferramenta a diversas bibliotecas digitais. Como estes sistemas disponibilizam as informações necessárias para a avaliação da 5SQual nas maneiras mais variadas, a arquitetura aqui apresentada busca proporcionar flexibilidade em diferentes aspectos:

\begin{itemize}
\item Flexibility in \textit{data collection}. Data for evaluation may be gathered from Web pages, from the DL repositories via the Open Archives Protocol for Metadata Harvesting \citep{OAI} or from the local filesystem;


%\item Flexibilidade para Coleta de Dados: Os dados para a avaliação podem ser buscados em páginas Web, em repositórios através do protocolo OAI \citep{OAI} e no sistema de arquivos local;

\item Flexibility in \textit{data extraction}. Since the log files of a DL may use disparate formatting rules, the 5SQual architecture supports internal recognizers that come with the tool,  for example, for the XMLLog format \citep{goncalves:03}, or specific external recognizers for a particular format.

%\item Flexibilidade para Extração de Dados: como os arquivos de \textit{log} geralmente apresentam formatações diferentes de uma biblioteca para outra, a arquitetura da 5SQual permite que o usuário utilize os reconhecedores internos da ferramenta - para o formato XMLLog \citep{goncalves:03}, por exemplo - ou indique programas reconhecedores externos à 5SQual para serem utilizados na extração dos dados. 

\item Flexibility in \textit{evaluation}. The tool allows the user to specify which set of dimensions she wants to evaluate. 

%Evaluations may also be repeated 

%\item Flexibilidade de Avaliação: a ferramenta permite que o usuário defina qual o conjunto de dimensões que deseja avaliar.

\item Flexibility in \textit{utilization}. 5SQual receives as input an XML file with the  parameters necessary for retrieving and extracting the data for the calculation of the dimensions defined in an evaluation. To facilitate the construction of this input file, a special graphical user interface was implemented to guide the user throughout this configuration process in order to generate the file and execute the evaluation. It also is possible to generate only the configuration file and execute the evaluation later, via interface or via command line. The saved input file with all the configurations may be re-used in posterior evaluations. 

%\item Flexibilidade de Utilização: 5SQual recebe como entrada um arquivo XML contendo os parâmetros necessários para buscar e extrair os dados para o cálculo das dimensões definidas em uma avaliação. Para facilitar a construção deste arquivo de entrada, foi implementada uma interface capaz de guiar o usuário por essa configuração, gerar o arquivo e chamar a execução da avaliação. Também é possível somente gerar o arquivo e deixar para executar a avaliação mais tarde, pela interface ou pela linha de comando.


\end{itemize}

Figure \ref{fig:architecture} shows the 5SQual architecture. The necessary information for the evaluation resides in the DL and should be retrieved through the DL application layer (e.g., through an OAI interface). The 5SQual architecture is organized as follows:

\begin{figure}
\centering
\epsfig{file = imgs/5SQualArchitecture, width=4in}
\caption{5SQual Architecture}
\label{fig:architecture}
\end{figure}

\begin{itemize}
\item Processing Layer - In this layer we have three modules: the retrieval module, the extraction module, and the calculation module.
\begin{itemize}
\item Retrieval module: This module is responsible for obtaining the necessary information for evaluation through the Web or in the local file system. It collects log files that record the behavior of the DL services, its digital objects, or metadata with information about these objects. For retrieving metadata on the Web, it  uses the OAI interface. Digital objects and logs can be retrieved from the Web or from the local file system, through previously indicated file paths.

\item Extraction module: 5SQual uses parsers that have been specified by the user or the ones that already come with the tool. These parsers extract data from the collected files and convert them to the 5SQual standard formats that describe the necessary information for each dimension. The set of built-in parsers includes content parsers (e.g., for PDF and PS files), specific metadata format parsers (e.g., for Dublin Core and RFC1807 formats), and specific log format parsers (e.g., for the XMLLog format \citep{goncalves:03}).
 
\item The Calculation module: In this module, 5SQual implements a set of numeric indicators for each quality dimension.
 
\end{itemize}
 
\item The Interface Layer - The configuration module is responsible for storing the parameters defined for the evaluation. According to the choices made by the user, 5SQual generates XML reports and charts for each dimension, considering the evaluation results.
\end{itemize}

\subsection{Potential Benefits of 5SQual}
Here, we present a number of potential benefits that 5SQual could bring to a DL setting in some specific usage scenarios. Some of these scenarios are actually explored in the next section, thus demonstrating that this potential can be effectivelly realized in a  real world digital libraries. Some of these potential benefits and the corresponding scenarios are as follows: 

\begin{itemize}

\item  Determining the percentage of digital objects completely, partially, or not accessible at all to a number of people in the DL community 
\begin{itemize}
\item \textit{Scenario of use}: Assessing the effectiveness of the implementation of an Electronic Theses and Dissertations Program in a University
\end{itemize}

\item  Determining the average significance of the  digital objects of a DL
\begin{itemize}
\item \textit{Scenario of use}: Assessing the impact of a digital library
\end{itemize}

\item Determining the similarity of pairs of digital objects
\begin{itemize}
\item \textit{Scenario of use}: Finding near-duplicates, replicas, or copies in a digital library created by harvesting methods or detecting plagiarism. 
\end{itemize}


\item Determining the average age of the digital objects of the DL or the average citation period
\begin{itemize}
\item \textit{Scenario of use}: Discovering the rate of update of a DL or the impact of its digital objects
\end{itemize}

\item Determining the level of detail contained in the metadata specifications of a DL
\begin{itemize}
\item \textit{Scenario of use}: Comparing the depth of description of digital objects in several DLs with similar content
\end{itemize}


\item Determining the level of agreement of metadata specifications with respect to a given standard
\begin{itemize}
\item \textit{Scenario of use}: Deduplication of records in a DL constructed by harvesting. If the values of the fields do not follow some patterns, finding near-duplicates or copies would be harder.
\end{itemize}

\item Determing the efficiency and robustness of the DL services
\begin{itemize}
\item \textit{Scenario of use}: A component in an analysis of the perceived quality of services of the DL by users of its target community
\end{itemize}

\end{itemize}

%\begin{table}[htbp]
%\small
%\caption{Potential Benefits of the Use of 5SQual and Scenarios of %%Usage}
%\label{tab:benefits}
%\begin{center}
%\begin{tabular}{||p{7cm}|p{7cm}||}
%  \hline
%  \textbf{Potential Benefit}&\textbf{Scenario of Usage}\\
%  \hline\hline
%  Determining the percentage of digital objects completely, %%%partially, or not accessible at all to a number of people in the DL community & Assessing the effectiveness of the implementation of an %Electronic Theses and Dissertations Program in a University\\\hline
%  Determining the average significance of the  digital objects of a %DL&Assessing the impact of a digital library\\\hline
%  Determining the similarity of pairs of digital objects&Finding %near-duplicates, replicas, or copies in a digital library created by %harvesting methods or detecting plagiarism. \\\hline
%  Determining the average age of the digital objects of the DL or %the average citation period&Discovering the rate of update of a DL or %the impact of its digital objects\\\hline
%  Determining the level of detail contained in the metadata %specifications of a DL&Comparing the depth of description of digital %objects in several DLs with similar content\\\hline
%  Determining the level of agreement of metadata specifications with %respect to a given standard&Deduplication of records in a DL %constructed by harvesting. If the values of the fields do not follow %some patterns, finding near-duplicates or copies would be harder.%%\\\hline
%  Determing the efficiency and robustness of the DL services&A %component in an analysis of the perceived quality of services of the %DL by users of its target community\\
%  \hline
%\end{tabular}
%\end{center}
%\end{table}

These are just some possibilities among innumerous due to the already mentioned flexibilities of the tool.

\subsection{Step-by-Step functioning of 5SQUAL}
In order to perform an evaluation as suggest above, the following steps should be followed by a DL administrator:
\begin{enumerate}
\item Identify the evaluation (so that it can be reused later)
\item Choose the quality dimensions to be evaluated
\item Specify the necessary resources to evaluate each selected dimension
\item Specify parameters about how to extract the data from the indicated resources and how to calculate the indicators for each selected dimension
\item Define the format of the output, including graphs and types of report to be generated
\end{enumerate}

These steps and the corresponding user interfaces of the tool are illustrated in Appendix A of this article. Table \ref{resources} shows the respective resources and other parameters necessary for calculating each numerical indicator for every defined quality dimension. For example,  resources necessary to calculate accessibility include pointers for collecting the data, the metadata standard to be used, indication of the rights field and indication of possible values for this field and respective weights, while for quality dimensions related to time (e.g., \textit{Timeliness} and \textit{Efficiency}) we need information about the datestamp standard to be used.  This is interesting to demonstrate how we materialize each indicator, allowing us to instantiate the quality model in a real usable tool. 

\begin{footnotesize}
\begin{table}[htbp]
\footnotesize{
\caption{Resources and Parameters for the Calculation of the Numerical Indicators for the Dimensions.}
\label{resources}
\begin{center}
\begin{footnotesize}
\begin{tabular}{|p{0.85in}|p{2.4in}|p{3in}|}
  \hline

{\textbf{Dimension}} & {\textbf{Resources}} & {\textbf{Other Parameters}} \\ \hline 

Acessibility &
(1) URL and (2) metadata standard to be harvested through OAI-PMH, and (3) name of the metadata field providing information about access policies &
(4) Values that the access policies may take and (5) weights associated with each one of them, according to the defined access. Example: \textit{the rights field} may take the value \textit{unrestricted} which means complete access, having weight of 1 \\ \hline

Significance &
Path (local or remote) for the (1) files that contain information about the chosen significance factors (number of accesses, downloads, citations, etc) &
Indication of the  (2) recognizer program for the file chosen as resource. This program should be able to convert the input file to the standard 5SQual format for \textit{Significance}. \\ \hline
 
Similarity by Content &
List of pointers (local or remote) for (1) documents that will be compared &
Path or identifier to the (2) reference documents  (local or remote) and (3) list of \textit{stop words} (optional) \\  \hline

Similarity by citations
& Path (local or remote) for (1) file containing information about citations &
Path or identifier to the  (2) reference document, and (3) the recognizer program for the resource file. This program should be able to convert the input file to the standard 5SQual format for 
 \textit{Similarity by citations}. \\  \hline

Timeliness with File &
Path to (1) file containing information about reference dates  (creation date, last citation, etc) &
Indication of the (2) recognizer program of the resource file. This program should be able to convert the input file to the standard 5SQual format for 
 \textit{Timeliness}.  Moreover , it is necessary to provide the (3) timestamp standard used by the dates in the resource file (e.g., MM-dd-yyyy and yyyy-mm-dd).\\ \hline

Timeliness with Metadata &
(1) URL and (2) metadata standard to harvest through OAI-PMH and the name of the (3) metadata field where to search for reference dates (e.g., \textit{date} field) &
(4) The datestamp standard followed by the dates to be analysed (e.g., MM-dd-yyyy and  yyyy-mm-dd).\\ \hline


Completeness &
(1) URL and (2) metadata standard to harvest through OAI-PMH  &
There are no extra parameters.\\ \hline

Conformance &
(1) URL and (2) metadata standard to harvest through OAI-PMH  &
There are no extra parameters.\\ \hline

Efficiency &
(1) Path (local or remote) for file containing information about the time execution of services.&
Indication of the (2) recognizer program of the resource file. This program should be able to convert the input file to the standard 5SQual format for 
 \textit{Efficiency}. (3) The standard used for time (e.g., dd/MM/yy-HH:mm:ss).\\ \hline


Confidence &
(1) Path (local or remote) for file containing information about the final execution status of services &
(2) Indication of the recognizer program of the resource file. This program should be able to convert the input file to the standard 5SQual format for  \textit{Confidence}.\\ \hline

\end{tabular}
\end{footnotesize}
\end{center}
}
\end{table}
\end{footnotesize}

%\clearpage


\section{DL Evaluations using 5SQual} \label{sec:evaluation}

To show the functioning of 5SQual, we have performed a set of evaluations that cover all the dimensions implemented by the tool. For this, we used three different DLs, with distinct characteristics. We defined the set of dimensions to be evaluated on each DL, according to the availability of the resources required for each dimension to be evaluated. Below, we describe the three DLs and the chosen dimensions for each DL.

\begin{itemize}
\item Virginia Tech's Digital Library of Electronic Theses and Dissertations (VT-ETD)\footnote{http://scholar.lib.vt.edu/theses/}, a well-established DL that provides access to full-text documents with different levels of access rights. For this DL, we obtained metadata through the OAI-PMH, therefore allowing to evaluate \textit{Completeness} and \textit{Conformance}. Since the VT-ETD metadata provides information about access policies and creation date, we also evaluated \textit{Acessibility} and \textit{Timeliness}. 

%Finally, since the full-text of several electronic theses and %dissertations was  also available, we evaluated \textit{Similarity by %content} for a subset of these documents. 


%\begin{enumerate}
%\item Os metadados do catálogo estão disponíveis através do protocolo %OAI-PMH, portanto foi possível avaliar \textit{Completeza} e %\textit{Conformidade}. Como os metadados provêm informações sobre %política de acesso e data de criação dos objetos na biblioteca digital, %também foram avaliadas \textit{Acessibilidade} e \textit{Timeliness}.
%\item Como vários textos de teses e dissertações estão disponíveis, %avaliou-se \textit{Similaridade} por conteúdo para um subconjunto desses %documentos. 
%\end{enumerate}

\item The Brazilian Digital Library of Computing (BDBComp)\footnote{http://www.lbd.dcc.ufmg.br/bdbcomp/} \citep{laender:04}, a DL whose catalog has been built from several distinct sources. In this DL, we had easy access to its log files, allowing us to evaluate the \textit{Efficiency} and \textit{Reliability} of its services.

%Em relação às informações disponibilizadas e às dimensões avaliadas, %para este trabalho foi concedido fácil acesso aos arquivos de %\textit{logs} de serviços desta biblioteca, promovendo a avaliação das %dimensões \textit{Eficiência} e \textit{Confiança}.

\item The ACM 2002 collection (ACM) with 94,818 metadata records and their internal citations relationships. This collection has been obtained in connection with the CITIDEL project \citep{knox:02}. In this collection we evaluated the dimensions based on citation relationships (\textit{Similarity by citations}), \textit{Significance}, and  \textit{Timeliness} based on date of the last citation.

%, since the publishing dates for these documents are available in this %collection. 

%Para esta coleção foram avaliadas as dimensões baseadas no relacionamento de citações entre os objetos digitais, \textit{Similaridade} por citações e \textit{Significância}, além da dimensão \textit{Timeliness} baseada na data da última citação, já que as datas dos documentos publicados também são disponibilizadas pela coleção.
\end{itemize}

Following, we present the obtained results for each evaluated dimension, including charts and some data extracted from the evaluation report. The produced report shows the calculated indicators of the evaluated dimensions.  An excerpt of the report is shown in Figure \ref{fig:report}. The report gives the \textit{evaluation date} ($<$date$>$ field), the \textit{name of the DL} ($<$dlName$>$ field), the evaluations of the \textit{selected dimensions} ($<$Dimension$>$ field, in the particular case \textit{Efficiency}), and all the numerical indicators chosen in the configuration ($<$Indicator$>$ field, in the particular case \textit{Response Time in seconds}). For each pair $[$dimension, numerical indicator$]$, the report provides: \textit{the number of evaluated items} ($<$numItens$>$ field), \textit{the mean value} and the \textit{standard deviation} considering all the evaluated items ($<$avgValue$>$ and $<$stdDeviation$>$ fields), as well as the maximum and minimum values ($<$maxValue$>$ and $<$minValue$>$ fields). All the identifiers of the evaluated items are listed in the report in decreasing order of the numerical indicator value ($<$evaluation$>$ field, with corresponding $<$itemId$>$ field, for the evaluated item; the value in the $<$evaluation$>$ field corresponds to the value of the numerical indicator.). This helps to identify outliers or exceptions, for example.

%Nele estão registrados a \textbf{data da avaliação}, o \textbf{nome da %biblioteca} e as avaliações das \textbf{dimensões} selecionadas, para %todos indicadores numéricos escolhidos na configuração. Para cada par %\{dimensão, indicador numérico\}, são informados: o \textbf{número de %itens} avaliados, o \textbf{valor médio} e \textbf{desvio padrão} obtido %do indicador numérico considerando todos os itens avaliados, \textbf{o %maior e o menor valor} obtido para o indicador de qualidade. Todos os %identificadores dos itens que foram avaliados por um mesmo valor do %indicador numérico são listados juntos no relatório, seguindo ordenação %decrescente dos valores dos indicadores numéricos. Esta organização %facilita a detecção de itens que não obedecem um comportamento normal %(as exceções). Um trecho de relatóro pode ser visto na Figura %\ref{fig:report}. Os relatórios não serão mostrados nas avaliações %seguintes pois, na maioria das vezes, são muito extensos. 


\begin{figure}[ht]
    \centering{\scalebox{0.8}{\includegraphics{imgs/5SQualReport}\par}}
    \caption{5SQual Report Excerpt}
    \label{fig:report}
\end{figure}

\subsection{VT-ETD Evaluation}\label{sec:avVT}

From the VT-ETD catalog\footnote{http://scholar.lib.vt.edu/theses/OAI2/}, we harvested 8,708 metadata records on January 9, 2007 for calculating the dimensions \textit{Accessibility}, \textit{Completeness}, \textit{Timeliness}, and \textit{Conformance}. 

%Besides these dimensions, the \textit{Similarity By Content} also was %evaluated using information about the full-text of the ETDs available on %the VT-ETD Web site.

\subsubsection*{Accessibility}

The VT-ETD metadata presents the \textit{rights} field, with information about the policy for accessing the digital objects from the DL. Objects can be restricted (available only to the VT community), unrestricted (public), or mixed (parts of it are public and other parts are restricted). For quantitative evaluation, we associated a value of accessibility to each one of these categories - unrestricted: 1, restricted: 0, and mixed: 0.5. To define these values, we considered the view of an actor that does not belong to the VT community.

The chart obtained from the 5SQual evaluation, shown in Figure \ref{fig:accChart}, presents the number of objects with restricted, unrestricted, and mixed access. From the corresponding XML report, it is possible to get the identifiers of the documents for each access category.
%, in the case the administrator wants to verify the behavior of a %specific subset of documents.

%caso a administração precise verificar o comportamento de um subconjunto %específico de objetos digitais.

\begin{figure}[ht]
    \centering{\scalebox{0.3}{\includegraphics{imgs/5SQual_Chart_Accessibility}\par}}
    \caption{VT-ETD - \textit{Accessibility Chart}}
    \label{fig:accChart}
\end{figure}

As we can see, almost 35\% of the ETDs have restricted access outside of the university environment, which may reveal an apprehension from some of the new graduates that free availability of the material may cause problems in future attempts for publishing the result of their research  as scientific papers or patents. Also, the small number of restricted ETDs (less than 2\%) may be due to a lack of knowledge by these graduates about the possibility of releasing only parts of the ETDs, an interesting mechanism that can at the same time protect part of the content while publicizing some of the results.
 
The results of this evaluation, besides revealing to the administrator the behavior of the users who ingest content in the DL, also may indicate alternatives to modify  behavior in case it is not desired. A strategy to increase the accessibility of this material would be to identify the restricted ETDs (using the evaluation report) and to present to their respective authors, and also to other potential authors,  the possibility of releasing only portions of their work through mixed access.

%Os resultados dessa avaliação, além de mostrarem para o administrador o %comportamento dos usuários que inserem conteúdo na biblioteca, podem %indicar alternativas para que esse comportamento seja modificado (no %caso de não ser o comportamento desejado). Uma estratégia para se obter %mais conteúdo público seria, por exemplo, identificar os trabalhos %restritos (através do relatório de avaliação) e divulgar para os seus %autores a possibilidade de disponibilizar apenas partes dos trabalhos %através do acesso misto. Além disso, divulgar isso para os futuros %autores também seria importante.

\subsubsection*{Timeliness}

The creation time of the digital objects was extracted by 5SQual from the \textit{date} field of the VT ETD metadata records to calculate their \textit{timeliness}, which in this case was measured in years, given by the difference between the current time and the obtained creation time.

Figure \ref{fig:timChart} presents the chart generated by 5SQual for \textit{timeliness}. It shows the number of items concentrated under each of the shown timeliness values that were calculated based on the current time (date of this evaluation was January 9, 2007). The y axis shows the number of objects that were created on a specific date, and the x axis determines the date when the objects were inserted in the collection.

\begin{figure}[ht]
   \centering{\scalebox{0.3}{\includegraphics{imgs/5SQual_Chart_Timeliness}\par}}
    \caption{VT-ETD - Timeliness Chart}
    \label{fig:timChart}
\end{figure}

From this chart, we can see that objects have been continuously created in this DL over the last 10 years and that many objects (almost 100) were inserted on the same date, approximately 1.3 year ago, when scanning of the backfile was speeding up. We also notice that, in the early days of this DL, there was a very stable insertion pattern over the years, which might indicate that the insertion of new objects into the collection was related to some academic events. However, in the last three years this pattern has changed, increasing not only the number of objects per insertion, but also the frequency in which these insertions take place.

From the corresponding XML report, it is possible to find more specific information such as the age of each object, and the average object age (4.37 years) and the standard deviation (2.99 years) of the whole collection. In addition, we can see that the oldest object (identified by oai:VTETD:etd-81197-16953) is 13.76 years old and that the newest one (identified by oai:VTETD:etd-12142006-164331) was created on the date of the evaluation. This reveals that VT-ETD is a DL that keeps its content very timely, what might be due to the fact that the submission of electronic dissertations is mandatory at Virginia Tech.

%The data obtained from this \textit{Timeliness} evaluation indicates %whether the evaluated DL is regularly updated (as in the case of VT-ETD) %and may reveal important fatcs such as the impact  

%Os dados obtidos a partir da avaliação de \textit{Timeliness} indicam se %a biblioteca digital avaliada é constantemente atualizada (como no caso %da VT-ETD) e podem revelar associações entre fatos, como a inserção de %nova tecnologia ou o efeito da obrigatoriedade de submissão de objetos %digitais, e certos padrões de comportamento observados na avaliação.

\subsubsection*{Completeness}

For calculating \textit{completeness}, we retrieved the VT ETD metadata records which follow the Dublin Core format. This format defines fifteen fields and the \textit{completeness} of a metadata record will be given by the number of fields present in a record among the fifteen.

%Este formato define 15 campos e a \textit{Completeza} de um registro de %metadados será dada pelo número de campos, dentre os 15 definidos, que %se encontram presentes no registro em questão.

The chart in Figure \ref{fig:compChart} shows that there are four distinct \textit{completeness} values in the catalog. This indicates that there are four groups of records with the same number of fields. The largest group (7,470 records) presents the highest level for \textit{completeness} in the catalog. The records of this group include 13 of the 15 fields defined by the Dublin Core format, which corresponds to a \textit{completeness} value near to 0.87. Looking at the other groups, 24 records present \textit{completeness} equal to 0.67, 1,162 equal to 0.73, and 52 equal to 0.80. No record is totally complete.

\begin{figure}[ht]
    \centering{\scalebox{0.3}{\includegraphics{imgs/5SQual_Chart_Completeness}\par}}
    \caption{VT-ETD - Completeness Chart}
    \label{fig:compChart}
\end{figure}

From the corresponding XML report, we can obtain for this dimension its average value (0.85) and the standard deviation (0.05). The high average and low standard deviation shows that the catalog of this particular DL is quite complete. Furthermore, retrieving the metadata records using their corresponding identifiers in the XML report, it is possible to check which fields are missing. For instance, for the 0.67 group, the one with the lowest level of \textit{completeness}, we found that the missing fields were \textit{relation}, \textit{coverage}, \textit{description}, \textit{contributor} and \textit{subject}. Analyzing this dimension, the administrator of a DL can have a clear idea of what is missing from its catalog and therefore of the required work for complementing it.


\subsubsection*{Conformance}

The Dublin Core format does not place any restriction on the minimum and maximum number of times a field should appear. To evaluate \textit{conformance}, we have considered a specific set of Dublin Core fields (\textit{title}, \textit{creator}, \textit{subject}, \textit{publisher}, \textit{date} and \textit{rights}) as mandatory, i.e., we required that they should appear at least once.

The chart in Figure \ref{fig:confChart} shows the VT-ETD \textit{conformance} evaluation regarding this particular set of restrictions. As we can see, all records exhibit high levels of conformance. The fact that there are just two distinct values for this dimension indicates that either the records are totally in conformance with the imposed restrictions (conformance value equal to 1.0) or that they have just one field that is not in conformance with them (conformance value equal to 0.93). The chart also shows that only 25 records were not totally in conformance with these restrictions.

\begin{figure}[ht]
     \centering{\scalebox{0.3}{\includegraphics{imgs/5SQual_Chart_Conformance}\par}}
    \caption{VT-ETD - Conformance Chart}
    \label{fig:confChart}
\end{figure}

Analyzing the XML report, we can identify the 25 records in the 0.93 \textit{conformance} group. When we looked at these records, we find that 24 of them do not have the \textit{subject} field, and that one record, identified by oai:VTETD:etd-08292003-154546, does not have the \textit{title} field.

% (o que é grave, uma vez que boa parte das buscas realizadas na %biblioteca é baseada no título). O administrador pode utilizar as %informações obtidas numa avaliação como esta para auxiliá-lo a verificar %e obrigar a existência de certo conjunto de campos nos registros de %metadados da biblioteca digital.

%\subsubsection*{Similarity By Content}

%Similarity reflects the relatedness between two or more digital objects.  %An object similar to another one has a good chance of also having %similar qualitative properties, but an object \textit{too} similar to %another supposedly different one can reveal a lack of quality (e.g., %plagiarism).

%In order to demonstrate the use of \textit{Similarity by content} we %performed a test with documents from VT-ETD  specifically chosen to %simulate the detection of problems  that can be indentified with this %dimension. In this test, we used a list of common stopwords as parameter %and the similarity was calculated using the ``bag-of-words'' algorithm %(however, we note that 5SQual can be extended to use other similarity %measures such as Okapi or the cosine distance \citep{BYRN:99}):


%Para mostrar \textit{Similaridade por conteúdo} realizamos dois testes, o primeiro com documentos da VT-ETD e um segundo com documentos especialmente escolhidos para simular a detecção de problemas pela dimensão de similaridade. Ambos os testes receberam uma lista de \textit{stop words} da língua inglesa como parâmetro de avaliação e o indicador numérico de \textit{Similaridade} foi calculado utilizando o algoritmo de comparação ``bag-of-words'' (however, we note that 5SQual can be extended to use other similarity measures such as Okapi or the cosine distance \citep{BYRN:99}):

%\begin{itemize}
%\item  In the first test, we we selected 30 full-text documents from %theses and dissertations available on VT-ETD, i.e., the tool receibved %as parameter 30 URLs pointing to thse documents (in PDF). One of these %documents was chosen as the reference document, the one to be compared %against all the others, including itself. Besides these  29 documents, %we also included  a URL point to a duplicate of the reference document %which existed on the Web. This was done to demonstrate a possible %detection of an undesired copy (for example, a case of plagiarism). 


%In the test, we used eight documents: one of reference and other seven %for comparison. With the goal of detecting duplicates or near-%duplicates, three of these documents were generated based on the %reference document: one document containing only the introduction of the %reference document, one containing two large sections of the reference %document, and a complete copy of that document. Among the other four %documents there was also one with content related to the reference one.


%\end{itemize}

%\begin{itemize}
%\item No primeiro teste, we selected 30 full-text documents from theses and dissertations available on VT-ETD, ou seja, a ferramenta 5SQual recebeu como parâmetro 30 URLs que apontavam para esses documentos (em PDF). One of these documents was chosen as the reference document, the one to be compared against all the others, including itself. Além dos outros 29 documentos, incluiu-se na lista de endereços para documentos a serem comparados, uma URL que apontava para uma duplicata do documento de referência exixtente na Web. Isto foi feito para demonstrar o caso de detecção de uma cópia do documento (por exemplo, um caso de plágio). 
%
%\item Para o segundo teste, utilizaram-se oito documentos: um de referência e sete para comparação. Visando uma simulação de detecção de cópias ou cópias parciais, três dos documentos para comparação foram gerados a partir do documento escolhido como referência. Escolheu-se como referência um artigo em PDF e, a partir de fragmentos dele foram gerados outros documentos em formato texto: um documento contendo somente a introdução do artigo, outro contendo duas seções grandes do documento e um terceiro contendo todo o documento. Os outros quatro documentos para comparação eram artigos PDF completos e um deles continha assunto correlato ao assunto do documento de referência.
%\end{itemize}
 
%The result of this first test did not reveal any document similar to the %reference one (except itself). The bar graph in Figure %\ref{fig:simContChart} shows the number of documents whose similarity is %within a certain interval. This interval was obtained  dividing the %difference between the maximum and minimum similarity by the number of %chosen bars. From the graphic it is possible to seem that there is only %one document significantly similar (similarity of 1,0) and that the %other documents are very dissimilar (similarities between 0,04 and %0,23).

%mostra o número de documentos cuja similaridade obtida se enquadra dentro de um determinado intervalo de valores. Tal intervalo foi obtido através da divisão do intervalo completo de valores (dado pela diferença entre os valores máximo e mínimo de similaridades obtidos na avaliação) pelo número de barras escolhido. Pelo gráfico, é possível observar que apenas um documento é significantemente similar (verifica-se similaridade igual a 1,0 e correspondente à duplicata do documento de referência, quando consultado o relatório de avaliação) e que os outros documentos apresentam similaridades muito pequenas (entre 0,04 e 0,23).

%\begin{figure}[ht]
%    %\centering{\scalebox{0.3}{\includegraphics{imgs/5SQual_Chart_SimByConten%t1}\par}}
%    \caption{VT-ETD - \textit{Similarity} by Content Chart}
%    \label{fig:simContChart}
%\end{figure}

%The results of the test are shown in Figure \ref{fig:simContChart2}. The %Figure shows that three out of the seven had a more significant %similarity with the reference document. Among the four less similar %documents, there is one within a higher similarity interval. By checking %the evaluation report, it is possible to identify each one of these %documents, as well as their exact similarity values. The complete copy %did not have a similarity equal to one due to problems of the conversion %of the corresponding PDF document to a text format (e.g., hyphenated %words, etc). The report also shows that, among the   4 most dissimilar %documents, the one with a related subject was the most similar (0.402). %The reference document was entitled \textit{``Greenstone: A %Comprehensive Open-Source Digital Library Software System''} and the %related one had the title {``The Greenstone plugin architecture''}.


%Considerando o segundo teste, o gráfico da Figura %\ref{fig:simContChart2} mostra que três documentos apresentaram %similaridade mais significativa do que outros quatro. Dentre os quatro %menos similares, ainda observa-se que um deles se encontra dentro de um %intervalo de similaridade mais alta do que os demais. Ao consultar o %relatório de avaliação, é possível identificar cada um desses %documentos, assim como o valor exato das similaridades obtidas para %eles. O relatório mostra que os três documentos mais similares %correspondem às três cópias (uma integral e duas parciais) do documento %de referência, na seguinte ordem de similaridade: texto com introdução % (0,743), texto com duas seções grandes (0,762) e cópia completa do texto % (0,848). A cópia completa do texto não apresenta similaridade igual a 1 %devido às restrições do formato texto, que não segue a formatação do %documento PDF (formatação que aparece quando o documento passa pelo %programa extrator de texto). O relatório também revela que o documento %que se destaca como o mais similar, dentre os quatro menos similares, %corresponde ao artigo de assunto relacionado ao documento de referência % (similaridade igual a 0,402). A título de curiosidade, este artigo se %intitula \textit{``The Greenstone plugin architecture''} e o documento %de referência corresponde ao artigo \textit{``Greenstone: A %Comprehensive Open-Source Digital Library Software System''}.


%\begin{figure}[ht]
%    %\centering{\scalebox{0.3}{\includegraphics{imgs/5SQual_Chart_SimByConten%t2}\par}}
%    \caption{VT-ETD - - \textit{Similarity} by Content Chart}
%    \label{fig:simContChart2}
%\end{figure}

%The results of the evaluation using the \textit{Similarity by Content} %dimension may identify plagiarism or problems with duplicates and %indicate exactly where these problems occur. Moreover, the tool can be %extended to use other similarity measures such as Okapi BM25 and the %cosine distance \citep{BYRN:99}.

%Os resultados da avaliação de \textit{Similaridade por conteúdo} podem %mostrar se problemas de plágio ou duplicata ocorrem dentre objetos %digitais do sistema e indicar os pontos exatos em que estes acontecem. %Além disso, a ferramenta 5SQual pode ser estendida para utilizar outras %medidas de similaridade como Okapi ou distância do cosseno %\citep{BYRN:99}.

\subsection{BDBComp Evaluation}\label{sec:avBDBComp}

Due to the easy access to the BDBComp log files, we focused the evaluation of this DL on the dimensions \textit{efficiency} and \textit{reliability}. These dimensions were evaluated based on the behavior of the searching and browsing services. Initially, the necessary data to calculate these dimensions would be extracted from the XMLLog file \citep{goncalves:03} in use by BDBComp, but because of problems during the generation of this file, the data about the request and response times and the status of the executions were lost. Hence, to illustrate these two dimensions, we extracted information from the Apache logs for \textit{reliability} and simulated some requests for searching services (also according to Apache logs) to calculate \textit{efficiency}. This information would be easily extracted from the XMLLog file since 5SQual already comes with a suitable parser.

\subsubsection*{Efficiency}

To evaluate \textit{efficiency}, we generated a series of search executions based on the most common queries according to the BDBComp Apache log file. On January 5, 2007, 60 requests were sent to five different BDBComp searching services (Search By Author, By Year, By Event, By Title and By Journal), and for each execution we stored the identifier of the service and its request and response times, specified in seconds. The generated files followed the 5SQual internal format which means that an external parser was not required.

The chart in Figure \ref{fig:effChart} shows the number of executions for each distinct response time. For instance, we can see that 15 of the 60 executions were processed in less that one second and that the slowest execution lasted 11 seconds.

\begin{figure}[ht]
     \centering{\scalebox{0.3}{\includegraphics{imgs/5SQual_Chart_Efficiency}\par}}
    \caption{BDBComp - Efficiency Chart}
    \label{fig:effChart}
\end{figure}

From the XML report (an excerpt from it is shown in Figure \ref{fig:report}), we can obtain more specific information about this evaluation. For instance, the slowest service, which took about 11 seconds to produce an answer, was Search by Year. Further investigation revealed the reasons. Due to the structure of the relational database that implements the BDBComp catalog, the SQL query processing for this kind of search yields a response set that is relatively large when compared with the other ones. Since the search processing time varies linearly with the size of the response set, this explains the poor performance of this specific type of search. The range of the desired year period also has an impact on the query processing time, since it determines the relative size of the result set. When we analyzed the results, we noticed that the two slowest queries were the ones of type Search by Year, for which the largest year period (1900 to 2000 and 1990 to 1998) was specified.

\subsubsection*{Reliability}

To evaluate \textit{reliability}, we extracted data from the BDBComp Apache log files, with 5SQual employing an external parser we created. The Apache logs cover the period between April 14, 2005 and January 3, 2007. We analyzed the searching and browsing services. The chart in Figure \ref{fig:relChart} reflects that 634,250 executions were evaluated, where 35,657 (5.6\%) ended in a failure.

\begin{figure}[ht]
    \centering{\scalebox{0.3}{\includegraphics{imgs/5SQual_Chart_Reliability}\par}}
    \caption{BDBComp - Reliability Chart}
    \label{fig:relChart}
\end{figure}

The corresponding XML report shows additional information. For instance, it reveals that from the failures only one corresponds to Search By Title and that all the rest of the failures are browsing services. Further investigation revealed that these failures were due to a period of instability of the server that went down many times. Additionally, considering the score for a success as 1 and that for a failure as 0, the average value was 0.94, which means that BDBComp services were quite reliable during the analyzed period.


\subsection{2002 ACM Collection Evaluation}

The information about citations among digital objects is an important resource for quality evaluation, making it possible to calculate indicators for the dimensions \textit{similarity}, \textit{significance}, and \textit{timeliness}. To demonstrate the evaluation of these dimensions, we used the ACM 2002 collection of 94,919 metadata records which include citation information and publication dates.

%As informações sobre as citações entre os objetos digitais consiste em %um recurso importante para avaliação, possibilitando o cálculo de %indicadores para as dimensões de \textit{Similaridade}, %\textit{Significância} e \textit{Timeliness}. Para demonstrar a %avaliação destas dimensões foi utilizada a coleção ACM 2002, uma coleção %com 94.818 registros de metadados que contém dados sobre o %relacionamentos de citações entre os objetos digitais e as datas de %publicação de cada objeto.

\subsubsection*{Significance}

We evaluated the \textit{significance} of a digital object in the ACM collection according to the number of citations it receives from other objects in the collection. For this evaluation, 5SQual generated the chart in Figure \ref{fig:sigINChart}. It shows that the majority of the objects has very few citations within the collection and that there are just a small number of documents with a high significance value.

\begin{figure}[ht]
    \centering{\scalebox{0.3}{\includegraphics{imgs/5SQual_Chart_Significance}\par}}
    \caption{ACM - Significance Chart}
    \label{fig:sigINChart}
\end{figure}

The XML report details information about this evaluation, specifying for each object its respective number of citations within the collection. The maximum number of citations to a document is 328 in ACM 2002 (for the document ``\textit{Computer programming as an art}"). There are 51,925 objects without citations. The average number of citations an object receives is 2.35 (this number refers just to citations of papers that are in the ACM collection). The high standard deviation value (6.16) shows that the number of citations has high variability.

 
\subsubsection*{Similarity by Citations}

To illustrate \textit{similarity by citations}, we have chosen to compare, against the whole collection, two digital objects: the one with more out-citations (i.e., the references that appear in a document) and the one with more in-citations (i.e., the citations a document receives). We used two numeric indicators for these comparisons: co-citation \citep{small:73}, considering the document with more in-citations as the reference one, and bibliographic coupling \citep{kessler:63} to compare the document with more out-citations against the others.  Two documents are co-cited if a third one has citations to both of them (i.e., if they have in-citations in common).  The more in-citations in common the more related or similar they are. Bibliographic coupling looks for common out-citations in the two objects being compared.

% Formulas for computing the numeric indicators can be found in %\citep{goncalves:07}.

Bar charts for these evaluations are given in Figures \ref{fig:simCOChart} and \ref{fig:simBIChart}. They show the number of objects with similarity to the reference documents  inside certain intervals. The width of the intervals was obtained by dividing the size of the whole interval (given by the difference between the maximum and the minimum similarity value) by the number of bars.


\begin{figure}[ht]
 \centering{\scalebox{0.3}{\includegraphics{imgs/5SQual_Chart_SimCit}\par}}
  \caption{ACM - Similarity Chart - Co-citation}
  \label{fig:simCOChart}
\end{figure}

\begin{figure}[ht]
 \centering{\scalebox{0.3}{\includegraphics{imgs/5SQual_Chart_SimBib}\par}}
  \caption{ACM - Similarity Chart - Bibliographic Coupling}
  \label{fig:simBIChart}
\end{figure}

Both evaluations indicate similar behavior: the majority of the objects are concentrated in the first interval, with the smallest similarity values. But when we look at the values along the horizontal axis, we can see that the bibliographic coupling measure values are more significant (almost 10 times higher) than the values obtained by co-citation. References in the documents (out-citations) contribute more to similarity than the citations they receive (in-citations). In the ACM 2002 collection we can see that there are more digital objects without in-citations (51,925) than without out-citations (46,331). This result is consistent with  \citep{thierson:06}, where it was shown that for DLs containing scientific papers, measures based on bibliographic coupling are more appropriate for similarity detection.

\subsection*{Timeliness}

For the ACM 2002, we used \textit{timeliness} regarding the last date a specific digital object was cited, considering only the internal citations within the collection. This date spans the period of influence of the information contained in the object. This evaluation was performed on January 14, 2007.

%Para a coleção ACM 2002, definiu-se a \textit{timeliness} em relação à %data da última vez que o objeto digital em questão foi citado, %considerando apenas as citações internas da coleção. Esta data foi %obtida a partir dos metadados da coleção, que armazenam, além das %citações, a data de publicação de um objeto. O cálculo da dimensão %\textit{Timeliness} em relação à última vez que o objeto foi citado %consiste em avaliar a temporalidade da influência deste objeto, pois a %última citação delimita a atuação desta influência. Esta avaliação foi %realizada no dia 14 de janeiro de 2007.

As expected, since the analyzed collection is from 2002, the graphic in Figure \ref{fig:timACMChart}  shows that the objects more recently cited received these citations four years ago. Moreover it is possible that many objects received citations between 4.5 and 7 years ago and that there are objects that are not cited for more than 55 years.

%Conforme esperado, uma vez que a coleção analisada é do ano de 2002, o %gráfico da Figura \ref{fig:timACMChart} mostra que os objetos mais %recentemente citados receberam essas citações há mais de quatro anos. %Além disso, é possível perceber que muitos objetos receberam suas %últimas citações entre 4.5 e 7 anos atrás e que há objetos que não são %citados há mais de 55 anos.

\begin{figure}[ht]
    \centering{\scalebox{0.3}{\includegraphics{imgs/5SQual_Chart_TimACM}\par}}
    \caption{ACM - \textit{Timeliness} Chart}
    \label{fig:timACMChart}
\end{figure}

In the XML report, it is possible to identify each individual object and the \textit{timeliness} value associated with it.

%Pelo relatório de avaliação é possível identificar cada objeto digital e o valor de \textit{Timeliness} associado a ele.



\section{Evaluation of the Tool}
\label{sec:Eval}

We evaluated 5SQual in two ways: (1) through a usability evaluation of its user interface and (2) through a series of interviews with potential users about their expectations about the tool. These two evaluations are described next.

\subsection{Usability Evaluation of the 5SQual User Interface} \label{sec:evalsOn5SQual}

A heuristic evaluation, as defined by \citep{nielsen:90}, is a usability evaluation  method  in which an evaluator searches for usability problems in a user interface through the analysis and interpretation of a set of principles or heuristics. The participation of a usability specialist is essential and it is desirable that at least a portion of the evaluators have knowledge about the application domain.
%, since this evaluation method is based on the evaluator judgment.

%A avaliação heurística, definida por \citep{nielsen:90}, é um método de %avaliação de usabilidade onde um avaliador procura problemas de %usabilidade em uma interface através da análise e interpretação de um %conjunto de princípios ou heurísticas. A participação de especialistas %em usabilidade é essencial e é desejável que pelo menos uma parcela dos %avaliadores tenha conhecimento sobre o domínio da aplicação, uma vez que %este método de avaliação é baseado no julgamento do avaliador.

\subsubsection{Methodology}

For the heuristic evaluation performed with the 5SQual interface, we invited three usability specialists, all having some knowledge about digital libraries and, specifically, about the 5S quality model and 5SQual. One evaluator was also developing another application based on the 5S framework. The heuristics that guided the evaluation correspond to the basic set of heuristics defined in \citep{nielsen:93}, whose aim is to identify general problems in a generic user interface: (1) visibility of the system state, (2) compatibility between the system and the real world; (3) control and freedom for the user; (4) consistency and standardization; (5) error prevention; (6) recognition instead of memorization; (7) flexibility and efficiency in use; (8)  minimalist design; (9) capability of assisting users to recognize, diagnose and recover from errors; and (10) help and documentation.

%Para as avaliações heurísticas realizadas sobre a interface da 5SQual foram convidados três especialistas em usabilidade, sendo que todos eles tinham conhecimento sobre bibliotecas digitais e, especialmente, sobre o modelo de qualidade 5S e sobre a 5SQual. Um dos avaliadores também atuava no desenvolvimento de outra aplicação baseada no arcabouço 5S. As heurísticas que guiaram a avaliação correspondem ao conjunto básico de  heurísticas definidas por \citep{nielsen:93}, as quais visam cercar todos os problemas de uma interface genérica: 1, Visibilidade do estado do sistema; 2, Compatibilidade entre sistema e mundo real; 3, Controle e liberdade para o usuário; 4, Consistência e padrões; 5, Prevenção de erros; 6, Reconhecimento em lugar de memorização; 7, Flexibilidade e eficiência de uso; 8, Projeto minimalista e estético; 9, Auxiliar os usuários a reconhecer, diagnosticar e recuperar erros  e 10, Ajuda e documentação. 

A manual for the tool, a document with a series of tasks along with the parameters to configure a complete evaluation with 5SQual, and an evaluation guide, were created. A configuration task similar to what the specialists had to perform is shown in Appendix A, which also contains snapshots of the configuration interfaces. The evaluation guide included an explanation about the heuristics and a table to be filled with the problems found, associated with each heuristic, according to some pre-determined levels: 0 (it does not affect the operation of the interface; 1 (there is no need for an immediate solution); 2 (problem with low priority); 3 (problem with high priority); and 4 (severe problem). The evaluation followed this procedure: 

%Foram elaborados um manual da ferramenta, um roteiro de tarefas com %parâmetros válidos para a configuração de uma avaliação completa pela %5SQual e um guia para a avaliação, o qual inclui a explicação sobre as %heurísticas e uma tabela que deve ser preenchida com os problemas %encontrados associados a heurística que atingem e a um grau de %severidade (0, Não afeta a operação da interface; 1, Não há necessidade %imediata de solução; 2, Problema de baixa prioridade; 3, Problema de %alta prioridade e 4, Muito grave). O procedimento utilizado para a %avaliação foi o seguinte:

%\begin{enumerate}
%\item A ferramenta 5SQual foi disponibilizada para os avaliadores %juntamente com o manual, o roteiro de tarefas e o guia para avaliação.
%\item Foi explicado ao avaliador que ele deveria navegar e testar a %ferramenta livremente.
%\item Pediu-se que os avaliadores preenchessem a tabela com os problemas %encontrados, associando a estes as heurísticas atingidas e o grau de %severidade que julgassem apropriado.
%\item As três avaliações foram compiladas.
%\end{enumerate}

\begin{enumerate}
\item The 5SQual tool was given to the evaluators along with the manual, the list of tasks, and the evaluation guide.
 \item The evaluator was instructed to freely navigate and test the tool.
\item We asked the evaluators to fill in the table with each problem found, associating these problems with the heuristics and their respective levels.
\item The three evaluations were compiled and analyzed.
\end{enumerate}

\subsubsection{Results}
A total of 53 problems was found regarding the tool screens, the heuristics, and the defined levels. 

%Foram encontrados 53 problemas ao longo das telas da ferramenta, das %heurísticas apresentadas e dos graus de severidade definidos. 

The chart in Figure \ref{fig:telasProbs} shows the percentage of problems found in each one of the screens of the tool. It can be seen that the screen with most of the problems is screen 3, with 43\% of the found problems. In that screen, shown in Figure \ref{fig:ui3_1}, the user spends a great portion of the configuration time of an evaluation, since she needs to select the desired dimensions that will be evaluated and configure all the respective resources in the corresponding popup windows (a window for each resource). The problems found in these popup windows were also associated with screen 3, which justifies the high number of problem in only one window. More than half of these problems were related to corrections and improvements in the explanatory texts of the interface. The most severe problems found in this window were related to the lack of clarity in the exposition of the dependencies existent between the selected dimensions and the necessary resources for these dimensions. 

%O gráfico da Figura \ref{fig:telasProbs} mostra a porcentagem de %problemas encontrada em cada uma das telas da ferramenta. Observa-se que %a tela com maior número de problemas é a Tela 3, com 43\% dos problemas %encontrados. Nesta tela, mostrada nas Figuras \ref{fig:ui3_1} e %\ref{fig:ui3_2}, o usuário despende grande parte do tempo da %configuração de uma avaliação, pois deve selecionar as dimensões que %deseja avaliar e configurar todos os recursos associados às dimensões %escolhidas, através de janelas de \textit{popup} (uma janela para cada %recurso). Os problemas encontrados nestas janelas de \textit{popup} %também foram associados à Tela 3, o que justifica o grande número de %problemas em uma só tela. Mais da metade desses problemas se refere a %correções e melhorias nos textos explicativos da interface. Os problemas %mais graves encontrados nesta tela se referem à falta de clareza na %exposição das dependências entre as dimensões selecionadas e os recursos %necessários para estas dimensões.

Screens 2 and 5, shown in Figures \ref{fig:ui2} e \ref{fig:ui5} respectively, are the most simple screens, and therefore presented fewer problems.

%As Telas 2 e 5, mostradas nas Figuras \ref{fig:ui2} e \ref{fig:ui5}, %respectivamente, são as telas mais simples, e portanto apresentaram %menos problemas.
 
\begin{figure}[ht]
    \centering{\scalebox{0.6}{\includegraphics{imgs/tela}\par}}
    \caption{Percentage of problems found per screen}
    \label{fig:telasProbs}
\end{figure}

In Figure \ref{fig:heuProbs}, the chart shows that the heuristics more associated with problems were: \#1 (visibility of the system status), \#4 (consistency and standardization), and \#5 (error prevention), each one with approximately 17\% of the total set of problems found.

%Na Figura \ref{fig:heuProbs}, o gráfico mostra que as heurísticas mais %associadas a indicações de problemas foram as de número 1 (Visibilidade %do estado do sistema), 4 (Consistência e padrões) e 5 (Prevenção de %erros), cada uma com uma quantidade de problemas associados %correspondentes a 17\% do conjunto total dos problemas encontrados.

\begin{figure}[ht]
    \centering{\scalebox{0.7}{\includegraphics{imgs/heu}\par}}
    \caption{Portion of Found Problems per Heuristics}
    \label{fig:heuProbs}
\end{figure}


In the chart in Figure \ref{fig:grauProbs}, it is seen the distribution of the problems and their levels as given by the specialists. The majority of the found problems have low priority, but 31\% of the problems have high priority. This indicates that there is a lot of room for improvements in the interface. The problems classified as level 4 were evaluated and properly corrected.

%No gráfico da Figura \ref{fig:grauProbs}, é mostrada a distribuição dos %problemas em relação ao grau de severidade atribuído pelos %especialistas. A maioria dos problemas encontrados é de baixa %prioridade, mas  31\% dos problemas são de alta prioridade. Isso indica %que a interface ainda deve ser bastante melhorada, para que esta seja %realmente acessível e usável. Os problemas classificados com grau de %maior severidade (grau 4) foram avaliados e devidamente corrigidos.

\begin{figure}[ht]
    \centering{\scalebox{0.6}{\includegraphics{imgs/grau}\par}}
    \caption{Percentage of Found Problems per Level}
    \label{fig:grauProbs}
\end{figure}

This evaluation allowed us to identify possible limitations of the 5SQual interface, providing directions for its improvement. Since the interface was implemented independently of the core functionality of the tool, it is possible to develop new interfaces adapted for different user profiles.

%Esta avaliação permitiu identificar várias das limitações da interface %desenvolvida para a ferramenta 5SQual neste trabalho, provendo direções %para o seu aprimoramento. Como a interface foi implementada independente %do núcleo da ferramenta 5SQual, é possível inclusive desenvolver uma %nova interface, ou até várias, adaptáveis a diferentes perfis de %utilização.


\subsection{Evaluation of the Expectations and Opinions of Potential Users}\label{sec:focus}

In order to capture the expectations of possible users of the tool and the potential of the practical application of 5SQual, we performed a study with a group of potentially interested users who were administrators of running DLs. The study was planned based on the exploratory qualitative method described in  \citep{nicolai:04}. This method uses open questions to collect opinions from the subjects and prioritizes the depth of the results that will be obtained, focusing on a few objective questions. Moreover, the choice of the subjects aims at a homogenous group according to some predefined criteria relevant to the proposed investigation. In the case here, these criteria include having performed administrative tasks in a real digital library.


%Para capturar as expectativas de possíveis usuários da ferramenta e o %potencial de aplicação prática da 5SQual, foi elaborado um estudo com um %grupo de interesse, composto por administradores de bibliotecas digitais %reais. Este estudo foi planejado tendo como base o método qualitativo %exploratório, descrito em \citep{nicolai:04}. Este método utiliza %questões abertas para coletar opiniões dos entrevistados e prioriza a %profundidade dos resultados a serem obtidos, fechando o escopo das %entrevistas através de poucas perguntas objetivas. Além disso, a escolha %dos entrevistados visa um grupo homogêneo de acordo com determinado %critério relevante para a investigação proposta. No presente caso, este %critério consiste em ter exercido atividades de administração de uma %biblioteca digital. 

\subsubsection{Methodology}
To perform the interviews, we executed the following procedure:

%Para a realização das entrevistas foi executado o seguinte procedimento:

\begin{enumerate}


\item We produced a list of 10 possible subjects for the interviews, composed of administrators of DLs in several parts of the world.

\item The interviews of this evaluation were conducted by email, so that the administrators could have more freedom to answer the questions  appropriate to them. To present 5SQual to these subjects, a document with instructions and screen shots was prepared. This document describes an evaluation process step-by-step and includes samples of graphics and reports that result from evaluations conducted with the tool.

%\item As entrevistas desta avaliação foram conduzidas por meio de %mensagens de correio eletrônico, para que o administrador tivesse maior %liberdade para responder às perguntas no momento que lhe fosse mais %apropriado. Esse meio de entrevista possibilitou a coleta de um maior %número de opiniões. Para apresentar a 5SQual de forma não presencial, %foi elaborado um manual ilustrado. Este manual mostra o passo-a-passo de %uma avaliação, além de apresentar também amostras de gráficos e %relatórios resultantes de avaliações conduzidas com a ferramenta.

\item A questionnaire was elaborated with seven open questions, covering several aspects of the tool. The questions were:

%\item Foi elaborado um questionário com sete perguntas abertas, %abrangendo diversos aspectos da ferramenta. O questionário foi composto %das seguintes questões:

\begin{enumerate}
%\item Você conhece alguma forma de avaliação Bibliotecas Digitais? Se %sim: Qual(is)? O que acha delas?
\item Are you aware of methods/models for evaluating digital libraries? If  yes: which ones? What do you think about them?

%\item Na sua opinião, quais os pontos positivos e negativos em relação %ao tipo de avaliação realizado pela 5SQual?
\item In your opinion, what are the positive and negative points regarding the type of evaluation carried out by 5SQual?

%\item Na sua opinião, quais os pontos positivos e negativos em relação aos recursos e parâmetros necessários para realizar uma avaliação com a 5SQual?
\item In your opinion, what are the positive and negative aspects regarding the resources and parameters necessary to perform an evaluation with 5SQual?

%\item Na sua opinião, quais os pontos positivos e negativos em relação à interface da ferramenta 5SQual?
\item In your opinion, what are the positive and negative aspects regarding the 5SQual interface?

%\item Na sua opinião, quais os pontos positivos e negativos em relação à apresentação dos resultados de uma avaliação produzidos pela 5SQual?
\item In your opinion, what are the positive and negative aspects regarding the presentation of results of an evaluation with 5SQual?

%\item Você acredita que a 5SQual seria útil para você na administração %de uma biblioteca digital? Por quê?
\item Do you believe that 5SQual is useful for you in the administration of a digital library? Why?

%\item Por favor, sinta-se a vontade para acrescentar quaisquer outros comentários e sugestões.
\item Please, feel free to add any comments and suggestions.

\end{enumerate}

%\item Foi enviada uma mensagem eletrônica requisitando a colaboração dos %administradores na entrevista. Nessa mensagem explicou-se o propósito da %avaliação, e procurou-se fornecer todas as informações necessárias sobre %a pesquisa. Foi enfatizada a participação voluntária, a preservação do %anonimato dos participantes e a liberdade que os entrevistados deveriam %ter ao responder as questões do questionário.

\item An electronic message was sent requesting the collaboration of the administrators. In this message, we explained the goal of the evaluation and we provided all the necessary information about our research. We emphasized the voluntary nature of the interview, the preservation of anonymous identities, and the freedom to answer any questions of the questionnaire.

%\item Das dez mensagens enviadas, quase duas semanas depois, obteve-se o %conjunto de cinco questionários completamente respondidos, a partir dos %quais realizou-se uma análise.

\item From ten messages sent, we obtained, two weeks later, five answers completely filled in, on which we based our analysis.


\end{enumerate}

\subsubsection{Results}


The DL administrators that answered our questionnaire had different and diverse profiles with respect to their previous experience. The number of DLs which they have administered varied from 1 to 4 different systems and the experience on this task varied from 1,5 to 12 years. Among the 5 interviewees, one was from abroad (outside from Brazil). With respect to knowledge about evaluation of DLs, part of the group mentioned no knowledge at all and the remainder said they had some knowledge but nothing that was formal or involved automation.

%Os administradores que responderam às perguntas da entrevista %apresentam perfis variados em relação à experiência prévia. O número de bibliotecas digitais já administradas por eles varia entre 1 e 4 sistemas distintos e o tempo de atuação como administrador varia de 1,5 a 12 anos de experiência. Dentre os cinco entrevistados, um é administrador de bibliotecas digitais fora do Brasil. Em relação ao conhecimento sobre formas de avaliação em bibliotecas digitais, parte não tinha conhecimento a respeito e parte listou avaliações que consideram alguns aspectos de bibliotecas digitais de forma não automatizada ou formal.



In order to interpret the results of the interviews, we analyzed recurrent answers. This analysis was performed in two-steps. First, we analyzed the opinions recurrent among the subjects within each question. After, we expanded this analysis to consider recurrent opinions along the complete questionnaire. From the found recurrences, we proposed the following comments to be considered and analyzed, as a result of the interview:     

%Para interpretar as entrevistas, analisaram-se as recorrências %existentes dentre todas as respostas do questionário. A análise ocorreu %em dois momentos. Primeiro, analisaram-se todas as respostas para cada %pergunta, identificando as recorrências de opinião entre os %entrevistados dentro de cada questão. Depois, expandiu-se a fronteira de %análise, analisando as recorrências de opinião ao longo de todo o %questionário. A partir das recorrências encontradas são propostas as %seguintes categorias para serem consideradas e analisadas, como %resultado da entrevista:

\begin{enumerate}
\item ``The users may have difficulties to indicate recognizers for the input files of 5SQual''. This observation was made by all the subjects. The development of recognizers demands that the administrator or other people in her staff have programming skills. One way to alleviate this problem is to release a larger set of recognizers along with 5SQual. The standardization of formats (e.g., log formats, citation references) would make this effort even more effective.
 
%\item Os usuários podem apresentar dificuldades para indicar %reconhecedores para os arquivos de entrada da 5SQual. Esta foi uma %observação feita por todos os entrevistados e por alguns de forma %recorrente. O desenvolvimento de reconhecedores exige que o %administrador domine aspectos de programação, o que destoa do cenário %atual, onde os administradores de bibliotecas digitais muitas vezes são %bibliotecários. Uma maneira para amenizar este problema seria disponibilizar junto à 5SQual um conjunto maior de reconhecedores, mas %para que esta ação se torne uma solução real para a maioria dos %administradores, deve haver padronização nos formatos utilizados nas %bibliotecas digitais.

%\item Os entrevistados consideraram a interface sucinta e clara, com a %exceção de que a obrigatoriedade dos parâmetros deveria ser mais %explicitamente apresentada.

\item The subjects considered the interface clear and simple with the exception of the fact that the mandatory aspect of some parameters should be made more explicit.

%\item Em relação às saídas emitidas pela 5SQual, os administradores %gostaram da visão geral e praticidade dos gráficos. Em relação ao %relatório, consideraram que os dados, por serem muito detalhados, podem %dificultar a leitura dos resultados. Como o relatório é gerado em XML, %para a solução deste problema, aplicações podem ser desenvolvidas para %gerarem arquivos mais sucintos e claros a partir do arquivo original %(utilizando XSLT\footnote{http://www.w3.org/TR/xslt}, por exemplo).

\item Regarding the outputs generated by 5SQual, the administrators enjoyed the global perspective and practical nature of the graphics. Regarding the reports, they considered that, by being very detailed, these reports may be difficult to read and interpret.  Since the report is generated in an XML format, several style sheets (XSLT\footnote{http://www.w3.org/TR/xslt}) or similar applications may be developed to generate more succinct and clear summaries of these files. 

%\item Os entrevistados salientaram as vantagens do processo de avaliação %ser automático, sistematizado e de cobrir um conjunto importante de %componentes de uma biblioteca digital (objetos digitais, especificações %de metadados e serviços). Esta categoria indica a existência da %necessidade de uma ferramenta como a 5SQual, e valida o seu potencial de %aplicação como ferramenta de administração de bibliotecas digitais. 

\item The subjects made very clear that they appreciate the advantages of having an automatic and systematic evaluation process capable of covering an important set of dimensions and components of a digital library. This indicates that there exists a real need for a tool such as 5SQual and partially validates its potential for administrators.

\end{enumerate}

These results, besides highlighting the aspects of the tool that were appreciated by the users, also provide indications for future work and improvements that can be incorporated in 5SQual to better address the expectations of its target community.

%além de mostrarem os pontos da ferramenta que agradaram aos usuários, %geram também indicações de trabalhos futuros e melhorias que podem ser %realizados para que a 5SQual se adeque melhor às expectativas do seu %público alvo.

\section{Conclusions and Future Work} \label{sec:conclusions}

5SQual is a tool to help with the evaluation of digital libraries that can be used by administrators in the maintenance of this type of system. The results of evaluations performed with 5SQual can guide the design, development, and improvement of digital libraries. Moreover, the tool can help in the identification of problematic items and, when used over time, can demonstrate the evolution of many aspects of the evaluated systems. 

%A 5SQual é uma ferramenta de apoio à avaliação de bibliotecas digitais, %que pode ser utilizada por administradores na construção e manutenção %desses sistemas. Os resultados das avaliações realizadas pela 5SQual %podem guiar o projeto, desenvolvimento e aperfeiçoamento de bibliotecas %digitais. Além disso, a ferramenta pode auxiliar na identificação de %itens problemáticos e, quando utilizada repetidamente ao longo do tempo, %pode mostrar a evolução de muitos aspectos do sistema avaliado. 
 
Digital libraries are heterogeneous systems that may present and make available information in many disparate formats. In order to allow the application of the tool in many diverse systems, 5SQual provides solutions to retrieve data in the local file system, on the Web, and through OAI-PMH. Regarding data extraction, 5SQual is flexible and allows the user to specify recognizers that come with the tool or to specify external programs capable of converting specific formats to the ones used by the tool.

%Bibliotecas digitais são sistemas heterogêneos que podem apresentar e %disponibilizar dados de diferentes e variadas formas. Para permitir a %aplicação da ferramenta em diversos sistemas, 5SQual provê soluções para %a busca de dados no sistema de arquivos local, na Web e através do %protocolo padrão para coleta de metadados OAI-PMH. Em relação à extração %de dados, 5SQual busca ser flexível, permitindo a seu usuário a %indicação de programas externos que sejam capazes de converter o formato %específico do arquivo da biblioteca para um formato padrão de arquivo %que a ferramenta 5SQual consiga ler.

Evaluations of the tool conducted with a group of potential users showed that there is a demand for a tool such as 5SQual and that the users approved the presentation of its interface and considered practical the graphics produced by the tool. Moreover, the results also indicate possible improvements and extensions of functionality and applicability of the tool, such as the inclusion of new dimensions and indicators, and the availability of new recognizers and standards that can help to expand the utilization of 5SQual. 

%Uma avaliação com um grupo de usuários potenciais mostra que há uma %demanda por uma ferramenta como a 5SQual e que os usuários aprovam a %apresentação da interface e consideram práticos os gráficos gerados pela %ferramenta. Além disso, indicam também melhorias e possíveis extensões %de funcionalidade e aplicabilidade da ferramenta, como a inclusão de %novos indicadores e dimensões, a disponibilização de reconhecedores e %padrões que possam ajudar a expandir a utilização da 5SQual e a criação %de aplicações que possam auxiliar o processamento da avaliação %realizada.

\section{Acknowledgments}
This work was partially supported by CNPq, CAPES and Fapemig. This material is based upon work supported by the National Science Foundation under Grant Nos. 0736055, 0722259, 0535057, 0532825, and 0325579. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.


\bibliographystyle{elsart-harv}
\bibliography{bibfile}

\newpage

% redefine the command that creates the equation no.
\setcounter{figure}{0} % reset counter 

\appendix

\section{5SQual Operation and Configuration}

\textbf{In this Appendix we present how a typical DL administrator can operate and configure the 5SQual tool.}

Before starting with 5SQual, a user, typically the administrator of a DL, has to configure the parameters for the evaluation through an interface that was specially developed to help with this task. The 5SQual interface works like a setup wizard that guides the user through the necessary configuration steps, assuring that the mandatory parameters have been filled in before undertaking an evaluation. An XML file with the configured parameters is generated and can be imported later through the same interface to repeat the evaluation, making it easier for the user to analyze the system over time.


The parameters indicate where 5SQual should find information for the evaluation and how to extract them to calculate the selected dimensions. Once the documents, metadata, and any other necessary files are available, 5SQual extracts the required information. To accomplish this, 5SQual uses external programs specified by the user or the built-in parsers that come with the tool. Then, the extracted information is used to calculate numeric indicators for each dimension to be evaluated. In the following, we show a step-by-step configuration of an evaluation carried out using 5SQual.

%If the user chooses to look for the information remotely, 5SQual %connects to the Internet for file transfer or uses the OAI protocol. If %the information should be collected locally, the user should define the %right paths in the local file system.


The interface first presents to the user two options: (1) start a new evaluation from the beginning by following all the steps to configure the necessary parameters for this purpose or (2) import a previously generated file with all the parameters already specified (see Figure \ref{fig:ui1}).

\begin{figure}[ht]
    \centering{\scalebox{0.85}{\includegraphics{imgs/5SQualUI_1}\par}}
    \caption{5SQual Interface - Starting Configuration}
    \label{fig:ui1}
\end{figure}

If the user chooses to fill the parameters through the interface, she is then asked to identify this evaluation by giving the name of the DL that is being assessed and adding an optional description (see Figure \ref{fig:ui2}). This serves to facilitate re-use of this configuration in a next evaluation.

\begin{figure}[ht]
    \centering{\scalebox{0.85}{\includegraphics{imgs/5SQualUI_2}\par}}
    \caption{5SQual Interface - Evaluation Identification}
    \label{fig:ui2}
\end{figure}

Now, the user must choose which quality dimensions to evaluate. The dimensions are selected from a set of \textit{checkboxes} located in the left portion of the screen (see Figure \ref{fig:ui3_1}). When a dimension is selected, the necessary resources for calculating the respective indicators are presented in the right portion of the screen.  This is important to make the user aware of the resources that the DL must provide in order to be evaluated under that dimension. If this resource cannot be obtained, the dimension must be deselected. Another interesting aspect of the interface is the \textit{help icons} (with the question marks) shown in front of the name of each dimension. If the user presses one of these icons, a definition of the dimension along with the explanation of its numerical indicators is presented.

% (as seen in Figure\ref{fig:ui3_1}).

%Agora, o usuário deve escolher quais dimensões de qualidade avaliar. As dimensões são selecionadas a partir de um conjunto de \textit{checkboxes} localizadas à esquerda da tela (vide a Figura \ref{fig:ui3_1}) e, à medida em que vão sendo selecionadas, os recursos necessários para calcular as dimensões escolhidas vão sendo listados à direita da tela. Esta informação é importante para o usuário pois, ao selecionar uma dimensão, ele tem consciência de que a biblioteca digital deve prover tal recurso para poder ser avaliada em relação àquela dimensão, caso contrário, a seleção daquela dimensão deve ser retirada. Outro tipo de ajuda oferecida ao usuário nesta tela são os ícones com o símbolo de interrogação em frente a cada nome de dimensão. Ao pressionar este símbolo, o usuário pode verificar a definição da dimensão e o seu indicador numérico (como destacado na Figura \ref{fig:ui3_1}).

\begin{figure}[ht]
    \centering{\scalebox{0.85}{\includegraphics{imgs/5SQualUI_3}\par}}
    \caption{5SQual Interface -– Selection of Dimensions and Indication of Resources}
    \label{fig:ui3_1}
\end{figure}

%After choosing all the dimensions that will be evaluated, the user must %configure the resources that are shown in the list on the right. For %such, she must select in the list a specific resource and press the %button \textbf{Configure Resource} (see Figure \ref{fig:ui3_2}). That %done, a window requesting information about where to retrieve the chosen %resource pops up. In Figure \ref{fig:ui3_2}, a configuration window of %the resource of the dimension \textit{Similarity by citations} requests %the path for the file containing data about citations among the digital %objects. All the listed resources are mandatory and must be configured %before the user advances to the next step. 

After choosing all the dimensions that will be evaluated, the user must configure the resources that are shown in the list on the right. For such, she must select in the list a specific resource and press the button \textbf{Configure Resource} (see Figure \ref{fig:ui3_1}). That done, a window requesting information about where to retrieve the chosen resource pops up. In Figure \ref{fig:ui3_1}, a configuration window of the resource of the dimension \textit{Reliability} requests the path for the file containing data about the status of several DL services executions, during a period of time. The user can choose to look for this file in the local file system or remotely, on the Web. This resource is mandatory and must be configured before the user advances to the next step.

%Após escolher todas as dimensões que serão avaliadas, o usuário deverá então configurar os recursos que se encontram na lista à direita. Para tanto, deve selecionar na lista um recurso de cada vez e pressionar o botão \textbf{Configurar Recurso} (veja Figura \ref{fig:ui3_2}). Ao fazer isso, surgirá uma janela requisitando informações sobre onde buscar o recurso em questão. Na Figura \ref{fig:ui3_2}, a janela de configuração do recurso da dimensão \textit{Similaridade por citações} requisita o caminho para o arquivo que contenha dados sobre as citações entre os objetos digitais. Todos os recursos listados são obrigatórios e devem ser configurados antes que o usuário avance para o próximo passo.


%\begin{figure}[ht]
%    \centering{\scalebox{0.85}{\includegraphics{imgs/5SQualUI_3_2}\par}}
%    \caption{5SQual Interface - Selecting Dimensions and Indicating %Resources}
%    \label{fig:ui3_2}
%\end{figure}

%The user must then specify parameters about how to extract the data from %the indicated resources and how to calculate the indicators for each %selected dimension. In the screen shown in Figure \ref{fig:ui4}, it is %on the left the dimensions selected in the previous step. Once a %dimension is selected, the area on the right changes, presenting a panel %that requests information about the parameters of that dimension. In %Figure \ref{fig:ui4}, the user chose the dimension \textit{Similarity by %citations} and configured the necessary parameters: the identifier of %the reference digital object that appears in the citation file informed %in the previous screen, the metrics to be used to calculate the quality %indicators of this dimension and the recognizer program used to extract %the data from the citation file. In that case, the user informed a %resource that was already in the standard format used by 5SQual, %therefore allowing her to use the standard recognizer. 

The user then must specify parameters about how to extract the data from the indicated resources and how to calculate the indicators for each selected dimension. In Figure \ref{fig:ui4}, the dimensions chosen in the previous step are shown on the left portion of the screen. Once a dimension is selected, the area on the right changes, presenting a panel that requests information about the parameters for the selected dimension. In Figure \ref{fig:ui4}, the user chose the dimension \textit{Reliability} and configured the necessary parameter: the recognizer program used to extract information from the log file containing data about the status of the executions of the DL services. In that case, the user specified a resource to be parsed using the 5SQual plugin for the XMLLog format.


%O usuário deve especificar parâmetros sobre como extrair os dados dos %recursos informados previamente e como calcular os indicadores para cada %dimensão selecionada. Nesta tela, à é mostrada esquerda uma lista com as %dimensões escolhidas no passo anterior (vide Figura \ref{fig:ui4}). Uma %por vez, as dimensões devem ser selecionadas desta lista para que seus %parâmetros sejam configurados. Quando uma dimensão é selecionada, a área %à direita da lista muda, apresentando um painel que requisita %informações sobre os parâmetros daquela dimensão. Na Figura %\ref{fig:ui4}, o usuário selecionou a dimensão \textit{Similaridade por %citações} e configurou os parâmetros necessários: o identificador do %objeto digital de referência que aparece no arquivo de citações %informado na tela anterior, as métricas a utilizar para calcular os %indicadores de qualidade desta dimensão e o programa reconhecedor para %extrair dados do arquivo com as citações. Neste caso, o usuário informou %um recurso que já se encontrava no formato padrão da 5SQual, portanto %especificou o programa reconhecedor padrão.

\begin{figure}[ht]
    \centering{\scalebox{0.85}{\includegraphics{imgs/5SQualUI_4}\par}}
    \caption{5SQual Interface – Specification of Parameters}
    \label{fig:ui4}
\end{figure}

The user now must define which files the 5SQual tool will generate and where to save them, as shown in Figure \ref{fig:ui5}. First, the user must define where to save the configuration file, which can be used for future evaluations. After, she chooses whether 5SQual should generate graphics and the final report to show the results of the evaluation, and defines in which directory to save them.

%O usuário deve então definir que arquivos a 5SQual deve gerar e onde %salvá-los, como mostrado na Figura \ref{fig:ui5}. Primeiramente, o %usuário deve definir onde salvar o arquivo de configuração, que poderá %ser útil para futuras reavaliações. Depois, ele deve escolher se deseja %que a 5SQual gere gráficos e o relatório para mostrar os resultados da %avaliação e definir em que diretório salvá-los.

\begin{figure}[ht]
    \centering{\scalebox{0.85}{\includegraphics{imgs/5SQualUI_5}\par}}
    \caption{5SQual Interface - Definition of Target for the Outputs}
    \label{fig:ui5}
\end{figure}

Before calling the configured evaluation, the user can verify a summary of the performed configuration as shown in Figure \ref{fig:ui6}. From there, she can choose to go to a previous step and redo some configurations or confirm the current ones. In case of a confirmation, the user can execute the evaluation immediately or run it later. The configuration file is generated in either case. 


%Antes de chamar a avaliação configurada, o usuário pode verificar um %resumo da configuração realizada na tela apresentada na Figura %\ref{fig:ui6}, a partir da qual pode optar por retroceder e reconfigurar %quaisquer dos parâmetros. Caso seja confirmada a configuração, o usuário %pode escolher executar a avaliação imediatamente ou postergá-la, gerando %apenas o arquivo de configuração. O arquivo de configuração será sempre %gerado, executando ou não a avaliação.


\begin{figure}[ht]
    \centering{\scalebox{0.85}{\includegraphics{imgs/5SQualUI_6}\par}}
    \caption{5SQual Interface – Confirmation of the Configuration}
    \label{fig:ui6}
\end{figure}

\newpage



\section{Examples of the Metrics}

\subsection{``Bag-of-words example''}

Assuming  that documents are represented as vectors of terms or features, the “bag-of-words” measure is defined below. For vectors $d_i$ and $d_j$, $d_i \cap d_j$ is the set of terms that are components of both $d_i$ and $d_j$. $|d_i|$ means the dimension of the vector $d_i$ (number of terms in the document). So, with respect to a reference document $d_j$, the bag-of-words measure is defined as:

\begin{equation}
bag-of-words(d_i,d_j) = \frac{|d_i \cap d_j|}{|d_j|}
\end{equation}

An example of its calculation is shown below.

\begin{itemize}
\item Consider the following excerpt from the paper \textit{``Greenstone: A Comprehensive Open-Source Digital Library Software System''} as the reference document:

\textit{``This paper describes the Greenstone digital library software, a comprehensive, open-source system for the construction and presentation of information collections.''}

\item Considere the following excerpt from the paper \textit{``The Greenstone plugin architecture''} as the document to be compared with the reference document:

\textit{``This note describes how the Greenstone digital library system uses “plugins� to import documents and metadata in different formats, and associate metadata with the appropriate documents.''}

\item The set of 8 terms that are components of both documents is: \{and, describes, digital, Greenstone, library, the, this, system\}.

\item The count of terms in the document being compared with the reference document is 22: \{and,  appropriate, associate, describes, different, digital, documents, formats, Greenstone, how, import, in, library, metadata, note, plugins, the, this, to, system, uses, with\}.

\item The similarity can then be calculated as: 8/22 = 0.36.

\end{itemize}

5SQual allows the user to specify a list of words to be excluded while evaluating content similarity. This is valuable when dealing with words like \textit{and}, \textit{this}, and \textit{to} which are very common or with words that are very common to a specific domain, like the word ``\textit{system}'' in the ``\textit{Computer Science}'' domain. 



\subsection{Bibliographic coupling and co-citation example}

Similarity measures also may use link or citation information to compute the relatedeness of two objects. 
Among the most popular citation-based measures of similarity are: co-citation and bibliographic 
coupling, which we explain below.

Two documents are co-cited if a third paper has citations to both of them. This reflects the assumption that the author of a scientific paper will cite only papers related to his own work. More formally, let $d_i$ be a digital object and let $Pd_i$ be the set of documents that cite or link to $d_i$, called the parents of $d_i$. The co-citation similarity $cocit(d_i, d_j)$  between two documents $d_i$ and $d_j$ is defined as: 
\begin{equation}
\label{cocit}
cocit(d_i,do_j) = \frac{|Pd_i \cap Pd_j|}{max |P|}
\end{equation}
where $max |P|$ is the maximum number of parents for any object in the whole collection. If both $Pd_i$ and $Pd_j$ are empty, we define 
the co-citation similarity as zero. Equation \ref{cocit} tells us that, the more parents $d_i$ and $d_j$ have in common, the more similar they are.  However, 
co-citation is a measure between pairs of digital objects. The absolute degree of co-citation of document $d_i$ in collection $C$ is defined as 
$\sum_{ d_j  \in C - \{d_i\}} 
cocit(d_i,d_j)$. 

Also with the goal of determining how related two documents are, Kessler ~\citep{kessler:63} introduced the measure of bibliographic coupling. Two 
documents share one unit of bibliographic coupling if both cite a same document. The idea is based on the notion that authors who work on the same subject tend to cite the same documents. More formally, let $d_i$ be a digital object. We define $Cd_i$ as the set of documents that $d_i$ links to, also called the children of $d_i$. Bibliographic coupling $bibcoup(d_i, d_j)$  between two pages $d_i$ and $d_j$ is defined as 
\begin{equation}
\label{bibcoup}
bibcoup(d_i, d_j) = \frac{|Cd_i \cap Cd_j|}{max |C|}
\end{equation}
where $max |C|$ is the maximum value of children for any object in the whole collection. According to Equation \ref{bibcoup}, the more 
children in common document $d_i$ has with document $d_j$, the more related they are. This value is normalized by the total set of children, to fit 
between 0 and 1. If both $Cd_i$ and $Cd_j$ are empty, we define the bibliographic coupling similarity as zero. The absolute degree of bibliographic 
coupling of a document $d_i$ in collection $C$ is defined as $\sum_{d_j \in C - \{d_i\}} bibcoup(d_i, d_j)$.

\subsubsection{Co-citation}

An example of its calculation is shown below.

\begin{itemize}

\item Given the context of the 2002 ACM colection, below we show the data about the in-links of 2 documents: oai:ACMDL:articles.359417,01-01-1977 and oai:ACMDL:articles.271137,12-01-1997 for calculating co-citation.

\begin{itemize}
\item oai:ACMDL:articles.359417,01-01-1977: 7 in-links
\item oai:ACMDL:articles.271137,12-01-1997: 26 in-links
\item The common in-links are 2: oai:ACMDL:articles.359083,03-01-1979 and oai:ACMDL:articles.804242,08-14-1978.
\end{itemize}

\item Then, the co-citation can be calculated as: 2/26 = 0.077.
\end{itemize}

\subsubsection{Bibliographic coupling}

An example of its calculation is shown below.

\begin{itemize}

\item Given the context of the 2002 ACM colection, below we show data about the out-links of 2 documents: oai:ACMDL:articles.359417,01-01-1977 and oai:ACMDL:articles.804242,08-14-1978, for calculating bibliographic coupling.

\item For bibliographic coupling we have to look at the out-links:
\begin{itemize}
\item oai:ACMDL:articles.804242,08-14-1978: 32 out-links.
\item oai:ACMDL:articles.359417,01-01-1977: 121 out-links.
\item The common out-links are 11: oai:ACMDL:articles.805003, 03-01-1972; oai:ACMDL:articles.382499, 12-01-1971; oai:ACMDL:articles.382506, 09-01-1971; oai:ACMDL:articles.803984, 12-09-1973; oai:ACMDL:articles.811140, 01-01-1975; oai:ACMDL:articles.808068, 01-01-1973; oai:ACMDL:articles.808066, 01-01-1973; oai:ACMDL:articles.811148, 01-01-1975; oai:ACMDL:articles.805012, 03-01-1972; oai:ACMDL:articles.810557, 01-01-1970 and oai:ACMDL:articles.271137, 12-01-1997.	
\end{itemize}
\item Then, the bibliographic coupling can be calculated as: 11/121 = 0.091.

\end{itemize}


\end{document}