Further documentation

Copyright and Contributions

Installation

# apt-get install libextractor0-dev extract

$ tar xzvf libextractor.x.x.x.tar.gz
$ cd libextractor.x.x.x
$ ./configure
$ make
# make install

Using the extract tool

$ wget -q http://www.copyright.gov/legislation/dmca.pdf
$ extract -b ~/dmca.pdf
% BiBTeX file
@misc{ unite2001the_d,
  title = "The Digital Millennium Copyright Act of 1998",
  author = "United States Copyright Office - jmf",
  note = "digital millennium copyright act circumvention...",
  year = "2001",
  month = "10",
  key = "Copyright Office Summary of the DMCA",
  pages = "18"
}

Another interesting option is -B"LANG. This option loads one of the language specific (but format-agnostic) plugins. These plugins attempt to find plaintext in a document by matching strings in the document against a dictionary. If the need for 200 MB of memory to compile libextractor seems mysterious, the answer lies in these plugins. In order to be able to perform a fast dictionary search, a bloomfilter is created that allows fast probabilistic matching; gcc finds the resulting datastructure a bit hard to swallow. The option -B is useful for formats that are undocumented, currently unsupported or for full-text search. Note that the printable plugins typically print the entire text of the document in order.

The supported languages at the moment are Danish (da), German (de), English (en), Spanish (es), Italian (it) and Norvegian (no). Supporting other languages is merely a question of adding (free) dictionaries in an appropriate character set. Further options are described in the extract manpage (man 1 extract).

Examples:

$ extract libextractor-0.1.3-1.src.rpm
os - linux
resource-identifier - http://ovmj.org/libextractor/
group -System Environment/Libraries
license - LGPL
copyright - LGPL
size - 251545
build-host - wedge.cs.purdue.edu
creation date - Wed Dec 25 07:50:07 2002
description - libextractor is a simple library...
summary - keyword extraction library
release - 1
version - 0.1.3
title - libextractor
unknown - SOURCE RPM 3.0
mimetype - application/x-rpm

$ extract extractor_logo.png
unknown - The libextractor logo
mimetype - image/png

The following is the output of extract for a Winword document using the plaintext extractors:

$ wget -q http://www.bayern.de/HDBG/polges.doc
$ extract -B de polges.doc | head -n 4 
unknown - FEE Politische Geschichte Bayerns
Herausgegeben vom Haus der Geschichte als Heft 
der zur Geschichte und Kultur Redaktion Manfred
 Bearbeitung Otto Copyright Haus der Geschichte
München Gestaltung fürs Internet Rudolf Inhalt im.
unknown - und das Deutsche Reich.
unknown - und seine.
unknown - Henker im Zeitalter von Reformation und Gegenreformation.

Using the libextractor library

int main(int argc, char * argv[]) {
  EXTRACTOR_ExtractorList *extractors
    = EXTRACTOR_loadDefaultLibraries();
  EXTRACTOR_KeywordList *keywords
    = EXTRACTOR_getKeywords(extractors, argv[1]);
  EXTRACTOR_printKeywords(stdout,
                          keywords);
  EXTRACTOR_freeKeywords(keywords);
  EXTRACTOR_removeAll(extractors);
}

Current Plugins

Writing new Plugins

struct EXTRACTOR_Keywords *
libextractor_XXX_extract
(char * filename,
 char * data,
 size_t size,
 struct EXTRACTOR_Keywords * prev,
 const char* options);

The argument filename specifies the name of the file that is being processed. data is a pointer to the (typically mmapped) contents of the file, and size is the filesize. Most plugins to not make use of the filename and just directly parse data directly, staring by verifying that the header of the data matches the specific format. prev is the list of keywords that have been extracted so far by other plugins for the file. The function is expected to return an updated list of keywords. The keywords are supposed to be converted into the UTF-8 character set by the plugin. If the format does not match the expectations of the plugin, prev is returned. Most plugins use a function like addKeyword to extend the list:

static void addKeyword
(struct EXTRACTOR_Keywords ** list,
 char * keyword,
 EXTRACTOR_KeywordType type)
{
  EXTRACTOR_KeywordList * next;
  next = malloc(sizeof(EXTRACTOR_KeywordList));
  next->next = *list;
  next->keyword = keyword;
  next->keywordType = type;
  *list = next;
}

A typical use of addKeyword is to add the mime-type once the file format has been established. For example, the JPEG-extractor checks the first bytes of the JPEG header and then either aborts or claims the file to be a JPEG. Note that the strdup in the code is important since the string will be deallocated later, typically in EXTRACTOR_freeKeywords(). A list of supported keyword classifications (in the example EXTRACTOR_MIMETYPE) can be found in the extractor.h header file.