By default this project uses LSA to select 3 sentences to summarise the document, but this can be changed using the plugin UI. This plugin provides an interface for three popular text summarisation algorithms including TextRank, KL-Sum and LSA. The text summarisation was performed using Dataiku's Text Summarisation plugin ( documentation). Full details of the model are available here. The model uses a multiclass logistic regression classifier built on top of a tf-idf transformer and achieves an AUC of 0.995. Model for Document ClassificationĪs described above, the model was trained on the 20 newsgroups dataset available from sci-kit learn here. The OCR is then performed using Google's open-source Tesseract library ( documentation) here. The PDFs are first converted to image files using pdf2image ( documentation) here. Project Components OCR for Text Extraction The document classification model was trained on the 20 newsgroups dataset ( documentation) which is publicly available from sci-kit learn as described here. DataĪrbitrary PDF files can be placed into the input folder for classfication and summarisation. Please see below for further information. The scenario to process the input PDFs can be triggered from here, and the text file contatining the classification, summary and full text will appear in the folder on the right hand side. Documents can be uploaded by dragging and dropping onto the left hand side of the dashboard.

The dashboard shows a view of the input PDFs and output files. This project takes PDF files as input, uses Optical Character Recognition (OCR) to extract the text for each input PDF, classifies the document into one of twenty article categories and summarises the text to an arbitrary number of sententences (by default 3 using Latent Semantic Analysis (LSA)), before composing the document title, classification, summary and original text into a text file which is emailed to the user.

Project to automatically read, summarise and classify PDFs.

Automatic pdf processor