- updated docs

- added text conversion
This commit is contained in:
ari committed 2018-12-22 23:32:11 -05:00
1 parent 0590d8c8f9
commit b8b167e827
9 files changed
+129 -472

No files matched your search

+10 -14
View File
@@ -11,9 +11,7 @@
</tr>
</table>
PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability.
The ultimate goal is to have a functional-programming with typing fork of the original library.
PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. The ultimate goal of this project is to provide a simpler, faster and more functional library that both casual and low-level developers could build upon.
## Documentation
@@ -21,34 +19,32 @@ See documentation [here](https://asosnovsky.github.io/pdfmajor/).
## Basic Example
```py
from pdfmajor.extractor import extract_items_from_pdf
from pdfmajor.interpreter import PageInterpreter
for page in extract_items_from_pdf('path/to/your/file.pdf'):
print('page-start', page)
for page in PDFInterpreter("/path/to/pdf.pdf"):
print("page start", page.page_num)
for item in page:
print(' ', item)
print('page-end', page)
print(" >", item)
print("page end", page.page_num)
```
## Why Another Fork?
`PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with.
`PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. However, after attempting to build and extend upon `PDFMiner.six` I found it rather difficult to extend and identify additional details regarding the items encoded in the pdf. This library will attempt to expose as much information as possible to end-users, without having them solely rely on to-xml or to-json exports.
## Features
* Functional extraction method based on generators
* Written entirely in Python.
* Parse, analyze, and convert PDF documents.
* PDF-1.7 specification support. (well, almost)
* Font-Color extraction
* Shape fill and stroke color extraction
* CJK languages and vertical writing scripts support.
* Various font types (Type1, TrueType, Type3, and CID) support.
* Basic encryption (RC4) support.
* Font-Color extraction
* Shape fill and stroke color extraction
**Note**: We took out the layout-analysis process in this version (there is no more LTTextHorizontal or LTTextVertical). While the mathematics behind the grouping process was sound, the coupling of the layout-analysis process with the parsing and interpretation process produced unfriendly-code. This feature could be brought back by extending the `pdfmajor.layouts.PDFLayoutAnalyzer` class, but at the current time is not supported.
**Note**: We took out the layout-analysis process in this version (there is no more LTTextHorizontal or LTTextVertical). While the mathematics behind the grouping process was sound, the coupling of the layout-analysis process with the parsing and interpretation process produced unfriendly-code. This feature could be brought back by running your own implementation of it on the `pdfmajor.interpreter.PageInterpreter` class, but at the current time is not supported.
## How to Install
+54 -103
View File
@@ -2,34 +2,37 @@
The library is constructed from 4 base modules:
* **extractor**: contains a high-level function for functional extraction of pdf-assets
* **interpreter**: contains low-level classes for extracting fundamental data-structures from the documents
* **converters**: contains high-level classes for conversion of the fundamental pdf structures to other formats
* **parser**: contains low-level classes for extracting fundamental data-structures from the documents
* **interpreter**: a interpreter of the pdf-standard commands
* **converters**: contains high-level functions for conversion of the fundamental pdf structures to other formats
* **imagewriter**: contains a simple implementation for converting PDF Image Streams to png/bmp/img formats
* **layouts**: contains an abstraction layer for the data-structures found in the pdf
## extractors
Contains a high-level function for functional extraction of pdf-assets
## parser
WIP
## interpreter
a interpreter of the pdf-standard commands
### Example
```py
from pdfmajor.extractor import extract_items_from_pdf
from pdfmajor.interpreter import PageInterpreter
for page in extract_items_from_pdf('path/to/your/file.pdf'):
print('page-start', page)
for page in PDFInterpreter("/path/to/pdf.pdf"):
print("page start", page.page_num)
for item in page:
print(' ', item)
print('page-end', page)
print(" >", item)
print("page end", page.page_num)
```
### extract_items_from_pdf(input_file_path,maxpages,password,caching,check_extractable,pagenos,debug_level) -> generator[LTPage]
### interpreter.PDFInterpreter
This generator-function yields individual pages which contain their respected items.
#### Arguments
- `input_file_path`: [str](str)
- `input_file_path`: [str](#)
- `preload`: [bool](#) defaults to False
- `maxpages`: [int](#) defaults to 0
- `password`: [str](#) defaults to None
- `caching`: [bool](#) defaults to True
@@ -37,39 +40,57 @@ This generator-function yields individual pages which contain their respected it
- `pagenos`: [List[int]](#) defaults to None
- `debug_level`: [logging.levels](#https://docs.python.org/3/library/logging.html#levels) defaults logging.WARNING
#### Return Value
This function returns a generator that makes `LTPage`
#### Yield Value
This function returns a generator that yields [PDFInterpreter](#interpreterpageinterpreter).
## interpreter
WIP
### interpreter.PageInterpreter
This generator-function-class yields individual [layout items](#layout-items).
### Layout Items
All layout items extend the `LTItem` class. There are two kinds of layout items:
- LTComponent: extends the base `LTItem` class, this class will have additional values such as boundary boxes, height and width
- LTContainer: extends the `LTComponent` class, this class is used to contain elements of the pdf that would have child elements. Iterating on this element will output its child elements.
#### Layout Containers
All of these classes extend the LTContainer class.
- LTXObject: a layout item containing other additional layout items
- LTCharBlock: a layout item containing LTChars
#### Layout Components
All of these classes extend the LTComponent class.
- LTChar: an individual character
- LTCurves: represents a collection of svg-paths (available under `self.paths`)
- LTImage: a component containing information regarding an image
## converters
Contains high-level classes for conversion of the fundamental pdf structures to other formats. This library includes 4 high-level conversion classes:
Contains high-level functions for conversion of the fundamental pdf structures to other formats. This library includes 4 high-level conversion cases:
- HTMLConverter
- JSONConverter
- XMLConverter
- TextConverter
- HTML
- JSON
- XML
- Text
These classes all extend the [PDFConverter](#converters.PDFConverter). To use them simply call the static method [parse_file](#converterspdfconverterparse_fileinput_file-output_file-image_folder_path-codec-maxpages-password-caching-check_extractable-pagenos).
These formats are all generated using the [PageInterpreter](#PDFInterpreter). To use them simply call the static method [parse_file](#converterspdfconverter).
### Example
```py
from pdfmajor.converters import HTMLConverter
from pdfmajor.converters import convert_file
input_file = open('path/to/input/file.pdf', 'rb')
output_file = open('path/to/output/file.html', 'wb')
HTMLConverter.parse_file(
input_file,
output_file
convert_file(
"path/to/input/file.pdf",
"path/to/output/file.html",
out_type="html"
)
input_file.close()
output_file.close()
```
### converters.convert_file(input_file, output_file, image_folder_path, codec, maxpages, password, caching, check_extractable, pagenos, out_type)
### converters.convert_file
A high-level abstraction for the conversion classes.
@@ -84,76 +105,6 @@ A high-level abstraction for the conversion classes.
- `pagenos`: [List[int]](#) defaults to None
- `out_type`: [str](#) defaults to 'html'
### Example (converters.convert_file)
```py
from pdfmajor.converters import convert_file
input_file = open('path/to/input/file.pdf', 'rb')
output_file = open('path/to/output/file.html', 'wb')
convert_file(
input_file,
output_file
)
input_file.close()
output_file.close()
```
### converters.PDFConverter
Extends `layouts.PDFLayoutAnalyzer`.
#### converters.PDFConverter.\_\_init\_\_(rsrcmgr, outfp, imagewriter, codec, pageno)
- `rsrcmgr`: [PDFResourceManager](#interpreter.PDFResourceManager)
- `outfp`: [TextIOWrapper](https://docs.python.org/3/library/io.html#io.TextIOWrapper)
- `imagewriter`: [ImageWriter](#imagewriter.ImageWriter)
- `codec`: [str](#) defaults to 'utf-8'
- `pageno`: [int](#) defaults to 1
initialization function for the class
#### converters.PDFConverter.parse_file(input_file, output_file, image_folder_path, codec, maxpages, password, caching, check_extractable, pagenos)
returns the output file [TextIOWrapper](https://docs.python.org/3/library/io.html#io.TextIOWrapper).
- `input_file`: [TextIOWrapper](https://docs.python.org/3/library/io.html#io.TextIOWrapper)
- `output_file`: [TextIOWrapper](https://docs.python.org/3/library/io.html#io.TextIOWrapper)
- `image_folder_path`: [str](#) defaults to None
- `codec`: [str](#) defaults to 'utf-8'
- `maxpages`: [int](#) defaults to 0
- `password`: [str](#) defaults to None
- `caching`: [bool](#) defaults to True
- `check_extractable`: [bool](#) defaults to True
- `pagenos`: [List[int]](#) defaults to None
## imagewriter
WIP
## layouts
WIP: this documentation set is not yet complete.
### Layout Items
All layout items extend the `LTItem` class. There are two kinds of layout items:
- LTComponent: extends the base `LTItem` class, this class will have additional values such as boundary boxes, height and width
- LTContainer: extends the `LTComponent` class, this class is used to contain elements of the pdf that would have child elements. Iterating on this element will output its child elements.
#### Layout Containers
All of these classes extend the LTContainer class.
- LTPage: a layout item containing all elements of the given page.
- LTFigure: a layout item containing LTCurves and LTImages
- LTCharBlock: a layout item containing LTChars
#### Layout Components
All of these classes extend the LTComponent class.
- LTChar: an individual character
- LTCurves: represents a collection of svg-paths (available under `self.paths`)
- LTImage: a component containing information regarding an image
+10 -14
View File
@@ -11,9 +11,7 @@
</tr>
</table>
PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability.
The ultimate goal is to have a functional-programming with typing fork of the original library.
PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. The ultimate goal of this project is to provide a simpler, faster and more functional library that both casual and low-level developers could build upon.
## Documentation
@@ -21,34 +19,32 @@ See documentation [here](https://asosnovsky.github.io/pdfmajor/).
## Basic Example
```py
from pdfmajor.extractor import extract_items_from_pdf
from pdfmajor.interpreter import PageInterpreter
for page in extract_items_from_pdf('path/to/your/file.pdf'):
print('page-start', page)
for page in PDFInterpreter("/path/to/pdf.pdf"):
print("page start", page.page_num)
for item in page:
print(' ', item)
print('page-end', page)
print(" >", item)
print("page end", page.page_num)
```
## Why Another Fork?
`PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with.
`PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. However, after attempting to build and extend upon `PDFMiner.six` I found it rather difficult to extend and identify additional details regarding the items encoded in the pdf. This library will attempt to expose as much information as possible to end-users, without having them solely rely on to-xml or to-json exports.
## Features
* Functional extraction method based on generators
* Written entirely in Python.
* Parse, analyze, and convert PDF documents.
* PDF-1.7 specification support. (well, almost)
* Font-Color extraction
* Shape fill and stroke color extraction
* CJK languages and vertical writing scripts support.
* Various font types (Type1, TrueType, Type3, and CID) support.
* Basic encryption (RC4) support.
* Font-Color extraction
* Shape fill and stroke color extraction
**Note**: We took out the layout-analysis process in this version (there is no more LTTextHorizontal or LTTextVertical). While the mathematics behind the grouping process was sound, the coupling of the layout-analysis process with the parsing and interpretation process produced unfriendly-code. This feature could be brought back by extending the `pdfmajor.layouts.PDFLayoutAnalyzer` class, but at the current time is not supported.
**Note**: We took out the layout-analysis process in this version (there is no more LTTextHorizontal or LTTextVertical). While the mathematics behind the grouping process was sound, the coupling of the layout-analysis process with the parsing and interpretation process produced unfriendly-code. This feature could be brought back by running your own implementation of it on the `pdfmajor.interpreter.PageInterpreter` class, but at the current time is not supported.
## How to Install
-183
View File
@@ -1,183 +0,0 @@
import logging
from typing import Dict, Union
from io import BytesIO
from contextlib import contextmanager
from ..layouts import LTPage
from ..layouts import LTItem
from ..layouts import LTCurve
from ..layouts import LTLine
from ..layouts import LTRect
from ..layouts import LTFigure
from ..layouts import LTImage
from ..layouts import LTChar, LTCharBlock
from ..utils import enc
from .PDFConverter import PDFConverter
from ..interpreter.PDFGraphicState import PDFGraphicStateColor
log = logging.getLogger(__name__)
def get_color(col: PDFGraphicStateColor):
color_type, color_val = col.get_color()
if color_type is not None:
if color_type == 'rgb':
return f"rgb({','.join(map(str, color_val))})"
elif color_type == 'cmyk':
return f"cmyk({','.join(map(str, color_val))})"
elif color_type == 'gray':
return f'gray({color_val})'
return ""
class JSONConverter(PDFConverter):
def __init__(self,
rsrcmgr,
outfp: BytesIO,
codec: str='utf-8',
pageno: int=1,
imagewriter=None,
):
PDFConverter.__init__(self, rsrcmgr, outfp, imagewriter=imagewriter, codec=codec, pageno=pageno)
self.__last_levels_deep:int = None
self.__levels_deep = 0
self.write_header()
@property
def _levels_deep(self): return self.__levels_deep
@_levels_deep.setter
def _levels_deep(self, val: int):
self.__last_levels_deep = self.__levels_deep
self.__levels_deep = val
def write(self, text: str):
if self.__last_levels_deep == self._levels_deep:
text = ',' + text
self.write_raw(text)
def write_raw(self, text):
if self.codec:
text = text.encode(self.codec)
self.outfp.write(text)
self._levels_deep = self._levels_deep
def write_header(self):
self.write("{")
self._levels_deep += 1
if self.codec:
self.write('"encoding": "%s"' % self.codec)
self.write('"pages": [')
self._levels_deep += 1
def write_footer(self):
self._levels_deep -= 1
self.write(']')
self._levels_deep -= 1
self.write('}')
@contextmanager
def place_elm_with_child(self, tag_name: str, attr_dict: Dict[str, str] = {}):
attrs = ", ".join([
f'"{name}": "{value}"'
for name, value in attr_dict.items()
])
self.write('{ "type": "%s", "attrs": { %s } , "children": [' % (tag_name, attrs) )
self._levels_deep += 1
yield
self._levels_deep -= 1
self.write("]}")
def place_elm_close(self, tag_name: str, attr_dict: Dict[str, str] = {}):
attrs = ", ".join([
f'"{name}": "{value}"'
for name, value in attr_dict.items()
])
self.write('{ "type": "%s", "attrs": { %s } }' % (tag_name, attrs) )
def place_image(self, item: LTImage):
name = item.name
if self.imagewriter is not None:
name = self.imagewriter.export_image(item)
self.place_elm_close('img', {
'src': enc(name, None),
"x0": item.x0,
"x1": item.x1,
"y0": item.y0,
"y1": item.y1,
})
def render_curve(self, item: LTCurve):
attr = {
"x0": item.x0,
"x1": item.x1,
"y0": item.y0,
"y1": item.y1,
"stroke-color": get_color(item.stroke),
"fill-color": get_color(item.fill),
}
if isinstance(item, LTRect):
return self.place_elm_close('rect', attr)
elif isinstance(item, LTLine):
return self.place_elm_close('line', attr)
with self.place_elm_with_child('curve', attr):
for path in item.paths:
with self.place_elm_with_child('path', { "type": path.method.name }):
for point in path.points:
self.place_elm_close('point', { 'x': point.x, 'y': point.y })
def receive_layout(self, ltpage: LTPage):
def render(item: LTItem):
if isinstance(item, LTPage):
with self.place_elm_with_child('page', {
"width": item.width,
"height": item.height
}):
for child in item:
render(child)
elif isinstance(item, LTCurve):
self.render_curve(item)
elif isinstance(item, LTFigure):
with self.place_elm_with_child('figure', {
"x0": item.x0,
"x1": item.x1,
"y0": item.y0,
"y1": item.y1,
}):
for child in item:
render(child)
elif isinstance(item, LTImage):
self.place_image(item)
elif isinstance(item, LTChar):
self.place_elm_close('char', {
'font-size': item.size,
'font-family': item.fontname,
"x0": item.x0,
"x1": item.x1,
"y0": item.y0,
"y1": item.y1,
"stroke-color": get_color(item.graphicstate.scolor),
"fill-color": get_color(item.graphicstate.ncolor),
"text": item.get_text()
})
elif isinstance(item, LTCharBlock):
self.place_elm_close('char-block', {
'font-size': item.size,
'font-family': item.fontname,
"x0": item.x0,
"x1": item.x1,
"y0": item.y0,
"y1": item.y1,
"stroke-color": get_color(item.graphicstate.scolor),
"fill-color": get_color(item.graphicstate.ncolor),
"text": item.get_text()
})
render(ltpage)
def close(self):
self.write_footer()
-87
View File
@@ -1,87 +0,0 @@
import re
import os
from typing import List
from io import BytesIO, StringIO, TextIOWrapper
from tqdm import tqdm
from ..layouts import PDFLayoutAnalyzer
from ..interpreter.PDFResourceManager import PDFResourceManager
from ..interpreter.PDFPage import PDFPage
from ..interpreter.PDFPageInterpreter import PDFPageInterpreter
from ..imagewriter import ImageWriter
from ..utils import set_log_level, get_logger, logging
log = get_logger(__name__)
## PDFConverter
##
class PDFConverter(PDFLayoutAnalyzer):
@classmethod
def parse_file(cls,
input_file: TextIOWrapper,
output_file: TextIOWrapper,
image_folder_path: str = None,
codec: str = 'utf-8',
maxpages: int = 0,
password: str = None,
caching: bool = True,
check_extractable: bool = True,
pagenos: List[int] = None,
dont_export_images: bool = False,
debug_level: int = logging.WARNING,
) -> TextIOWrapper:
set_log_level(debug_level)
if dont_export_images:
image_folder_path = None
if not dont_export_images and image_folder_path is None and output_file.name is not None:
image_folder_path = os.path.dirname(output_file.name)
log.debug("Creating resources....")
rsrcmgr = PDFResourceManager(caching=caching)
imagewriter=ImageWriter(image_folder_path)
device = cls(
rsrcmgr,
output_file,
codec=codec,
imagewriter=imagewriter
)
interpreter = PDFPageInterpreter(rsrcmgr, device)
log.debug("Reading pages....")
pages = PDFPage.get_pages(input_file, pagenos=pagenos, maxpages=maxpages, password=password, caching=caching, check_extractable=check_extractable)
page_waiter = tqdm(enumerate(pages), disable=debug_level > logging.INFO)
for _, page in page_waiter:
page_waiter.set_description("Processing pdf...")
interpreter.process_page(page)
page_waiter.close()
device.close()
return output_file
def __init__(self, rsrcmgr: PDFResourceManager, outfp: TextIOWrapper, imagewriter: ImageWriter, codec: str='utf-8', pageno: int=1):
PDFLayoutAnalyzer.__init__(self, rsrcmgr, pageno=pageno)
self.imagewriter = imagewriter
self.outfp = outfp
self.codec = codec
if hasattr(self.outfp, 'mode'):
if 'b' in self.outfp.mode:
self.outfp_binary = True
else:
self.outfp_binary = False
else:
if isinstance(self.outfp, BytesIO):
self.outfp_binary = True
elif isinstance(self.outfp, StringIO):
self.outfp_binary = False
else:
try:
self.outfp.write(u"é")
self.outfp_binary = False
except TypeError:
self.outfp_binary = True
return
-52
View File
@@ -1,52 +0,0 @@
import logging
from typing import Dict
from contextlib import contextmanager
from ..layouts import LTPage
from ..layouts import LTItem
from ..layouts import LTCurve
from ..layouts import LTLine
from ..layouts import LTRect
from ..layouts import LTFigure
from ..layouts import LTImage
from ..layouts import LTChar, LTCharBlock
from ..interpreter.PDFGraphicState import PDFGraphicStateColor
from ..utils import enc
from .PDFConverter import PDFConverter
log = logging.getLogger(__name__)
class TextConverter(PDFConverter):
def __init__(self,
rsrcmgr,
outfp,
codec='utf-8',
pageno=1,
imagewriter=None,
):
PDFConverter.__init__(self, rsrcmgr, outfp, imagewriter=imagewriter, codec=codec, pageno=pageno)
def write(self, text: str):
if self.codec:
text = text.encode(self.codec)
self.outfp.write(text)
def place_text(self, char: LTChar):
self.write(char.get_text())
def receive_layout(self, ltpage: LTPage):
def render(item: LTItem):
if isinstance(item, LTPage):
self.write(f'\n-----[Page {self.pageno-1}]-----\n')
for child in item:
render(child)
self.write(f'\n-----[Page {self.pageno-1}]-----\n')
elif isinstance(item, LTFigure):
for child in item:
render(child)
elif isinstance(item, LTChar) or isinstance(item, LTCharBlock):
self.place_text(item)
render(ltpage)
+15 -18
View File
@@ -7,10 +7,7 @@ from typing import List
from .html import convert_to_html
from .xml import convert_to_xml
from .json import convert_to_json
# from .HTMLConverter import HTMLConverter
# from .JSONConverter import JSONConverter
# from .XMLConverter import XMLConverter
# from .TextConverter import TextConverter
from .text import convert_to_text
from ..utils import logging
def convert_file(
@@ -78,18 +75,18 @@ def convert_file(
dont_export_images=dont_export_images,
debug_level=debug_level,
)
# elif out_type == 'text':
# return TextConverter.parse_file(
# input_file=input_file,
# output_file=output_file,
# image_folder_path=image_folder_path,
# codec=codec,
# maxpages=maxpages,
# password=password,
# caching=caching,
# check_extractable=check_extractable,
# pagenos=pagenos,
# dont_export_images=dont_export_images,
# debug_level=debug_level,
# )
elif out_type == 'text':
return convert_to_text(
input_file_path=input_file,
output_file_path=output_file,
image_folder_path=image_folder_path,
codec=codec,
maxpages=maxpages,
password=password,
caching=caching,
check_extractable=check_extractable,
pagenos=pagenos,
dont_export_images=dont_export_images,
debug_level=debug_level,
)
else: raise Exception("Please specify out_type as 'html' or 'xml' or 'json' or 'text'")
+39
View File
@@ -0,0 +1,39 @@
from typing import List
from ..interpreter import PDFInterpreter, PageInterpreter, logging
from ..interpreter import LTCharBlock, LTXObject
def convert_to_text(
input_file_path: str,
output_file_path: str,
image_folder_path: str = None,
dont_export_images: bool = False,
codec: str = 'utf-8',
maxpages: int = 0,
password: str = None,
caching: bool = True,
check_extractable: bool = True,
pagenos: List[int] = None,
debug_level: int = logging.WARNING,
):
intepreter = PDFInterpreter(input_file_path,
maxpages=maxpages,
password=password,
caching=caching,
check_extractable=check_extractable,
pagenos=pagenos,
debug_level=debug_level
)
with open(output_file_path, 'wb') as outfp:
def process_container(container: LTXObject):
for item in container:
if isinstance(item, LTCharBlock):
for char in item:
outfp.write(char.get_text().encode(codec))
elif isinstance(item, LTXObject):
process_container(item)
for page in intepreter:
outfp.write(f"========== [ page {page.page_num} ] ==========\n".encode(codec))
process_container(page)
outfp.write('\n'.encode(codec))
@@ -12,7 +12,7 @@ from tqdm import tqdm
def rel_path(*arg: List[str]) -> str:
return os.path.normpath(os.path.join(os.path.dirname(__file__), *arg))
EXTS = ['json','xml','html']
EXTS = ['text', 'json','xml','html']
INPUT_FOLDER = rel_path("./samples/pdf")
OUTPUT_FOLDER = rel_path(f"./samples/output/")