- added a functional library access

This commit is contained in:
ari committed 2018-12-09 16:46:43 -05:00
1 parent 924596e38d
commit 56dd4c7ef4
13 files changed
+195 -20

No files matched your search

-3
View File
@@ -2,8 +2,5 @@ language: python
python: python:
- "3.6.4" - "3.6.4"
- "3.7" - "3.7"
install:
- pip install pycryptodome
- pip install chardet
script: script:
nosetests --nologcapture nosetests --nologcapture
+1 -1
View File
@@ -1 +1 @@
v1.1.12 v1.2.0
+15
View File
@@ -13,10 +13,24 @@
PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability. PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability.
The ultimate goal is to have a functional-programming with typing fork of the original library.
## Documentation ## Documentation
See documentation [here](https://asosnovsky.github.io/pdfmajor/). See documentation [here](https://asosnovsky.github.io/pdfmajor/).
## Basic Example
```py
from pdfmajor.extractor import extract_items_from_pdf
for page in extract_items_from_pdf('path/to/your/file.pdf'):
print('page-start', page)
for item in page:
print(' ', item)
print('page-end', page)
```
## Why Another Fork? ## Why Another Fork?
`PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with. `PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with.
@@ -24,6 +38,7 @@ See documentation [here](https://asosnovsky.github.io/pdfmajor/).
## Features ## Features
* Functional extraction method based on generators
* Written entirely in Python. * Written entirely in Python.
* Parse, analyze, and convert PDF documents. * Parse, analyze, and convert PDF documents.
* PDF-1.7 specification support. (well, almost) * PDF-1.7 specification support. (well, almost)
+57 -1
View File
@@ -2,11 +2,44 @@
The library is constructed from 4 base modules: The library is constructed from 4 base modules:
* **extractor**: contains a high-level function for functional extraction of pdf-assets
* **interpreter**: contains low-level classes for extracting fundamental data-structures from the documents * **interpreter**: contains low-level classes for extracting fundamental data-structures from the documents
* **converters**: contains high-level classes for conversion of the fundamental pdf structures to other formats * **converters**: contains high-level classes for conversion of the fundamental pdf structures to other formats
* **imagewriter**: contains a simple implementation for converting PDF Image Streams to png/bmp/img formats * **imagewriter**: contains a simple implementation for converting PDF Image Streams to png/bmp/img formats
* **layouts**: contains an abstraction layer for the data-structures found in the pdf * **layouts**: contains an abstraction layer for the data-structures found in the pdf
## extractors
Contains a high-level function for functional extraction of pdf-assets
### Example
```py
from pdfmajor.extractor import extract_items_from_pdf
for page in extract_items_from_pdf('path/to/your/file.pdf'):
print('page-start', page)
for item in page:
print(' ', item)
print('page-end', page)
```
### extract_items_from_pdf(input_file_path,maxpages,password,caching,check_extractable,pagenos,debug_level) -> generator[LTPage]
This generator-function yields individual pages which contain their respected items.
#### Arguments
- `input_file_path`: [str](str)
- `maxpages`: [int](#) defaults to 0
- `password`: [str](#) defaults to None
- `caching`: [bool](#) defaults to True
- `check_extractable`: [bool](#) defaults to True
- `pagenos`: [List[int]](#) defaults to None
- `debug_level`: [logging.levels](#https://docs.python.org/3/library/logging.html#levels) defaults logging.WARNING
#### Return Value
This function returns a generator that makes `LTPage`
## interpreter ## interpreter
WIP WIP
@@ -100,4 +133,27 @@ WIP
## layouts ## layouts
WIP WIP: this documentation set is not yet complete.
### Layout Items
All layout items extend the `LTItem` class. There are two kinds of layout items:
- LTComponent: extends the base `LTItem` class, this class will have additional values such as boundary boxes, height and width
- LTContainer: extends the `LTComponent` class, this class is used to contain elements of the pdf that would have child elements. Iterating on this element will output its child elements.
#### Layout Containers
All of these classes extend the LTContainer class.
- LTPage: a layout item containing all elements of the given page.
- LTFigure: a layout item containing LTCurves and LTImages
- LTCharBlock: a layout item containing LTChars
#### Layout Components
All of these classes extend the LTComponent class.
- LTChar: an individual character
- LTCurves: represents a collection of svg-paths (available under `self.paths`)
- LTImage: a component containing information regarding an image
+15
View File
@@ -13,10 +13,24 @@
PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability. PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability.
The ultimate goal is to have a functional-programming with typing fork of the original library.
## Documentation ## Documentation
See documentation [here](https://asosnovsky.github.io/pdfmajor/). See documentation [here](https://asosnovsky.github.io/pdfmajor/).
## Basic Example
```py
from pdfmajor.extractor import extract_items_from_pdf
for page in extract_items_from_pdf('path/to/your/file.pdf'):
print('page-start', page)
for item in page:
print(' ', item)
print('page-end', page)
```
## Why Another Fork? ## Why Another Fork?
`PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with. `PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with.
@@ -24,6 +38,7 @@ See documentation [here](https://asosnovsky.github.io/pdfmajor/).
## Features ## Features
* Functional extraction method based on generators
* Written entirely in Python. * Written entirely in Python.
* Parse, analyze, and convert PDF documents. * Parse, analyze, and convert PDF documents.
* PDF-1.7 specification support. (well, almost) * PDF-1.7 specification support. (well, almost)
+16 -1
View File
@@ -18,10 +18,24 @@
PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability. PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability.
The ultimate goal is to have a functional-programming with typing fork of the original library.
## Documentation ## Documentation
See documentation [here](https://asosnovsky.github.io/pdfmajor/). See documentation [here](https://asosnovsky.github.io/pdfmajor/).
## Basic Example
```py
from pdfmajor.extractor import extract_items_from_pdf
for page in extract_items_from_pdf('path/to/your/file.pdf'):
print('page-start', page)
for item in page:
print(' ', item)
print('page-end', page)
```
## Why Another Fork? ## Why Another Fork?
`PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with. `PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with.
@@ -29,6 +43,7 @@ See documentation [here](https://asosnovsky.github.io/pdfmajor/).
## Features ## Features
* Functional extraction method based on generators
* Written entirely in Python. * Written entirely in Python.
* Parse, analyze, and convert PDF documents. * Parse, analyze, and convert PDF documents.
* PDF-1.7 specification support. (well, almost) * PDF-1.7 specification support. (well, almost)
@@ -86,7 +101,7 @@ OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
""" """
__version__ = (1, 1, 12) __version__ = (1, 2, 0)
if __name__ == '__main__': if __name__ == '__main__':
print( print(
+12
View File
@@ -0,0 +1,12 @@
from ..utils import get_logger
log = get_logger(__name__)
from ..layouts import PDFLayoutAnalyzer, PDFPage, LTPage
class PDFExctractor(PDFLayoutAnalyzer):
def end_page(self, page: PDFPage):
assert not self._stack, str(len(self._stack))
assert isinstance(self.cur_item, LTPage), str(type(self.cur_item))
self.pageno += 1
return self.cur_item
+46
View File
@@ -0,0 +1,46 @@
import re
import os
from typing import List
from io import BytesIO, StringIO, TextIOWrapper
from tqdm import tqdm
from ..interpreter.PDFResourceManager import PDFResourceManager
from ..interpreter.PDFPage import PDFPage
from ..interpreter.PDFPageInterpreter import PDFPageInterpreter
from ..imagewriter import ImageWriter
from ..utils import set_log_level, get_logger, logging
from .PDFExtractor import PDFExctractor
log = get_logger(__name__)
def extract_items_from_pdf(
input_file_path: str,
maxpages: int = 0,
password: str = None,
caching: bool = True,
check_extractable: bool = True,
pagenos: List[int] = None,
debug_level: int = logging.WARNING,
):
set_log_level(debug_level)
log.debug("Creating resources....")
rsrcmgr = PDFResourceManager(caching=caching)
device = PDFExctractor(
rsrcmgr=rsrcmgr,
pageno=0
)
interpreter = PDFPageInterpreter(rsrcmgr, device)
log.debug("Reading pages....")
with open(input_file_path, 'rb') as input_file:
pages = PDFPage.get_pages(input_file, pagenos=pagenos, maxpages=maxpages, password=password, caching=caching, check_extractable=check_extractable)
page_waiter = tqdm(enumerate(pages), disable=debug_level > logging.INFO)
for _, page in page_waiter:
page_waiter.set_description("Processing pdf...")
yield interpreter.process_page(page)
page_waiter.close()
device.close()
+1 -2
View File
@@ -624,8 +624,7 @@ class PDFPageInterpreter(object):
ctm = (1, 0, 0, 1, -x0, -y0) ctm = (1, 0, 0, 1, -x0, -y0)
self.device.begin_page(page, ctm) self.device.begin_page(page, ctm)
self.render_contents(page.resources, page.contents, ctm=ctm) self.render_contents(page.resources, page.contents, ctm=ctm)
self.device.end_page(page) return self.device.end_page(page)
return
# render_contents(resources, streams, ctm) # render_contents(resources, streams, ctm)
# Render the content streams. # Render the content streams.
+6 -9
View File
@@ -1,6 +1,6 @@
from typing import List from typing import List
from ._base import LTComponent from ._base import LTContainer
from .LTChar import LTChar from .LTChar import LTChar
from ..utils import apply_matrix_norm, apply_matrix_pt, matrix2str, Bbox, Point from ..utils import apply_matrix_norm, apply_matrix_pt, matrix2str, Bbox, Point
@@ -11,7 +11,7 @@ from ..interpreter.PDFTextState import PDFTextState
## LTCharBlock ## LTCharBlock
## ##
class LTCharBlock(LTComponent): class LTCharBlock(LTContainer):
def __init__(self, def __init__(self,
chars: List[Point], chars: List[Point],
@@ -20,7 +20,7 @@ class LTCharBlock(LTComponent):
graphicstate: PDFGraphicState graphicstate: PDFGraphicState
): ):
((x0, y0), (x1, y1)) = chars[0][1] ((x0, y0), (x1, y1)) = chars[0][1]
self.chars = [ ltchars = [
LTChar( LTChar(
bbox=Bbox(x0, y0, x1, y1), bbox=Bbox(x0, y0, x1, y1),
char= chars[0][0], char= chars[0][0],
@@ -36,7 +36,7 @@ class LTCharBlock(LTComponent):
x1 = max([ x0, x1, nx0, nx1 ]) x1 = max([ x0, x1, nx0, nx1 ])
y1 = max([ y0, y1, ny0, ny1 ]) y1 = max([ y0, y1, ny0, ny1 ])
y0 = min([ y0, y1, ny0, ny1 ]) y0 = min([ y0, y1, ny0, ny1 ])
self.chars.append(LTChar( ltchars.append(LTChar(
bbox=Bbox(nx0, ny0, nx1, ny1), bbox=Bbox(nx0, ny0, nx1, ny1),
char= char[0], char= char[0],
textstate=textstate, textstate=textstate,
@@ -44,7 +44,7 @@ class LTCharBlock(LTComponent):
graphicstate=graphicstate graphicstate=graphicstate
)) ))
LTComponent.__init__(self, Bbox(x0, y0, x1, y1)) LTContainer.__init__(self, Bbox(x0, y0, x1, y1), ltchars)
self.font = textstate.font self.font = textstate.font
self.textstate = textstate self.textstate = textstate
@@ -60,11 +60,8 @@ class LTCharBlock(LTComponent):
def __repr__(self): def __repr__(self):
return f"""<{self.__class__.__name__} font="{self.fontname}" text="{self.get_text()}"/>""" return f"""<{self.__class__.__name__} font="{self.fontname}" text="{self.get_text()}"/>"""
def __iter__(self):
return iter(self.chars)
def get_text(self): def get_text(self):
return "".join([ c.get_text() for c in self.chars ]) return "".join([ c.get_text() for c in self ])
@property @property
def fontname(self) -> str: def fontname(self) -> str:
+2 -2
View File
@@ -90,9 +90,9 @@ class LTComponent(LTItem):
## ##
class LTContainer(LTComponent): class LTContainer(LTComponent):
def __init__(self, bbox: Bbox): def __init__(self, bbox: Bbox, objs: List[LTItem] = []):
LTComponent.__init__(self, bbox) LTComponent.__init__(self, bbox)
self._objs: List[LTItem] = [] self._objs: List[LTItem] = objs
return return
def __iter__(self): def __iter__(self):
+23
View File
@@ -0,0 +1,23 @@
#!/usr/bin/env python
import re
import os
import logging
from typing import List
from pdfmajor.converters import convert_file
from xml.etree import ElementTree
from tqdm import tqdm
from pdfmajor.extractor import extract_items_from_pdf
from pdfmajor.layouts import LTPage
def rel_path(*arg: List[str]) -> str:
return os.path.join(os.path.dirname(__file__), *arg)
print(extract_items_from_pdf(rel_path('pdf/chars.pdf')))
# for page in extract_items_from_pdf(rel_path('pdf/chars.pdf')):
# print('page-start', page)
# for item in page:
# print(' ', item)
# print('page-end', page)
+1 -1
View File
@@ -10,7 +10,7 @@ import sys
with open('README.md') as readme_file: with open('README.md') as readme_file:
SETUP = { SETUP = {
"name": "pdfmajor", "name": "pdfmajor",
"version": "1.1.12", "version": "1.2.0",
"packages": find_packages(include=['pdfmajor*']), "packages": find_packages(include=['pdfmajor*']),
"install_requires": [ "install_requires": [
'pycryptodome', 'pycryptodome',