22 Commits
Author SHA1 Message Date
ari 56dd4c7ef4 - added a functional library access 2018-12-09 16:46:43 -05:00
ari 924596e38d - added consistency to PDFGraphicStateCol 2018-11-29 15:39:16 -05:00
ari 2ea0b717f9 - added better logging options
- added test case for bad chars
2018-11-23 14:44:09 -05:00
ari 38f8ca76a6 - better loggin system
- adde test case for bad characters
2018-11-23 14:43:51 -05:00
ari ac49ad1bcf - sanitizing the xml chars 2018-11-21 09:19:33 -05:00
ari d74bf1eaa9 - added support for ICCSBAsed colors 2018-11-09 10:24:04 -05:00
ari cf3c89bf25 supporting ICCSBased colors 2018-11-09 10:22:44 -05:00
ari b491cd2181 - v1.1.8 2018-11-08 14:53:28 -05:00
ari 70144c8fe1 - updated docx
- sanitizing xml attributes
2018-11-04 16:28:29 -05:00
ari 0d832bf7fa version inc 2018-10-31 14:45:37 -04:00
ari 7319b7f584 xml chars output bbx instead of width/height 2018-10-31 14:45:02 -04:00
ari 5b641d6243 comment out leading for now 2018-10-29 16:35:32 -04:00
ari 10d679d379 clean up 2018-10-29 16:26:57 -04:00
ari 5378a4f34c - better color grabbing for scn
- show custom type in converters
2018-10-29 16:25:50 -04:00
ari 42cbc8e39e - using snake case
- not outputting is_bold
2018-10-29 13:18:12 -04:00
ari 66004988f4 actually cleaning it all prior to deployment 2018-10-29 13:04:32 -04:00
ari 7722b99f70 Deployong new version - v1.1.4 2018-10-29 13:03:28 -04:00
ari 66b5fea65c - compatible xml-converter with char-blocks
- consitent attr naming convention for fonts
2018-10-29 13:03:11 -04:00
ari 659b19dae9 Deployong new version - v1.1.3 2018-10-29 12:51:48 -04:00
ari 337c1b4a76 - pulling out boldness from font
- better html converter
2018-10-29 12:51:34 -04:00
ari 76deefb9dc Deployong new version - v1.1.2 2018-10-28 23:52:59 -04:00
ari e51618231d testing new version ctrl 2018-10-28 23:50:51 -04:00
33 changed files with 577 additions and 198 deletions

No files matched your search

-3
View File
@@ -2,8 +2,5 @@ language: python
python: python:
- "3.6.4" - "3.6.4"
- "3.7" - "3.7"
install:
- pip install pycryptodome
- pip install chardet
script: script:
nosetests --nologcapture nosetests --nologcapture
+1
View File
@@ -0,0 +1 @@
v1.2.0
+27 -1
View File
@@ -1,11 +1,36 @@
# PDFMajor # PDFMajor
<table>
<tr>
<td>Latest Release</td>
<td>
<a href="https://pypi.org/project/pdfmajor/">
<img src="https://img.shields.io/pypi/v/pdfmajor.svg" alt="latest release" />
</a>
</td>
</tr>
</table>
PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability. PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability.
The ultimate goal is to have a functional-programming with typing fork of the original library.
## Documentation ## Documentation
See documentation [here](https://asosnovsky.github.io/pdfmajor/). See documentation [here](https://asosnovsky.github.io/pdfmajor/).
## Basic Example
```py
from pdfmajor.extractor import extract_items_from_pdf
for page in extract_items_from_pdf('path/to/your/file.pdf'):
print('page-start', page)
for item in page:
print(' ', item)
print('page-end', page)
```
## Why Another Fork? ## Why Another Fork?
`PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with. `PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with.
@@ -13,6 +38,7 @@ See documentation [here](https://asosnovsky.github.io/pdfmajor/).
## Features ## Features
* Functional extraction method based on generators
* Written entirely in Python. * Written entirely in Python.
* Parse, analyze, and convert PDF documents. * Parse, analyze, and convert PDF documents.
* PDF-1.7 specification support. (well, almost) * PDF-1.7 specification support. (well, almost)
+9 -1
View File
@@ -4,7 +4,15 @@ WK_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
cd $WK_DIR cd $WK_DIR
cd .. cd ..
sh bin/clean/sh ./bin/clean.sh
python setup.py sdist bdist_wheel python setup.py sdist bdist_wheel
git add .
git commit
git push
git tag $(cat .version)
git push -u origin $(cat .version)
twine upload dist/* twine upload dist/*
+4
View File
@@ -27,3 +27,7 @@ def update_pdfmajor__init___(version: str, doc: str):
long_description=doc, long_description=doc,
version_tuple=re.sub(r"\.", ", ", version, count=2) version_tuple=re.sub(r"\.", ", ", version, count=2)
)) ))
def update_current_version_lock(version: str):
with open(".version", 'w') as file:
file.write("v"+version)
+57 -1
View File
@@ -2,11 +2,44 @@
The library is constructed from 4 base modules: The library is constructed from 4 base modules:
* **extractor**: contains a high-level function for functional extraction of pdf-assets
* **interpreter**: contains low-level classes for extracting fundamental data-structures from the documents * **interpreter**: contains low-level classes for extracting fundamental data-structures from the documents
* **converters**: contains high-level classes for conversion of the fundamental pdf structures to other formats * **converters**: contains high-level classes for conversion of the fundamental pdf structures to other formats
* **imagewriter**: contains a simple implementation for converting PDF Image Streams to png/bmp/img formats * **imagewriter**: contains a simple implementation for converting PDF Image Streams to png/bmp/img formats
* **layouts**: contains an abstraction layer for the data-structures found in the pdf * **layouts**: contains an abstraction layer for the data-structures found in the pdf
## extractors
Contains a high-level function for functional extraction of pdf-assets
### Example
```py
from pdfmajor.extractor import extract_items_from_pdf
for page in extract_items_from_pdf('path/to/your/file.pdf'):
print('page-start', page)
for item in page:
print(' ', item)
print('page-end', page)
```
### extract_items_from_pdf(input_file_path,maxpages,password,caching,check_extractable,pagenos,debug_level) -> generator[LTPage]
This generator-function yields individual pages which contain their respected items.
#### Arguments
- `input_file_path`: [str](str)
- `maxpages`: [int](#) defaults to 0
- `password`: [str](#) defaults to None
- `caching`: [bool](#) defaults to True
- `check_extractable`: [bool](#) defaults to True
- `pagenos`: [List[int]](#) defaults to None
- `debug_level`: [logging.levels](#https://docs.python.org/3/library/logging.html#levels) defaults logging.WARNING
#### Return Value
This function returns a generator that makes `LTPage`
## interpreter ## interpreter
WIP WIP
@@ -100,4 +133,27 @@ WIP
## layouts ## layouts
WIP WIP: this documentation set is not yet complete.
### Layout Items
All layout items extend the `LTItem` class. There are two kinds of layout items:
- LTComponent: extends the base `LTItem` class, this class will have additional values such as boundary boxes, height and width
- LTContainer: extends the `LTComponent` class, this class is used to contain elements of the pdf that would have child elements. Iterating on this element will output its child elements.
#### Layout Containers
All of these classes extend the LTContainer class.
- LTPage: a layout item containing all elements of the given page.
- LTFigure: a layout item containing LTCurves and LTImages
- LTCharBlock: a layout item containing LTChars
#### Layout Components
All of these classes extend the LTComponent class.
- LTChar: an individual character
- LTCurves: represents a collection of svg-paths (available under `self.paths`)
- LTImage: a component containing information regarding an image
+27 -1
View File
@@ -1,11 +1,36 @@
# PDFMajor # PDFMajor
<table>
<tr>
<td>Latest Release</td>
<td>
<a href="https://pypi.org/project/pdfmajor/">
<img src="https://img.shields.io/pypi/v/pdfmajor.svg" alt="latest release" />
</a>
</td>
</tr>
</table>
PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability. PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability.
The ultimate goal is to have a functional-programming with typing fork of the original library.
## Documentation ## Documentation
See documentation [here](https://asosnovsky.github.io/pdfmajor/). See documentation [here](https://asosnovsky.github.io/pdfmajor/).
## Basic Example
```py
from pdfmajor.extractor import extract_items_from_pdf
for page in extract_items_from_pdf('path/to/your/file.pdf'):
print('page-start', page)
for item in page:
print(' ', item)
print('page-end', page)
```
## Why Another Fork? ## Why Another Fork?
`PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with. `PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with.
@@ -13,6 +38,7 @@ See documentation [here](https://asosnovsky.github.io/pdfmajor/).
## Features ## Features
* Functional extraction method based on generators
* Written entirely in Python. * Written entirely in Python.
* Parse, analyze, and convert PDF documents. * Parse, analyze, and convert PDF documents.
* PDF-1.7 specification support. (well, almost) * PDF-1.7 specification support. (well, almost)
+14 -3
View File
@@ -53,7 +53,7 @@
<a class="current" href=".">Home</a> <a class="current" href=".">Home</a>
<ul class="subnav"> <ul class="subnav">
<li class="toctree-l2"><a href="#pdf-major">PDF-Major</a></li> <li class="toctree-l2"><a href="#pdfmajor">PDFMajor</a></li>
<ul> <ul>
@@ -110,7 +110,18 @@
<div role="main"> <div role="main">
<div class="section"> <div class="section">
<h1 id="pdf-major">PDF-Major</h1> <h1 id="pdfmajor">PDFMajor</h1>
<table>
<tr>
<td>Latest Release</td>
<td>
<a href="https://pypi.org/project/pdfmajor/">
<img src="https://img.shields.io/pypi/v/pdfmajor.svg" alt="latest release" />
</a>
</td>
</tr>
</table>
<p>PDF-Major is a fork of <code>PDFMiner.six</code>. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability.</p> <p>PDF-Major is a fork of <code>PDFMiner.six</code>. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability.</p>
<h2 id="documentation">Documentation</h2> <h2 id="documentation">Documentation</h2>
<p>See documentation <a href="https://asosnovsky.github.io/pdfmajor/">here</a>.</p> <p>See documentation <a href="https://asosnovsky.github.io/pdfmajor/">here</a>.</p>
@@ -216,5 +227,5 @@ SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.</p>
<!-- <!--
MkDocs version : 1.0.4 MkDocs version : 1.0.4
Build Date UTC : 2018-10-12 22:22:34 Build Date UTC : 2018-11-04 21:27:38
--> -->
+1 -1
View File
@@ -1 +1 @@
{"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"PDF-Major PDF-Major is a fork of PDFMiner.six . It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability. Documentation See documentation here . Why Another Fork? PDFMiner was designed to run in python 2.7>, PDFMiner.six was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with. Features Written entirely in Python. Parse, analyze, and convert PDF documents. PDF-1.7 specification support. (well, almost) CJK languages and vertical writing scripts support. Various font types (Type1, TrueType, Type3, and CID) support. Basic encryption (RC4) support. Font-Color extraction Shape fill and stroke color extraction Note : We took out the layout-analysis process in this version (there is no more LTTextHorizontal or LTTextVertical). While the mathematics behind the grouping process was sound, the coupling of the layout-analysis process with the parsing and interpretation process produced unfriendly-code. This feature could be brought back by extending the pdfmajor.layouts.PDFLayoutAnalyzer class, but at the current time is not supported. How to Install Source Install Python 3.6.4 or newer. clone this repo git clone https://github.com/asosnovsky/pdfmajor * install repo python setup.py install Pypi Install Python 3.6.4 or newer. install repo pip install pdfmajor Terms and Conditions (This is so-called MIT/X License) Copyright (c) 2018-2019 Ariel Sosnovsky Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.","title":"Home"},{"location":"#pdf-major","text":"PDF-Major is a fork of PDFMiner.six . It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability.","title":"PDF-Major"},{"location":"#documentation","text":"See documentation here .","title":"Documentation"},{"location":"#why-another-fork","text":"PDFMiner was designed to run in python 2.7>, PDFMiner.six was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with.","title":"Why Another Fork?"},{"location":"#features","text":"Written entirely in Python. Parse, analyze, and convert PDF documents. PDF-1.7 specification support. (well, almost) CJK languages and vertical writing scripts support. Various font types (Type1, TrueType, Type3, and CID) support. Basic encryption (RC4) support. Font-Color extraction Shape fill and stroke color extraction Note : We took out the layout-analysis process in this version (there is no more LTTextHorizontal or LTTextVertical). While the mathematics behind the grouping process was sound, the coupling of the layout-analysis process with the parsing and interpretation process produced unfriendly-code. This feature could be brought back by extending the pdfmajor.layouts.PDFLayoutAnalyzer class, but at the current time is not supported.","title":"Features"},{"location":"#how-to-install","text":"","title":"How to Install"},{"location":"#source","text":"Install Python 3.6.4 or newer. clone this repo git clone https://github.com/asosnovsky/pdfmajor * install repo python setup.py install","title":"Source"},{"location":"#pypi","text":"Install Python 3.6.4 or newer. install repo pip install pdfmajor","title":"Pypi"},{"location":"#terms-and-conditions","text":"(This is so-called MIT/X License) Copyright (c) 2018-2019 Ariel Sosnovsky Permission is hereby granted, free of charge, to any person obtaining aLine truncated {"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"PDFMajor Latest Release PDF-Major is a fork of PDFMiner.six . It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability. Documentation See documentation here . Why Another Fork? PDFMiner was designed to run in python 2.7>, PDFMiner.six was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with. Features Written entirely in Python. Parse, analyze, and convert PDF documents. PDF-1.7 specification support. (well, almost) CJK languages and vertical writing scripts support. Various font types (Type1, TrueType, Type3, and CID) support. Basic encryption (RC4) support. Font-Color extraction Shape fill and stroke color extraction Note : We took out the layout-analysis process in this version (there is no more LTTextHorizontal or LTTextVertical). While the mathematics behind the grouping process was sound, the coupling of the layout-analysis process with the parsing and interpretation process produced unfriendly-code. This feature could be brought back by extending the pdfmajor.layouts.PDFLayoutAnalyzer class, but at the current time is not supported. How to Install Source Install Python 3.6.4 or newer. clone this repo git clone https://github.com/asosnovsky/pdfmajor * install repo python setup.py install Pypi Install Python 3.6.4 or newer. install repo pip install pdfmajor Terms and Conditions (This is so-called MIT/X License) Copyright (c) 2018-2019 Ariel Sosnovsky Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.","title":"Home"},{"location":"#pdfmajor","text":"Latest Release PDF-Major is a fork of PDFMiner.six . It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability.","title":"PDFMajor"},{"location":"#documentation","text":"See documentation here .","title":"Documentation"},{"location":"#why-another-fork","text":"PDFMiner was designed to run in python 2.7>, PDFMiner.six was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with.","title":"Why Another Fork?"},{"location":"#features","text":"Written entirely in Python. Parse, analyze, and convert PDF documents. PDF-1.7 specification support. (well, almost) CJK languages and vertical writing scripts support. Various font types (Type1, TrueType, Type3, and CID) support. Basic encryption (RC4) support. Font-Color extraction Shape fill and stroke color extraction Note : We took out the layout-analysis process in this version (there is no more LTTextHorizontal or LTTextVertical). While the mathematics behind the grouping process was sound, the coupling of the layout-analysis process with the parsing and interpretation process produced unfriendly-code. This feature could be brought back by extending the pdfmajor.layouts.PDFLayoutAnalyzer class, but at the current time is not supported.","title":"Features"},{"location":"#how-to-install","text":"","title":"How to Install"},{"location":"#source","text":"Install Python 3.6.4 or newer. clone this repo git clone https://github.com/asosnovsky/pdfmajor * install repo python setup.py install","title":"Source"},{"location":"#pypi","text":"Install Python 3.6.4 or newer. install repo pip install pdfmajor","title":"Pypi"},{"location":"#terms-and-conditions","text":"(This is so-called MIT/X License) Copyright (c) 2018-2019 Ariel Sosnovsky Permission is hereby granted, free of chargeLine truncated
+2 -2
View File
@@ -2,12 +2,12 @@
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url> <url>
<loc>None</loc> <loc>None</loc>
<lastmod>2018-10-12</lastmod> <lastmod>2018-11-04</lastmod>
<changefreq>daily</changefreq> <changefreq>daily</changefreq>
</url> </url>
<url> <url>
<loc>None</loc> <loc>None</loc>
<lastmod>2018-10-12</lastmod> <lastmod>2018-11-04</lastmod>
<changefreq>daily</changefreq> <changefreq>daily</changefreq>
</url> </url>
</urlset> </urlset>
Binary file not shown.
+28 -2
View File
@@ -3,14 +3,39 @@
# DO NOT EDIT THIS PAGE # DO NOT EDIT THIS PAGE
# #
""" """
# PDFMajor # PDFMajor
<table>
<tr>
<td>Latest Release</td>
<td>
<a href="https://pypi.org/project/pdfmajor/">
<img src="https://img.shields.io/pypi/v/pdfmajor.svg" alt="latest release" />
</a>
</td>
</tr>
</table>
PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability. PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability.
The ultimate goal is to have a functional-programming with typing fork of the original library.
## Documentation ## Documentation
See documentation [here](https://asosnovsky.github.io/pdfmajor/). See documentation [here](https://asosnovsky.github.io/pdfmajor/).
## Basic Example
```py
from pdfmajor.extractor import extract_items_from_pdf
for page in extract_items_from_pdf('path/to/your/file.pdf'):
print('page-start', page)
for item in page:
print(' ', item)
print('page-end', page)
```
## Why Another Fork? ## Why Another Fork?
`PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with. `PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with.
@@ -18,6 +43,7 @@ See documentation [here](https://asosnovsky.github.io/pdfmajor/).
## Features ## Features
* Functional extraction method based on generators
* Written entirely in Python. * Written entirely in Python.
* Parse, analyze, and convert PDF documents. * Parse, analyze, and convert PDF documents.
* PDF-1.7 specification support. (well, almost) * PDF-1.7 specification support. (well, almost)
@@ -75,7 +101,7 @@ OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
""" """
__version__ = (1, 1, 1) __version__ = (1, 2, 0)
if __name__ == '__main__': if __name__ == '__main__':
print( print(
+71 -55
View File
@@ -25,7 +25,10 @@ def get_color(col: PDFGraphicStateColor):
elif color_type == 'cmyk': elif color_type == 'cmyk':
return f"cmyk({','.join(map(str, color_val))})" return f"cmyk({','.join(map(str, color_val))})"
elif color_type == 'gray': elif color_type == 'gray':
return f'rgba(0,0,0,{1-color_val})' return f'rgba(0,0,0,{1-color_val[0]})'
elif color_type == "custom":
# Todo: convert these to more friendly color types
return f"custom({col.custom['type']})"
return "none" return "none"
class HTMLConverter(PDFConverter): class HTMLConverter(PDFConverter):
@@ -42,8 +45,8 @@ class HTMLConverter(PDFConverter):
self.__current_width = None self.__current_width = None
self.write_header() self.write_header()
def write(self, text: str): def write(self, text: str, lineend: str = '\n', deep_space: str = ' '):
text = " "*self.__levels_deep + text + '\n' text = deep_space*self.__levels_deep + text + lineend
self.write_raw(text) self.write_raw(text)
def write_raw(self, text): def write_raw(self, text):
@@ -78,7 +81,13 @@ class HTMLConverter(PDFConverter):
self.__levels_deep -= 1 self.__levels_deep -= 1
@contextmanager @contextmanager
def place_elm_with_child(self, tag_name: str, attr_dict: Dict[str, str] = {}, css: Dict[str, str] = None): def place_elm_with_child(self, tag_name: str, attr_dict: Dict[str, str] = {}, css: Dict[str, str] = None, no_additional_char = False):
lineend = '\n'
deep_space = ' '
if no_additional_char:
lineend = ''
deep_space = ''
if css is not None: if css is not None:
attr_dict['style'] = ";".join([ attr_dict['style'] = ";".join([
f'{name}: {value}' f'{name}: {value}'
@@ -88,11 +97,11 @@ class HTMLConverter(PDFConverter):
f'{name}="{value}"' f'{name}="{value}"'
for name, value in attr_dict.items() for name, value in attr_dict.items()
]) ])
self.write(f"<{tag_name} {attrs}>") self.write(f"<{tag_name} {attrs}>", lineend=lineend)
self.__levels_deep += 1 self.__levels_deep += 1
yield yield
self.__levels_deep -= 1 self.__levels_deep -= 1
self.write(f"</{tag_name}>") self.write(f"</{tag_name}>", deep_space=deep_space)
def place_elm_close(self, tag_name: str, attr_dict: Dict[str, str] = {}, css: Dict[str, str] = None, as_childless = True): def place_elm_close(self, tag_name: str, attr_dict: Dict[str, str] = {}, css: Dict[str, str] = None, as_childless = True):
if css is not None: if css is not None:
@@ -125,6 +134,8 @@ class HTMLConverter(PDFConverter):
'position': 'absolute', 'position': 'absolute',
'left': f'{item.x0}px', 'left': f'{item.x0}px',
'bottom': f'{item.y0}px', 'bottom': f'{item.y0}px',
'width': f'{item.width}px',
'height': f'{item.height}px',
} }
path = "" path = ""
svg_attr = { svg_attr = {
@@ -132,42 +143,36 @@ class HTMLConverter(PDFConverter):
'z-index': 2 'z-index': 2
} }
if item.height > 0: p_vals = { "x": [], "y": [] }
dy = 1.0/item.height
dx = (item.width/item.height)/(item.x1-item.x0)
svg_attr.update({
"viewBox": "0 0 %s %s" % ( item.width/item.height, 1 )
})
else:
dx = 1.0/(item.x1-item.x0)
dy = 1.0
svg_attr.update({
"viewBox": "0 0 1 1"
})
if item.height > 1: for p in item.paths:
svg_attr.update({ for pt in p.points:
'height': f'{item.height}px', p_vals['x'].append(pt.x)
}) p_vals['y'].append(pt.y)
else:
svg_attr.update({ mx_x = max([*p_vals['x'], 1])
'height': '1px', mn_x = min([*p_vals['x'], 0])
}) mx_y = max([*p_vals['y'], 1])
mn_y = min([*p_vals['y'], 0])
if item.width > 1:
svg_attr.update({ if mx_x == mn_x:
'width': f'{item.width}px', mn_x -= 1
}) if mx_y == mn_y:
else: mn_y -= 1
svg_attr.update({
'width': '1px', dx = 1/( mx_x - mn_x )
}) dy = (item.height/item.width)/( mx_y - mn_y )
ROUND_VAL = 5
svg_attr.update({
# "viewBox": f"{min(p_vals['x'])} {min(p_vals['y'])} {max(p_vals['x'])} {max(p_vals['y'])}",
"viewBox": f"{round(mn_x*dx, ROUND_VAL)} {round(mn_y*dy, ROUND_VAL)} {round(mx_x*dx , ROUND_VAL)} {round(mx_y*dy, ROUND_VAL)}",
})
for p in item.paths: for p in item.paths:
if p.method == CurvePath.METHOD.MOVE_TO: if p.method == CurvePath.METHOD.MOVE_TO:
path += f'M{" ".join( f"{(item.width - (p.x-item.x0) )*dx} {(item.height- (p.y-item.y0) )*dy}" for p in p.points )} ' path += f'M{" ".join( f"{round(dx*pt.x, ROUND_VAL)} {round(dy*pt.y, ROUND_VAL)}" for pt in p.points )} '
if p.method == CurvePath.METHOD.LINE_TO: if p.method == CurvePath.METHOD.LINE_TO:
path += f'L{" ".join( f"{(item.width - (p.x-item.x0) )*dx} {( item.height - (p.y-item.y0) )*dy}" for p in p.points )} ' path += f'L{" ".join( f"{round(dx*pt.x, ROUND_VAL)} {round(dy*pt.y, ROUND_VAL)}" for pt in p.points )} '
if p.method == CurvePath.METHOD.CLOSE_PATH: if p.method == CurvePath.METHOD.CLOSE_PATH:
path += 'Z' path += 'Z'
with self.place_elm_with_child('svg', svg_attr, css): with self.place_elm_with_child('svg', svg_attr, css):
@@ -201,28 +206,39 @@ class HTMLConverter(PDFConverter):
self.place_image(item) self.place_image(item)
elif isinstance(item, LTChar): elif isinstance(item, LTChar):
with self.place_elm_with_child('span', { 'class': 'char' }, { with self.place_elm_with_child('span', { 'class': 'char' }, {
'font-size': f'{item.size}px', 'width': item.width,
'font-family': item.fontname, 'min-width': item.width,
'font-weight': 'bold' if 'bold' in item.fontname.lower() else '', 'max-width': item.width,
'color': get_color(item.graphicstate.ncolor), 'height': item.height,
'position': 'absolute', 'min-height': item.height,
'left': f'{item.x0}', 'max-height': item.height,
'bottom': f'{item.y0}',
'transform': f'skew({item.font.italic_angle}deg, 0deg)'
}): }):
self.write(item.get_text()) self.write(item.get_text())
elif isinstance(item, LTCharBlock): elif isinstance(item, LTCharBlock):
with self.place_elm_with_child('span', { 'class': 'char-block' }, { with self.place_elm_with_child('span', { 'class': 'char-block' }, {
'font-size': f'{item.size}px', 'position': 'absolute',
'font-family': item.fontname, 'left': f'{item.x0}',
'font-weight': 'bold' if 'bold' in item.fontname.lower() else '', 'bottom': f'{item.y0}',
'color': get_color(item.graphicstate.ncolor), 'text-align': 'justify',
'position': 'absolute', 'font-size': f'{item.size}px',
'left': f'{item.x0}', 'font-family': item.fontname,
'bottom': f'{item.y0}', 'font-weight': item.font.font_weight,
'transform': f'skew({item.font.italic_angle}deg, 0deg)' 'color': get_color(item.graphicstate.ncolor),
}): 'width': item.width,
self.write(item.get_text()) 'min-width': item.width,
'max-width': item.width,
'height': item.height,
'min-height': item.height,
'max-height': item.height,
'transform': f'skew({item.font.italic_angle}deg, 0deg)',
'text-align': 'center',
'display': 'flex',
'flex-flow': 'row nowrap',
'justify-content': 'space-between',
# 'padding-left': item.font.leading
}):
for char in item:
render(char)
render(ltpage) render(ltpage)
def close(self): def close(self):
+12 -8
View File
@@ -1,17 +1,19 @@
import logging
import re import re
import os import os
from typing import List from typing import List
from io import BytesIO, StringIO, TextIOWrapper from io import BytesIO, StringIO, TextIOWrapper
from tqdm import tqdm
from ..layouts import PDFLayoutAnalyzer from ..layouts import PDFLayoutAnalyzer
from ..interpreter.PDFResourceManager import PDFResourceManager from ..interpreter.PDFResourceManager import PDFResourceManager
from ..interpreter.PDFPage import PDFPage from ..interpreter.PDFPage import PDFPage
from ..interpreter.PDFPageInterpreter import PDFPageInterpreter from ..interpreter.PDFPageInterpreter import PDFPageInterpreter
from ..imagewriter import ImageWriter from ..imagewriter import ImageWriter
from ..utils import set_log_level, get_logger, logging
log = logging.getLogger(__name__) log = get_logger(__name__)
## PDFConverter ## PDFConverter
## ##
@@ -29,11 +31,10 @@ class PDFConverter(PDFLayoutAnalyzer):
check_extractable: bool = True, check_extractable: bool = True,
pagenos: List[int] = None, pagenos: List[int] = None,
dont_export_images: bool = False, dont_export_images: bool = False,
debug: bool = False, debug_level: int = logging.WARNING,
) -> TextIOWrapper: ) -> TextIOWrapper:
if debug: set_log_level(debug_level)
log.setLevel(logging.DEBUG)
if dont_export_images: if dont_export_images:
image_folder_path = None image_folder_path = None
@@ -53,9 +54,12 @@ class PDFConverter(PDFLayoutAnalyzer):
interpreter = PDFPageInterpreter(rsrcmgr, device) interpreter = PDFPageInterpreter(rsrcmgr, device)
log.debug("Reading pages....") log.debug("Reading pages....")
pages = PDFPage.get_pages(input_file, pagenos=pagenos, maxpages=maxpages, password=password, caching=caching, check_extractable=check_extractable) pages = PDFPage.get_pages(input_file, pagenos=pagenos, maxpages=maxpages, password=password, caching=caching, check_extractable=check_extractable)
for idx, page in enumerate(pages):
log.debug(f"Reading page #{idx}") page_waiter = tqdm(enumerate(pages), disable=debug_level > logging.INFO)
interpreter.process_page(page) for _, page in page_waiter:
page_waiter.set_description("Processing pdf...")
interpreter.process_page(page)
page_waiter.close()
device.close() device.close()
return output_file return output_file
+36 -17
View File
@@ -1,7 +1,10 @@
import logging import logging
import re
import json
from typing import Dict from typing import Dict
from xml.sax import saxutils
from contextlib import contextmanager from contextlib import contextmanager
from ..layouts import LTPage from ..layouts import LTPage
@@ -28,8 +31,17 @@ def get_color(col: PDFGraphicStateColor):
return f"cmyk({','.join(map(str, color_val))})" return f"cmyk({','.join(map(str, color_val))})"
elif color_type == 'gray': elif color_type == 'gray':
return f'gray({color_val})' return f'gray({color_val})'
elif color_type == "custom":
# Todo: convert these to more friendly color types
return f"{col.custom['type']}({col.custom['defintion']})"
return "" return ""
FIRST_CAP_RE = re.compile('(.)([A-Z][a-z]+)')
ALL_CAP_RE = re.compile('([a-z0-9])([A-Z])')
def convert_capcase_to_snakecase(text: str) -> str:
s1 = FIRST_CAP_RE.sub(r'\1-\2', text)
return ALL_CAP_RE.sub(r'\1-\2', s1).lower()
class XMLConverter(PDFConverter): class XMLConverter(PDFConverter):
def __init__(self, def __init__(self,
rsrcmgr, rsrcmgr,
@@ -71,7 +83,7 @@ class XMLConverter(PDFConverter):
lineend = '' lineend = ''
deep_space = '' deep_space = ''
attrs = " ".join([ attrs = " ".join([
f'{name}="{value}"' f'{name}="{saxutils.escape(str(value))}"'
for name, value in attr_dict.items() for name, value in attr_dict.items()
]) ])
self.write(f"<{tag_name} {attrs}>", lineend=lineend) self.write(f"<{tag_name} {attrs}>", lineend=lineend)
@@ -82,7 +94,7 @@ class XMLConverter(PDFConverter):
def place_elm_close(self, tag_name: str, attr_dict: Dict[str, str] = {}, as_childless = True): def place_elm_close(self, tag_name: str, attr_dict: Dict[str, str] = {}, as_childless = True):
attrs = " ".join([ attrs = " ".join([
f'{name}="{value}"' f'{name}="{saxutils.escape(str(value))}"'
for name, value in attr_dict.items() for name, value in attr_dict.items()
]) ])
if as_childless: if as_childless:
@@ -113,22 +125,29 @@ class XMLConverter(PDFConverter):
}) })
def place_text(self, char: LTChar): def place_text(self, char: LTChar):
tag_name = 'char' with self.place_elm_with_child('char', {
if isinstance(char, LTCharBlock): 'x0': char.x0,
tag_name = 'char-block' 'x1': char.x1,
'y0': char.y0,
'y1': char.y1,
}, no_additional_char=True):
self.write(saxutils.escape(char.get_text()), lineend='', deep_space="")
def render_char_block(self, char_block: LTCharBlock):
attr = { attr = {
'size': char.size, 'size': char_block.size,
"color": get_color(char.graphicstate.ncolor), "color": get_color(char_block.graphicstate.ncolor),
"x0": char.x0, "x0": char_block.x0,
"x1": char.x1, "x1": char_block.x1,
"y0": char.y0, "y0": char_block.y0,
"y1": char.y1, "y1": char_block.y1,
} }
for key, value in char.font.descriptor.items(): for key, value in char_block.font.descriptor.items():
if key != "Type" and "FontFile" not in key: if key != "Type" and "FontFile" not in key:
attr["font-" + key] = value attr[convert_capcase_to_snakecase(key)] = value
with self.place_elm_with_child(tag_name, attr, no_additional_char=True): with self.place_elm_with_child("char-block", attr):
self.write(char.get_text(), lineend='', deep_space="") for char in char_block:
self.place_text(char)
def render_curve(self, item: LTCurve): def render_curve(self, item: LTCurve):
attr = { attr = {
@@ -164,8 +183,8 @@ class XMLConverter(PDFConverter):
render(child) render(child)
elif isinstance(item, LTImage): elif isinstance(item, LTImage):
self.place_image(item) self.place_image(item)
elif isinstance(item, LTChar) or isinstance(item, LTCharBlock): elif isinstance(item, LTCharBlock):
self.place_text(item) self.render_char_block(item)
render(ltpage) render(ltpage)
def close(self): def close(self):
+47 -41
View File
@@ -8,6 +8,7 @@ from .HTMLConverter import HTMLConverter
from .JSONConverter import JSONConverter from .JSONConverter import JSONConverter
from .XMLConverter import XMLConverter from .XMLConverter import XMLConverter
from .TextConverter import TextConverter from .TextConverter import TextConverter
from ..utils import logging
def convert_file( def convert_file(
input_file: io.TextIOWrapper, input_file: io.TextIOWrapper,
@@ -19,7 +20,8 @@ def convert_file(
caching: bool = True, caching: bool = True,
check_extractable: bool = True, check_extractable: bool = True,
pagenos: List[int] = None, pagenos: List[int] = None,
debug: bool = False, dont_export_images: bool = False,
debug_level: int = logging.WARNING,
out_type: str = 'html', out_type: str = 'html',
): ):
if output_file is None: if output_file is None:
@@ -36,54 +38,58 @@ def convert_file(
raise Exception("Please make sure that the file name and output type match!") raise Exception("Please make sure that the file name and output type match!")
if out_type == 'html': if out_type == 'html':
return HTMLConverter.parse_file( return HTMLConverter.parse_file(
input_file, input_file=input_file,
output_file, output_file=output_file,
image_folder_path, image_folder_path=image_folder_path,
codec, codec=codec,
maxpages, maxpages=maxpages,
password, password=password,
caching, caching=caching,
check_extractable, check_extractable=check_extractable,
pagenos, pagenos=pagenos,
debug, dont_export_images=dont_export_images,
debug_level=debug_level,
) )
elif out_type == 'xml': elif out_type == 'xml':
return XMLConverter.parse_file( return XMLConverter.parse_file(
input_file, input_file=input_file,
output_file, output_file=output_file,
image_folder_path, image_folder_path=image_folder_path,
codec, codec=codec,
maxpages, maxpages=maxpages,
password, password=password,
caching, caching=caching,
check_extractable, check_extractable=check_extractable,
pagenos, pagenos=pagenos,
debug, dont_export_images=dont_export_images,
debug_level=debug_level,
) )
elif out_type == 'json': elif out_type == 'json':
return JSONConverter.parse_file( return JSONConverter.parse_file(
input_file, input_file=input_file,
output_file, output_file=output_file,
image_folder_path, image_folder_path=image_folder_path,
codec, codec=codec,
maxpages, maxpages=maxpages,
password, password=password,
caching, caching=caching,
check_extractable, check_extractable=check_extractable,
pagenos, pagenos=pagenos,
debug, dont_export_images=dont_export_images,
debug_level=debug_level,
) )
elif out_type == 'text': elif out_type == 'text':
return TextConverter.parse_file( return TextConverter.parse_file(
input_file, input_file=input_file,
output_file, output_file=output_file,
image_folder_path, image_folder_path=image_folder_path,
codec, codec=codec,
maxpages, maxpages=maxpages,
password, password=password,
caching, caching=caching,
check_extractable, check_extractable=check_extractable,
pagenos, pagenos=pagenos,
debug, dont_export_images=dont_export_images,
debug_level=debug_level,
) )
else: raise Exception("Please specify out_type as 'html' or 'xml' or 'json' or 'text'") else: raise Exception("Please specify out_type as 'html' or 'xml' or 'json' or 'text'")
+12
View File
@@ -0,0 +1,12 @@
from ..utils import get_logger
log = get_logger(__name__)
from ..layouts import PDFLayoutAnalyzer, PDFPage, LTPage
class PDFExctractor(PDFLayoutAnalyzer):
def end_page(self, page: PDFPage):
assert not self._stack, str(len(self._stack))
assert isinstance(self.cur_item, LTPage), str(type(self.cur_item))
self.pageno += 1
return self.cur_item
+46
View File
@@ -0,0 +1,46 @@
import re
import os
from typing import List
from io import BytesIO, StringIO, TextIOWrapper
from tqdm import tqdm
from ..interpreter.PDFResourceManager import PDFResourceManager
from ..interpreter.PDFPage import PDFPage
from ..interpreter.PDFPageInterpreter import PDFPageInterpreter
from ..imagewriter import ImageWriter
from ..utils import set_log_level, get_logger, logging
from .PDFExtractor import PDFExctractor
log = get_logger(__name__)
def extract_items_from_pdf(
input_file_path: str,
maxpages: int = 0,
password: str = None,
caching: bool = True,
check_extractable: bool = True,
pagenos: List[int] = None,
debug_level: int = logging.WARNING,
):
set_log_level(debug_level)
log.debug("Creating resources....")
rsrcmgr = PDFResourceManager(caching=caching)
device = PDFExctractor(
rsrcmgr=rsrcmgr,
pageno=0
)
interpreter = PDFPageInterpreter(rsrcmgr, device)
log.debug("Reading pages....")
with open(input_file_path, 'rb') as input_file:
pages = PDFPage.get_pages(input_file, pagenos=pagenos, maxpages=maxpages, password=password, caching=caching, check_extractable=check_extractable)
page_waiter = tqdm(enumerate(pages), disable=debug_level > logging.INFO)
for _, page in page_waiter:
page_waiter.set_description("Processing pdf...")
yield interpreter.process_page(page)
page_waiter.close()
device.close()
+13
View File
@@ -531,6 +531,19 @@ class PDFFont(object):
@abstractmethod @abstractmethod
def to_unichr(self, cid): def to_unichr(self, cid):
raise NotImplementedError raise NotImplementedError
@property
def font_weight(self):
if 'FontWeight' in self.descriptor.keys():
return self.descriptor['FontWeight']
elif 'bold' in self.fontname.lower():
return 'bold'
else:
return None
@property
def is_bold(self):
return self.font_weight == 'bold' or self.font_weight > 549
# PDFSimpleFont # PDFSimpleFont
@@ -5,6 +5,7 @@ class PDFGraphicStateColor:
self.__rgb = None self.__rgb = None
self.__gray = None self.__gray = None
self.__cmyk = None self.__cmyk = None
self.__custom = None
def get_color(self): def get_color(self):
if self.__rgb is not None: if self.__rgb is not None:
@@ -13,6 +14,8 @@ class PDFGraphicStateColor:
return 'cmyk', self.cmyk return 'cmyk', self.cmyk
elif self.__gray is not None: elif self.__gray is not None:
return 'gray', self.gray return 'gray', self.gray
elif self.__custom is not None:
return 'custom', self.custom
else: return None, None else: return None, None
@property @property
@@ -31,7 +34,12 @@ class PDFGraphicStateColor:
@property @property
def gray(self): def gray(self):
if self.__gray is not None: return float(self.__gray) if self.__gray is not None: return [float(self.__gray)]
else: raise self.NotSet
@property
def custom(self):
if self.__custom is not None: return dict(self.__custom)
else: raise self.NotSet else: raise self.NotSet
@rgb.setter @rgb.setter
@@ -46,11 +54,16 @@ class PDFGraphicStateColor:
def gray(self, color): def gray(self, color):
self.__gray = color self.__gray = color
@custom.setter
def custom(self, color):
self.__custom = color
def copy(self): def copy(self):
cp = self.__class__() cp = self.__class__()
cp.gray = self.__gray cp.gray = self.__gray
cp.rgb = self.__rgb cp.rgb = self.__rgb
cp.cmyk = self.__cmyk cp.cmyk = self.__cmyk
cp.custom = self.__custom
return cp return cp
def __repr__(self): def __repr__(self):
@@ -58,4 +71,5 @@ class PDFGraphicStateColor:
if self.__gray is not None: rep += f' gray={self.__gray}' if self.__gray is not None: rep += f' gray={self.__gray}'
if self.__rgb is not None: rep += f' rgb=[{",".join(map(str, self.__rgb))}]' if self.__rgb is not None: rep += f' rgb=[{",".join(map(str, self.__rgb))}]'
if self.__cmyk is not None: rep += f' cmyk=[{",".join(map(str, self.__cmyk))}]' if self.__cmyk is not None: rep += f' cmyk=[{",".join(map(str, self.__cmyk))}]'
if self.__custom is not None: rep += f' custom={self.__custom}'
return rep + '/>' return rep + '/>'
+4 -2
View File
@@ -7,7 +7,9 @@ from .PDFStream import resolve1, int_value, list_value, dict_value
from .PDFParser import PDFParser from .PDFParser import PDFParser
from .PDFDocument import PDFDocument, PDFTextExtractionNotAllowed from .PDFDocument import PDFDocument, PDFTextExtractionNotAllowed
log = logging.getLogger(__name__) from ..utils import get_logger
log = get_logger(__name__)
# some predefined literals and keywords. # some predefined literals and keywords.
LITERAL_PAGE = LIT('Page') LITERAL_PAGE = LIT('Page')
@@ -89,7 +91,7 @@ class PDFPage(object):
tree_type = tree.get('type') tree_type = tree.get('type')
if tree_type is LITERAL_PAGES and 'Kids' in tree: if tree_type is LITERAL_PAGES and 'Kids' in tree:
log.info('Pages: Kids=%r', tree['Kids']) # log.info('Pages: Kids=%r', tree['Kids'])
for c in list_value(tree['Kids']): for c in list_value(tree['Kids']):
for x in search(c, tree): for x in search(c, tree):
yield x yield x
+77 -34
View File
@@ -1,6 +1,7 @@
import logging import logging
import operator import operator
from collections import OrderedDict
from typing import List from typing import List
@@ -26,7 +27,7 @@ from .PDFGraphicState import PDFGraphicState
from .PDFTextState import PDFTextState from .PDFTextState import PDFTextState
from .PDFContentParser import PDFContentParser from .PDFContentParser import PDFContentParser
from ..utils import settings, mult_matrix, MATRIX_IDENTITY from ..utils import mult_matrix, MATRIX_IDENTITY
from .constants import LITERAL_FORM, LITERAL_IMAGE from .constants import LITERAL_FORM, LITERAL_IMAGE
from .types import CurvePoint, CurvePath from .types import CurvePoint, CurvePath
@@ -54,7 +55,7 @@ class PDFPageInterpreter(object):
self.resources = resources self.resources = resources
self.fontmap = {} self.fontmap = {}
self.xobjmap = {} self.xobjmap = {}
self.csmap = PREDEFINED_COLORSPACE.copy() self.csmap:OrderedDict = PREDEFINED_COLORSPACE.copy()
if not resources: if not resources:
return return
@@ -303,8 +304,7 @@ class PDFPageInterpreter(object):
try: try:
self.scs = self.csmap[literal_name(name)] self.scs = self.csmap[literal_name(name)]
except KeyError: except KeyError:
if settings.STRICT: raise PDFInterpreterError('Undefined ColorSpace: %r' % name)
raise PDFInterpreterError('Undefined ColorSpace: %r' % name)
return return
# setcolorspace-non-strokine # setcolorspace-non-strokine
@@ -312,8 +312,7 @@ class PDFPageInterpreter(object):
try: try:
self.ncs = self.csmap[literal_name(name)] self.ncs = self.csmap[literal_name(name)]
except KeyError: except KeyError:
if settings.STRICT: raise PDFInterpreterError('Undefined ColorSpace: %r' % name)
raise PDFInterpreterError('Undefined ColorSpace: %r' % name)
return return
# setgray-stroking # setgray-stroking
@@ -352,25 +351,74 @@ class PDFPageInterpreter(object):
#self.do_cs(LITERAL_DEVICE_CMYK) #self.do_cs(LITERAL_DEVICE_CMYK)
return return
# setcolor def __process_ICCBased(self, n: int, components: list):
def do_SCN(self): if n == 1:
if self.scs: return {
n = self.scs.ncomponents "type": "gray",
else: "definition": components
if settings.STRICT: }
raise PDFInterpreterError('No colorspace specified!') elif n == 3:
n = 1 return {
self.pop(n) "type": "rgb",
return "definition": components
}
elif n == 4:
return {
"type": "cmyk",
"definition": components
}
raise PDFInterpreterError("Unsupported ICCBased color")
def do_scn(self): # setcolor
def __process_scn(self):
if self.ncs: if self.ncs:
n = self.ncs.ncomponents n = self.ncs.ncomponents
else: else:
if settings.STRICT: raise PDFInterpreterError('No colorspace specified!')
raise PDFInterpreterError('No colorspace specified!') components = self.pop(n)
n = 1 obj_id = components[0]
self.pop(n) if self.ncs.name == "ICCBased":
return self.__process_ICCBased(n, components)
if self.ncs.name not in self.resources.keys():
log.warning("Color space not in resources")
return {
"type": self.ncs.name,
"defintion": f"{n}|({','.join(map(str, components))})"
}
if obj_id.name not in self.resources[self.ncs.name].keys():
raise PDFInterpreterError("Invalid Color reference")
color_data = self.resources[self.ncs.name][obj_id.name].resolve()
if isinstance(color_data, PDFStream):
color_data = color_data.attrs
return {
"type": self.ncs.name,
"defintion": color_data
}
def do_SCN(self):
processed_col = self.__process_scn()
if processed_col['type'] == 'rgb':
self.graphicstate.scolor.rgb = processed_col['definition']
elif processed_col['type'] == 'gray':
self.graphicstate.scolor.gray = processed_col['definition'][0]
elif processed_col['type'] == 'cmyk':
self.graphicstate.scolor.cmyk = processed_col['definition']
else:
self.graphicstate.scolor.custom = processed_col
return
def do_scn(self):
processed_col = self.__process_scn()
if processed_col['type'] == 'rgb':
self.graphicstate.ncolor.rgb = processed_col['definition']
elif processed_col['type'] == 'gray':
self.graphicstate.ncolor.gray = processed_col['definition'][0]
elif processed_col['type'] == 'cmyk':
self.graphicstate.ncolor.cmyk = processed_col['definition']
else:
self.graphicstate.ncolor.custom = processed_col
return return
def do_SC(self): def do_SC(self):
@@ -432,6 +480,8 @@ class PDFPageInterpreter(object):
def do_Tw(self, space): def do_Tw(self, space):
self.textstate.wordspace = space self.textstate.wordspace = space
return return
# def do_Th(self, )
# textscale # textscale
def do_Tz(self, scale): def do_Tz(self, scale):
@@ -448,9 +498,8 @@ class PDFPageInterpreter(object):
try: try:
self.textstate.font = self.fontmap[literal_name(fontid)] self.textstate.font = self.fontmap[literal_name(fontid)]
except KeyError: except KeyError:
if settings.STRICT: raise PDFInterpreterError('Undefined Font id: %r' % fontid)
raise PDFInterpreterError('Undefined Font id: %r' % fontid) # self.textstate.font = self.rsrcmgr.get_font(None, {})
self.textstate.font = self.rsrcmgr.get_font(None, {})
self.textstate.fontsize = fontsize self.textstate.fontsize = fontsize
return return
@@ -495,9 +544,7 @@ class PDFPageInterpreter(object):
# show-pos # show-pos
def do_TJ(self, seq: bytearray): def do_TJ(self, seq: bytearray):
if self.textstate.font is None: if self.textstate.font is None:
if settings.STRICT: raise PDFInterpreterError('No font specified!')
raise PDFInterpreterError('No font specified!')
return
self.device.render_string(self.textstate.copy(), seq, self.ncs, self.graphicstate.copy()) self.device.render_string(self.textstate.copy(), seq, self.ncs, self.graphicstate.copy())
return return
@@ -540,9 +587,7 @@ class PDFPageInterpreter(object):
try: try:
xobj = PDFStream.validated_stream(self.xobjmap[xobjid]) xobj = PDFStream.validated_stream(self.xobjmap[xobjid])
except KeyError: except KeyError:
if settings.STRICT: raise PDFInterpreterError('Undefined xobject id: %r' % xobjid)
raise PDFInterpreterError('Undefined xobject id: %r' % xobjid)
return
log.info('Processing xobj: %r', xobj) log.info('Processing xobj: %r', xobj)
subtype = xobj.get('Subtype') subtype = xobj.get('Subtype')
if subtype is LITERAL_FORM and 'BBox' in xobj: if subtype is LITERAL_FORM and 'BBox' in xobj:
@@ -579,8 +624,7 @@ class PDFPageInterpreter(object):
ctm = (1, 0, 0, 1, -x0, -y0) ctm = (1, 0, 0, 1, -x0, -y0)
self.device.begin_page(page, ctm) self.device.begin_page(page, ctm)
self.render_contents(page.resources, page.contents, ctm=ctm) self.render_contents(page.resources, page.contents, ctm=ctm)
self.device.end_page(page) return self.device.end_page(page)
return
# render_contents(resources, streams, ctm) # render_contents(resources, streams, ctm)
# Render the content streams. # Render the content streams.
@@ -618,8 +662,7 @@ class PDFPageInterpreter(object):
log.debug('exec: %s', name) log.debug('exec: %s', name)
func() func()
else: else:
if settings.STRICT: raise PDFInterpreterError('Unknown operator: %r' % name)
raise PDFInterpreterError('Unknown operator: %r' % name)
else: else:
self.push(obj) self.push(obj)
return return
+1 -1
View File
@@ -12,7 +12,7 @@ from .exceptions import * # pylint: disable=W0614
## ##
class PDFObjRef(PDFObject): class PDFObjRef(PDFObject):
def __init__(self, doc, objid, _): def __init__(self, doc, objid, a):
if objid == 0: if objid == 0:
if settings.STRICT: if settings.STRICT:
raise PDFValueError('PDF object id cannot be 0.') raise PDFValueError('PDF object id cannot be 0.')
@@ -100,7 +100,6 @@ class PSBaseParser(object):
linebuf += self.buf[self.charpos:] linebuf += self.buf[self.charpos:]
self.charpos = len(self.buf) self.charpos = len(self.buf)
log.debug('nextline: %r, %r', linepos, linebuf) log.debug('nextline: %r, %r', linepos, linebuf)
return (linepos, linebuf) return (linepos, linebuf)
def revreadlines(self): def revreadlines(self):
+6 -6
View File
@@ -1,6 +1,6 @@
from typing import List from typing import List
from ._base import LTComponent from ._base import LTContainer
from .LTChar import LTChar from .LTChar import LTChar
from ..utils import apply_matrix_norm, apply_matrix_pt, matrix2str, Bbox, Point from ..utils import apply_matrix_norm, apply_matrix_pt, matrix2str, Bbox, Point
@@ -11,7 +11,7 @@ from ..interpreter.PDFTextState import PDFTextState
## LTCharBlock ## LTCharBlock
## ##
class LTCharBlock(LTComponent): class LTCharBlock(LTContainer):
def __init__(self, def __init__(self,
chars: List[Point], chars: List[Point],
@@ -20,7 +20,7 @@ class LTCharBlock(LTComponent):
graphicstate: PDFGraphicState graphicstate: PDFGraphicState
): ):
((x0, y0), (x1, y1)) = chars[0][1] ((x0, y0), (x1, y1)) = chars[0][1]
self.chars = [ ltchars = [
LTChar( LTChar(
bbox=Bbox(x0, y0, x1, y1), bbox=Bbox(x0, y0, x1, y1),
char= chars[0][0], char= chars[0][0],
@@ -36,7 +36,7 @@ class LTCharBlock(LTComponent):
x1 = max([ x0, x1, nx0, nx1 ]) x1 = max([ x0, x1, nx0, nx1 ])
y1 = max([ y0, y1, ny0, ny1 ]) y1 = max([ y0, y1, ny0, ny1 ])
y0 = min([ y0, y1, ny0, ny1 ]) y0 = min([ y0, y1, ny0, ny1 ])
self.chars.append(LTChar( ltchars.append(LTChar(
bbox=Bbox(nx0, ny0, nx1, ny1), bbox=Bbox(nx0, ny0, nx1, ny1),
char= char[0], char= char[0],
textstate=textstate, textstate=textstate,
@@ -44,7 +44,7 @@ class LTCharBlock(LTComponent):
graphicstate=graphicstate graphicstate=graphicstate
)) ))
LTComponent.__init__(self, Bbox(x0, y0, x1, y1)) LTContainer.__init__(self, Bbox(x0, y0, x1, y1), ltchars)
self.font = textstate.font self.font = textstate.font
self.textstate = textstate self.textstate = textstate
@@ -61,7 +61,7 @@ class LTCharBlock(LTComponent):
return f"""<{self.__class__.__name__} font="{self.fontname}" text="{self.get_text()}"/>""" return f"""<{self.__class__.__name__} font="{self.fontname}" text="{self.get_text()}"/>"""
def get_text(self): def get_text(self):
return "".join([ c.get_text() for c in self.chars ]) return "".join([ c.get_text() for c in self ])
@property @property
def fontname(self) -> str: def fontname(self) -> str:
+2 -2
View File
@@ -90,9 +90,9 @@ class LTComponent(LTItem):
## ##
class LTContainer(LTComponent): class LTContainer(LTComponent):
def __init__(self, bbox: Bbox): def __init__(self, bbox: Bbox, objs: List[LTItem] = []):
LTComponent.__init__(self, bbox) LTComponent.__init__(self, bbox)
self._objs: List[LTItem] = [] self._objs: List[LTItem] = objs
return return
def __iter__(self): def __iter__(self):
+2 -1
View File
@@ -9,4 +9,5 @@ from .generic import *
from .lzw import * from .lzw import *
from .matrix import * from .matrix import *
from .text import * from .text import *
from .types import * from .types import *
from .log import *
+17
View File
@@ -0,0 +1,17 @@
import logging
logging.basicConfig(
level=logging.WARNING
)
LOGGERS = {}
def set_log_level(debug_level: int):
logging.basicConfig(level=debug_level)
for _, logger in LOGGERS.items():
logger.setLevel(debug_level)
def get_logger(logger_name: str) -> logging.Logger:
if logger_name not in LOGGERS.keys():
LOGGERS[logger_name] = logging.getLogger(logger_name)
return LOGGERS[logger_name]
Regular → Executable
+20 -12
View File
@@ -1,37 +1,45 @@
#!/usr/bin/env python
import re import re
import os import os
import logging
from typing import List from typing import List
from pdfmajor.converters import convert_file from pdfmajor.converters import convert_file
from xml.etree import ElementTree
from tqdm import tqdm
def rel_path(*arg: List[str]) -> str: def rel_path(*arg: List[str]) -> str:
return os.path.join(os.path.dirname(__file__), *arg) return os.path.join(os.path.dirname(__file__), *arg)
EXT = 'xml' EXT = 'html'
INPUT_FOLDER = rel_path("./pdf") INPUT_FOLDER = rel_path("./pdf")
OUTPUT_FOLDER = rel_path(f"./output/{EXT}") OUTPUT_FOLDER = rel_path(f"./output/{EXT}")
if not os.path.exists(OUTPUT_FOLDER): if not os.path.exists(OUTPUT_FOLDER):
os.makedirs(OUTPUT_FOLDER) os.makedirs(OUTPUT_FOLDER)
# files = os.listdir(INPUT_FOLDER) files = os.listdir(INPUT_FOLDER)
# files = ['fonts.pdf']
# files = ['bar-charts.pdf']
# files = ['font-color.pdf']
files = ['tables.pdf']
TOTAL = len(files) TOTAL = len(files)
for idx, file_name in enumerate(files): waiter = tqdm(enumerate(files), total=TOTAL)
for idx, file_name in waiter:
file_name_no_ext = re.sub(r'\.pdf$',"", file_name) file_name_no_ext = re.sub(r'\.pdf$',"", file_name)
input_file_path = os.path.join(INPUT_FOLDER, file_name)
output_file_path = os.path.join(OUTPUT_FOLDER, file_name_no_ext + "." + EXT)
print(f"processing {file_name} {idx}/{TOTAL}")
if len(file_name) > len(file_name_no_ext): if len(file_name) > len(file_name_no_ext):
with open(os.path.join(INPUT_FOLDER, file_name), 'rb') as input_file: waiter.set_description(f"Processing {file_name}")
with open(os.path.join(OUTPUT_FOLDER, file_name_no_ext + "." + EXT), 'wb') as output_file: with open(input_file_path, 'rb') as input_file:
with open(output_file_path, 'wb') as output_file:
convert_file( convert_file(
input_file=input_file, input_file=input_file,
output_file=output_file, output_file=output_file,
out_type=EXT, out_type=EXT,
debug=True, debug_level=logging.NOTSET,
) )
print(f"processing {idx+1}/{TOTAL}") if EXT == 'xml':
waiter.set_description(f" > testing format {file_name}")
ElementTree.parse(output_file_path)
waiter.refresh()
waiter.set_description("Done")
waiter.close()
+23
View File
@@ -0,0 +1,23 @@
#!/usr/bin/env python
import re
import os
import logging
from typing import List
from pdfmajor.converters import convert_file
from xml.etree import ElementTree
from tqdm import tqdm
from pdfmajor.extractor import extract_items_from_pdf
from pdfmajor.layouts import LTPage
def rel_path(*arg: List[str]) -> str:
return os.path.join(os.path.dirname(__file__), *arg)
print(extract_items_from_pdf(rel_path('pdf/chars.pdf')))
# for page in extract_items_from_pdf(rel_path('pdf/chars.pdf')):
# print('page-start', page)
# for item in page:
# print(' ', item)
# print('page-end', page)
Binary file not shown.
Binary file not shown.
+3 -2
View File
@@ -10,7 +10,7 @@ import sys
with open('README.md') as readme_file: with open('README.md') as readme_file:
SETUP = { SETUP = {
"name": "pdfmajor", "name": "pdfmajor",
"version": "1.1.1", "version": "1.2.0",
"packages": find_packages(include=['pdfmajor*']), "packages": find_packages(include=['pdfmajor*']),
"install_requires": [ "install_requires": [
'pycryptodome', 'pycryptodome',
@@ -46,8 +46,9 @@ with open('README.md') as readme_file:
# ============== # ==============
# Update __init__.py # Update __init__.py
# ============== # ==============
from bin.recreate_init import update_pdfmajor__init___ from bin.recreate_init import update_pdfmajor__init___, update_current_version_lock
update_pdfmajor__init___(SETUP["version"], SETUP["long_description"]) update_pdfmajor__init___(SETUP["version"], SETUP["long_description"])
update_current_version_lock(SETUP["version"])
# ============== # ==============
# Run Setup # Run Setup