31 Commits
Author SHA1 Message Date
ari 3f561f3488 - removed none-develop dependency on tqdm
- ensured all test pdfs are properly pushed
2019-08-18 18:00:50 -04:00
ari 989de3122e corrected gitignore to not ignore lorem-v5.pdf 2019-08-18 17:59:14 -04:00
ari 20be0aa49e removed dependency on tqdm 2019-08-18 17:55:08 -04:00
ari d37b56a685 Merge branch 'master' of github.com:asosnovsky/pdfmajor 2019-06-10 16:28:23 -04:00
ari 9e923de7ec - added better mypy support
- updated most unit-test to use the unitttest module
- added a single unit-test run file
2019-06-10 16:26:52 -04:00
Ari Sosnovsky 2c9593ac21 Update README.md 2019-04-01 17:37:13 -04:00
Ari Sosnovsky 057cc0cf28 Update README.md 2019-04-01 17:36:36 -04:00
Ari Sosnovsky 82b47f1b19 Update README.md 2019-04-01 17:35:24 -04:00
ari 290de97cb5 - fixed invisible color bug 2019-01-26 23:08:57 -05:00
ari b32aef7fda - fixed the curve ignoring of the color error 2019-01-24 21:38:47 -05:00
ari 14ba20be55 - better repr for curves and colors 2019-01-23 22:12:14 -05:00
ari 18938d91ea - better names for commands 2019-01-23 22:11:58 -05:00
ari eb38854076 updated gitinore 2019-01-23 22:11:46 -05:00
ari 671ae5bc4a running tests before every pip deploy 2019-01-23 22:10:51 -05:00
ari 844408c6e3 - ensured that all tests are working
- added a main test script
2019-01-23 22:06:12 -05:00
ari bab104885f - fixed caching error when running the interpreter multiple times on items with xobjects 2019-01-23 15:56:46 -05:00
ari 98f18c3e32 - corrected caching error
- added lorem ipsum pdf test docs
- introduced a new test for detecting if some unwanted caching is occuring
- muting "Unsupported" warning
2019-01-23 15:56:25 -05:00
ari ac2c462561 - improved html converter 2019-01-22 13:16:46 -05:00
ari 19712f1f0f cleaner html converter 2019-01-22 13:16:17 -05:00
ari ec02a7d38b - yaml converter
- adjusting curve detection
2019-01-16 16:56:14 -05:00
ari 0345ca01ba - slightly more accurate shape detection 2019-01-16 16:55:30 -05:00
ari b36f6bc907 silencing the unknown operator warning 2019-01-16 12:08:15 -05:00
ari f9f9a59793 - added yaml converter 2019-01-16 12:08:03 -05:00
ari b7026793d7 - minor bug fix -- xobj invalid name 2019-01-13 13:13:37 -05:00
ari addd59ed0f updated readme 2019-01-11 15:25:03 -05:00
ari d43b617309 - new doc build 2019-01-11 15:23:15 -05:00
ari 17b25c7d9b - correcting the docs 2019-01-11 15:21:57 -05:00
ari b7fab64801 - correcting the doc 2019-01-11 15:21:27 -05:00
ari 609bcda76a - returned image writer
- added .save_image method to LTImage
2019-01-04 15:43:30 -05:00
ari ef230847ac - image extraction 2019-01-04 15:43:07 -05:00
ari d1d1b23f2f updated docs 2018-12-24 01:52:52 -05:00
62 changed files with 1351 additions and 674 deletions

No files matched your search

+4 -1
View File
@@ -10,4 +10,7 @@ tests/*.txt
.idea/ .idea/
.vscode .vscode
tests/samples/output tests/samples/output
**__pycache** ./samples
**__pycache**\
.mypy_cache
.ipynb_checkpoints
+1 -1
View File
@@ -1 +1 @@
v1.3.1 v1.3.9
+5 -5
View File
@@ -11,15 +11,15 @@
</tr> </tr>
</table> </table>
PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. The ultimate goal of this project is to provide a simpler, faster and more functional library that both casual and low-level developers could build upon. PDF-Major is a **complete re-write** of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. The ultimate goal of this project is to provide a simpler, faster and more functional library that both casual and low-level developers could build upon.
## Documentation ## Documentation
See documentation [here](https://asosnovsky.github.io/pdfmajor/). See [full API documentation here](https://asosnovsky.github.io/pdfmajor/api/).
## Basic Example ## Basic Example
```py ```py
from pdfmajor.interpreter import PageInterpreter from pdfmajor.interpreter import PDFInterpreter
for page in PDFInterpreter("/path/to/pdf.pdf"): for page in PDFInterpreter("/path/to/pdf.pdf"):
print("page start", page.page_num) print("page start", page.page_num)
@@ -29,9 +29,9 @@ for page in PDFInterpreter("/path/to/pdf.pdf"):
``` ```
## Why Another Fork? ## Why a re-write?
`PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. However, after attempting to build and extend upon `PDFMiner.six` I found it rather difficult to extend and identify additional details regarding the items encoded in the pdf. This library will attempt to expose as much information as possible to end-users, without having them solely rely on to-xml or to-json exports. `PDFMiner` was designed to run in python 2.7, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. However, after attempting to build and extend upon `PDFMiner.six` I found it rather difficult to extend and identify additional details regarding the items encoded in the pdf. This library will attempt to expose as much information as possible to end-users, without having them solely rely on to-xml or to-json exports.
## Features ## Features
+12 -7
View File
@@ -6,13 +6,18 @@ cd ..
./bin/clean.sh ./bin/clean.sh
python setup.py sdist bdist_wheel
git add . if ./bin/run_tests.sh; then
git commit python setup.py sdist bdist_wheel
git push
git tag $(cat .version) git add .
git push -u origin $(cat .version) git commit
git push
twine upload dist/* git tag $(cat .version)
git push -u origin $(cat .version)
twine upload dist/*
else
echo ">>> Stopped due to test failure!"
fi
+13
View File
@@ -0,0 +1,13 @@
#!/bin/bash
# Get working dir (this will make sure that the script will work the same anywhere you run it)
WK_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
cd $WK_DIR
cd ..
for filename in ./tests/test*.py; do
if python $filename ; then
echo woohoo
else
exit 1
fi
done
+3 -2
View File
@@ -16,7 +16,7 @@ a interpreter of the pdf-standard commands
### Example ### Example
```py ```py
from pdfmajor.interpreter import PageInterpreter from pdfmajor.interpreter import PDFInterpreter
for page in PDFInterpreter("/path/to/pdf.pdf"): for page in PDFInterpreter("/path/to/pdf.pdf"):
print("page start", page.page_num) print("page start", page.page_num)
@@ -58,7 +58,8 @@ All layout items extend the `LTItem` class. There are two kinds of layout items:
All of these classes extend the LTContainer class. All of these classes extend the LTContainer class.
- LTXObject: a layout item containing other additional layout items - LTXObject: a layout item containing other additional layout items
- LTCharBlock: a layout item containing LTChars - LTCharBlock: a layout item containing LTChars, this corresponds to whenever a `TJ` or `Tj` operators is issued within a text object.
- LTTextBlock: a layout item containing LTCharBlocks, note that this directly corresponds to the `BT` and `ET` operators pair in the pdf standard
#### Layout Components #### Layout Components
All of these classes extend the LTComponent class. All of these classes extend the LTComponent class.
+1 -1
View File
@@ -19,7 +19,7 @@ See documentation [here](https://asosnovsky.github.io/pdfmajor/).
## Basic Example ## Basic Example
```py ```py
from pdfmajor.interpreter import PageInterpreter from pdfmajor.interpreter import PDFInterpreter
for page in PDFInterpreter("/path/to/pdf.pdf"): for page in PDFInterpreter("/path/to/pdf.pdf"):
print("page start", page.page_num) print("page start", page.page_num)
+71 -65
View File
@@ -62,14 +62,14 @@
<ul> <ul>
<li><a class="toctree-l3" href="#parser">parser</a></li>
<li><a class="toctree-l3" href="#interpreter">interpreter</a></li> <li><a class="toctree-l3" href="#interpreter">interpreter</a></li>
<li><a class="toctree-l3" href="#converters">converters</a></li> <li><a class="toctree-l3" href="#converters">converters</a></li>
<li><a class="toctree-l3" href="#imagewriter">imagewriter</a></li> <li><a class="toctree-l3" href="#imagewriter">imagewriter</a></li>
<li><a class="toctree-l3" href="#layouts">layouts</a></li>
</ul> </ul>
@@ -111,37 +111,82 @@
<h1 id="api">API</h1> <h1 id="api">API</h1>
<p>The library is constructed from 4 base modules:</p> <p>The library is constructed from 4 base modules:</p>
<ul> <ul>
<li><strong>interpreter</strong>: contains low-level classes for extracting fundamental data-structures from the documents</li> <li><strong>parser</strong>: contains low-level classes for extracting fundamental data-structures from the documents</li>
<li><strong>converters</strong>: contains high-level classes for conversion of the fundamental pdf structures to other formats</li> <li><strong>interpreter</strong>: a interpreter of the pdf-standard commands</li>
<li><strong>converters</strong>: contains high-level functions for conversion of the fundamental pdf structures to other formats</li>
<li><strong>imagewriter</strong>: contains a simple implementation for converting PDF Image Streams to png/bmp/img formats</li> <li><strong>imagewriter</strong>: contains a simple implementation for converting PDF Image Streams to png/bmp/img formats</li>
<li><strong>layouts</strong>: contains an abstraction layer for the data-structures found in the pdf</li>
</ul> </ul>
<h2 id="parser">parser</h2>
<p>WIP</p>
<h2 id="interpreter">interpreter</h2> <h2 id="interpreter">interpreter</h2>
<pre><code>WIP <p>a interpreter of the pdf-standard commands</p>
</code></pre>
<h2 id="converters">converters</h2>
<p>Contains high-level classes for conversion of the fundamental pdf structures to other formats. This library includes 4 high-level conversion classes:</p>
<ul>
<li>HTMLConverter</li>
<li>JSONConverter</li>
<li>XMLConverter</li>
<li>TextConverter</li>
</ul>
<p>These classes all extend the <a href="#converters.PDFConverter">PDFConverter</a>. To use them simply call the static method <a href="#converterspdfconverterparse_fileinput_file-output_file-image_folder_path-codec-maxpages-password-caching-check_extractable-pagenos">parse_file</a>.</p>
<h3 id="example">Example</h3> <h3 id="example">Example</h3>
<pre><code class="py">from pdfmajor.converters import HTMLConverter <pre><code class="py">from pdfmajor.interpreter import PDFInterpreter
input_file = open('path/to/input/file.pdf', 'rb') for page in PDFInterpreter(&quot;/path/to/pdf.pdf&quot;):
output_file = open('path/to/output/file.html', 'wb') print(&quot;page start&quot;, page.page_num)
HTMLConverter.parse_file( for item in page:
input_file, print(&quot; &gt;&quot;, item)
output_file print(&quot;page end&quot;, page.page_num)
)
input_file.close()
output_file.close()
</code></pre> </code></pre>
<h3 id="convertersconvert_fileinput_file-output_file-image_folder_path-codec-maxpages-password-caching-check_extractable-pagenos-out_type">converters.convert_file(input_file, output_file, image_folder_path, codec, maxpages, password, caching, check_extractable, pagenos, out_type)</h3> <h3 id="interpreterpdfinterpreter">interpreter.PDFInterpreter</h3>
<p>This generator-function yields individual pages which contain their respected items.</p>
<h4 id="arguments">Arguments</h4>
<ul>
<li><code>input_file_path</code>: <a href="#">str</a></li>
<li><code>preload</code>: <a href="#">bool</a> defaults to False</li>
<li><code>maxpages</code>: <a href="#">int</a> defaults to 0 </li>
<li><code>password</code>: <a href="#">str</a> defaults to None </li>
<li><code>caching</code>: <a href="#">bool</a> defaults to True </li>
<li><code>check_extractable</code>: <a href="#">bool</a> defaults to True</li>
<li><code>pagenos</code>: <a href="#">List[int]</a> defaults to None</li>
<li><code>debug_level</code>: <a href="#https://docs.python.org/3/library/logging.html#levels">logging.levels</a> defaults logging.WARNING</li>
</ul>
<h4 id="yield-value">Yield Value</h4>
<p>This function returns a generator that yields <a href="#interpreterpageinterpreter">PDFInterpreter</a>.</p>
<h3 id="interpreterpageinterpreter">interpreter.PageInterpreter</h3>
<p>This generator-function-class yields individual <a href="#layout-items">layout items</a>.</p>
<h3 id="layout-items">Layout Items</h3>
<p>All layout items extend the <code>LTItem</code> class. There are two kinds of layout items:</p>
<ul>
<li>LTComponent: extends the base <code>LTItem</code> class, this class will have additional values such as boundary boxes, height and width</li>
<li>LTContainer: extends the <code>LTComponent</code> class, this class is used to contain elements of the pdf that would have child elements. Iterating on this element will output its child elements.</li>
</ul>
<h4 id="layout-containers">Layout Containers</h4>
<p>All of these classes extend the LTContainer class.</p>
<ul>
<li>LTXObject: a layout item containing other additional layout items</li>
<li>LTCharBlock: a layout item containing LTChars, this corresponds to whenever a <code>TJ</code> or <code>Tj</code> operators is issued within a text object.</li>
<li>LTTextBlock: a layout item containing LTCharBlocks, note that this directly corresponds to the <code>BT</code> and <code>ET</code> operators pair in the pdf standard</li>
</ul>
<h4 id="layout-components">Layout Components</h4>
<p>All of these classes extend the LTComponent class.</p>
<ul>
<li>LTChar: an individual character</li>
<li>LTCurves: represents a collection of svg-paths (available under <code>self.paths</code>)</li>
<li>LTImage: a component containing information regarding an image</li>
</ul>
<h2 id="converters">converters</h2>
<p>Contains high-level functions for conversion of the fundamental pdf structures to other formats. This library includes 4 high-level conversion cases:</p>
<ul>
<li>HTML</li>
<li>JSON</li>
<li>XML</li>
<li>Text</li>
</ul>
<p>These formats are all generated using the <a href="#PDFInterpreter">PageInterpreter</a>. To use them simply call the static method <a href="#converterspdfconverter">parse_file</a>.</p>
<h3 id="example_1">Example</h3>
<pre><code class="py">from pdfmajor.converters import convert_file
convert_file(
&quot;path/to/input/file.pdf&quot;,
&quot;path/to/output/file.html&quot;,
out_type=&quot;html&quot;
)
</code></pre>
<h3 id="convertersconvert_file">converters.convert_file</h3>
<p>A high-level abstraction for the conversion classes.</p> <p>A high-level abstraction for the conversion classes.</p>
<ul> <ul>
<li><code>input_file</code>: <a href="https://docs.python.org/3/library/io.html#io.TextIOWrapper">TextIOWrapper</a> </li> <li><code>input_file</code>: <a href="https://docs.python.org/3/library/io.html#io.TextIOWrapper">TextIOWrapper</a> </li>
@@ -155,46 +200,7 @@ output_file.close()
<li><code>pagenos</code>: <a href="#">List[int]</a> defaults to None</li> <li><code>pagenos</code>: <a href="#">List[int]</a> defaults to None</li>
<li><code>out_type</code>: <a href="#">str</a> defaults to 'html'</li> <li><code>out_type</code>: <a href="#">str</a> defaults to 'html'</li>
</ul> </ul>
<h3 id="example-convertersconvert_file">Example (converters.convert_file)</h3>
<pre><code class="py">from pdfmajor.converters import convert_file
input_file = open('path/to/input/file.pdf', 'rb')
output_file = open('path/to/output/file.html', 'wb')
convert_file(
input_file,
output_file
)
input_file.close()
output_file.close()
</code></pre>
<h3 id="converterspdfconverter">converters.PDFConverter</h3>
<p>Extends <code>layouts.PDFLayoutAnalyzer</code>.</p>
<h4 id="converterspdfconverter9595init9595rsrcmgr-outfp-imagewriter-codec-pageno">converters.PDFConverter.__init__(rsrcmgr, outfp, imagewriter, codec, pageno)</h4>
<ul>
<li><code>rsrcmgr</code>: <a href="#interpreter.PDFResourceManager">PDFResourceManager</a></li>
<li><code>outfp</code>: <a href="https://docs.python.org/3/library/io.html#io.TextIOWrapper">TextIOWrapper</a></li>
<li><code>imagewriter</code>: <a href="#imagewriter.ImageWriter">ImageWriter</a></li>
<li><code>codec</code>: <a href="#">str</a> defaults to 'utf-8'</li>
<li><code>pageno</code>: <a href="#">int</a> defaults to 1</li>
</ul>
<p>initialization function for the class</p>
<h4 id="converterspdfconverterparse_fileinput_file-output_file-image_folder_path-codec-maxpages-password-caching-check_extractable-pagenos">converters.PDFConverter.parse_file(input_file, output_file, image_folder_path, codec, maxpages, password, caching, check_extractable, pagenos)</h4>
<p>returns the output file <a href="https://docs.python.org/3/library/io.html#io.TextIOWrapper">TextIOWrapper</a>.</p>
<ul>
<li><code>input_file</code>: <a href="https://docs.python.org/3/library/io.html#io.TextIOWrapper">TextIOWrapper</a> </li>
<li><code>output_file</code>: <a href="https://docs.python.org/3/library/io.html#io.TextIOWrapper">TextIOWrapper</a> </li>
<li><code>image_folder_path</code>: <a href="#">str</a> defaults to None</li>
<li><code>codec</code>: <a href="#">str</a> defaults to 'utf-8'</li>
<li><code>maxpages</code>: <a href="#">int</a> defaults to 0 </li>
<li><code>password</code>: <a href="#">str</a> defaults to None </li>
<li><code>caching</code>: <a href="#">bool</a> defaults to True </li>
<li><code>check_extractable</code>: <a href="#">bool</a> defaults to True</li>
<li><code>pagenos</code>: <a href="#">List[int]</a> defaults to None</li>
</ul>
<h2 id="imagewriter">imagewriter</h2> <h2 id="imagewriter">imagewriter</h2>
<p>WIP </p>
<h2 id="layouts">layouts</h2>
<p>WIP </p> <p>WIP </p>
</div> </div>
+20 -7
View File
@@ -59,6 +59,8 @@
<li><a class="toctree-l3" href="#documentation">Documentation</a></li> <li><a class="toctree-l3" href="#documentation">Documentation</a></li>
<li><a class="toctree-l3" href="#basic-example">Basic Example</a></li>
<li><a class="toctree-l3" href="#why-another-fork">Why Another Fork?</a></li> <li><a class="toctree-l3" href="#why-another-fork">Why Another Fork?</a></li>
<li><a class="toctree-l3" href="#features">Features</a></li> <li><a class="toctree-l3" href="#features">Features</a></li>
@@ -122,23 +124,34 @@
</tr> </tr>
</table> </table>
<p>PDF-Major is a fork of <code>PDFMiner.six</code>. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability.</p> <p>PDF-Major is a fork of <code>PDFMiner.six</code>. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. The ultimate goal of this project is to provide a simpler, faster and more functional library that both casual and low-level developers could build upon.</p>
<h2 id="documentation">Documentation</h2> <h2 id="documentation">Documentation</h2>
<p>See documentation <a href="https://asosnovsky.github.io/pdfmajor/">here</a>.</p> <p>See documentation <a href="https://asosnovsky.github.io/pdfmajor/">here</a>.</p>
<h2 id="basic-example">Basic Example</h2>
<pre><code class="py">from pdfmajor.interpreter import PDFInterpreter
for page in PDFInterpreter(&quot;/path/to/pdf.pdf&quot;):
print(&quot;page start&quot;, page.page_num)
for item in page:
print(&quot; &gt;&quot;, item)
print(&quot;page end&quot;, page.page_num)
</code></pre>
<h2 id="why-another-fork">Why Another Fork?</h2> <h2 id="why-another-fork">Why Another Fork?</h2>
<p><code>PDFMiner</code> was designed to run in python 2.7&gt;, <code>PDFMiner.six</code> was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with. </p> <p><code>PDFMiner</code> was designed to run in python 2.7&gt;, <code>PDFMiner.six</code> was written to bring cross-version support for python 2 and 3. However, after attempting to build and extend upon <code>PDFMiner.six</code> I found it rather difficult to extend and identify additional details regarding the items encoded in the pdf. This library will attempt to expose as much information as possible to end-users, without having them solely rely on to-xml or to-json exports.</p>
<h2 id="features">Features</h2> <h2 id="features">Features</h2>
<ul> <ul>
<li>Written entirely in Python.</li> <li>Functional extraction method based on generators</li>
<li>Parse, analyze, and convert PDF documents.</li> <li>Parse, analyze, and convert PDF documents.</li>
<li>PDF-1.7 specification support. (well, almost)</li> <li>PDF-1.7 specification support. (well, almost)</li>
<li>Font-Color extraction</li>
<li>Shape fill and stroke color extraction</li>
<li>CJK languages and vertical writing scripts support.</li> <li>CJK languages and vertical writing scripts support.</li>
<li>Various font types (Type1, TrueType, Type3, and CID) support.</li> <li>Various font types (Type1, TrueType, Type3, and CID) support.</li>
<li>Basic encryption (RC4) support.</li> <li>Basic encryption (RC4) support.</li>
<li>Font-Color extraction</li>
<li>Shape fill and stroke color extraction</li>
</ul> </ul>
<p><strong>Note</strong>: We took out the layout-analysis process in this version (there is no more LTTextHorizontal or LTTextVertical). While the mathematics behind the grouping process was sound, the coupling of the layout-analysis process with the parsing and interpretation process produced unfriendly-code. This feature could be brought back by extending the <code>pdfmajor.layouts.PDFLayoutAnalyzer</code> class, but at the current time is not supported.</p> <p><strong>Note</strong>: We took out the layout-analysis process in this version (there is no more LTTextHorizontal or LTTextVertical). While the mathematics behind the grouping process was sound, the coupling of the layout-analysis process with the parsing and interpretation process produced unfriendly-code. This feature could be brought back by running your own implementation of it on the <code>pdfmajor.interpreter.PageInterpreter</code> class, but at the current time is not supported.</p>
<h2 id="how-to-install">How to Install</h2> <h2 id="how-to-install">How to Install</h2>
<h3 id="source">Source</h3> <h3 id="source">Source</h3>
<ul> <ul>
@@ -227,5 +240,5 @@ SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.</p>
<!-- <!--
MkDocs version : 1.0.4 MkDocs version : 1.0.4
Build Date UTC : 2018-11-04 21:27:38 Build Date UTC : 2019-01-11 20:22:59
--> -->
+1 -1
View File
@@ -1 +1 @@
{"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"PDFMajor Latest Release PDF-Major is a fork of PDFMiner.six . It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability. Documentation See documentation here . Why Another Fork? PDFMiner was designed to run in python 2.7>, PDFMiner.six was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with. Features Written entirely in Python. Parse, analyze, and convert PDF documents. PDF-1.7 specification support. (well, almost) CJK languages and vertical writing scripts support. Various font types (Type1, TrueType, Type3, and CID) support. Basic encryption (RC4) support. Font-Color extraction Shape fill and stroke color extraction Note : We took out the layout-analysis process in this version (there is no more LTTextHorizontal or LTTextVertical). While the mathematics behind the grouping process was sound, the coupling of the layout-analysis process with the parsing and interpretation process produced unfriendly-code. This feature could be brought back by extending the pdfmajor.layouts.PDFLayoutAnalyzer class, but at the current time is not supported. How to Install Source Install Python 3.6.4 or newer. clone this repo git clone https://github.com/asosnovsky/pdfmajor * install repo python setup.py install Pypi Install Python 3.6.4 or newer. install repo pip install pdfmajor Terms and Conditions (This is so-called MIT/X License) Copyright (c) 2018-2019 Ariel Sosnovsky Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.","title":"Home"},{"location":"#pdfmajor","text":"Latest Release PDF-Major is a fork of PDFMiner.six . It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability.","title":"PDFMajor"},{"location":"#documentation","text":"See documentation here .","title":"Documentation"},{"location":"#why-another-fork","text":"PDFMiner was designed to run in python 2.7>, PDFMiner.six was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with.","title":"Why Another Fork?"},{"location":"#features","text":"Written entirely in Python. Parse, analyze, and convert PDF documents. PDF-1.7 specification support. (well, almost) CJK languages and vertical writing scripts support. Various font types (Type1, TrueType, Type3, and CID) support. Basic encryption (RC4) support. Font-Color extraction Shape fill and stroke color extraction Note : We took out the layout-analysis process in this version (there is no more LTTextHorizontal or LTTextVertical). While the mathematics behind the grouping process was sound, the coupling of the layout-analysis process with the parsing and interpretation process produced unfriendly-code. This feature could be brought back by extending the pdfmajor.layouts.PDFLayoutAnalyzer class, but at the current time is not supported.","title":"Features"},{"location":"#how-to-install","text":"","title":"How to Install"},{"location":"#source","text":"Install Python 3.6.4 or newer. clone this repo git clone https://github.com/asosnovsky/pdfmajor * install repo python setup.py install","title":"Source"},{"location":"#pypi","text":"Install Python 3.6.4 or newer. install repo pip install pdfmajor","title":"Pypi"},{"location":"#terms-and-conditions","text":"(This is so-called MIT/X License) Copyright (c) 2018-2019 Ariel Sosnovsky Permission is hereby granted, free of chargeLine truncated {"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"PDFMajor Latest Release PDF-Major is a fork of PDFMiner.six . It is meant to be a more light-weight implementation that makes fewer assumptions on the data. The ultimate goal of this project is to provide a simpler, faster and more functional library that both casual and low-level developers could build upon. Documentation See documentation here . Basic Example from pdfmajor.interpreter import PDFInterpreter for page in PDFInterpreter(\"/path/to/pdf.pdf\"): print(\"page start\", page.page_num) for item in page: print(\" >\", item) print(\"page end\", page.page_num) Why Another Fork? PDFMiner was designed to run in python 2.7>, PDFMiner.six was written to bring cross-version support for python 2 and 3. However, after attempting to build and extend upon PDFMiner.six I found it rather difficult to extend and identify additional details regarding the items encoded in the pdf. This library will attempt to expose as much information as possible to end-users, without having them solely rely on to-xml or to-json exports. Features Functional extraction method based on generators Parse, analyze, and convert PDF documents. PDF-1.7 specification support. (well, almost) Font-Color extraction Shape fill and stroke color extraction CJK languages and vertical writing scripts support. Various font types (Type1, TrueType, Type3, and CID) support. Basic encryption (RC4) support. Note : We took out the layout-analysis process in this version (there is no more LTTextHorizontal or LTTextVertical). While the mathematics behind the grouping process was sound, the coupling of the layout-analysis process with the parsing and interpretation process produced unfriendly-code. This feature could be brought back by running your own implementation of it on the pdfmajor.interpreter.PageInterpreter class, but at the current time is not supported. How to Install Source Install Python 3.6.4 or newer. clone this repo git clone https://github.com/asosnovsky/pdfmajor * install repo python setup.py install Pypi Install Python 3.6.4 or newer. install repo pip install pdfmajor Terms and Conditions (This is so-called MIT/X License) Copyright (c) 2018-2019 Ariel Sosnovsky Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.","title":"Home"},{"location":"#pdfmajor","text":"Latest Release PDF-Major is a fork of PDFMiner.six . It is meant to be a more light-weight implementation that makes fewer assumptions on the data. The ultimate goal of this project is to provide a simpler, faster and more functional library that both casual and low-level developers could build upon.","title":"PDFMajor"},{"location":"#documentation","text":"See documentation here .","title":"Documentation"},{"location":"#basic-example","text":"from pdfmajor.interpreter import PDFInterpreter for page in PDFInterpreter(\"/path/to/pdf.pdf\"): print(\"page start\", page.page_num) for item in page: print(\" >\", item) print(\"page end\", page.page_num)","title":"Basic Example"},{"location":"#why-another-fork","text":"PDFMiner was designed to run in python 2.7>, PDFMiner.six was written to bring cross-version support for python 2 and 3. However, after attempting to build and extend upon PDFMiner.six I found it rather difficult to extend and identify additional details regarding the items encoded in the pdf. This library will attempt to expose as much information as possible to end-users, without having them solely rely on to-xml or to-json exports.","title":"Why Another Fork?"},{"location":"#features","text":"Functional extraction method based on generators Parse, analyze, and convert PDF documents. PDF-1.7 specification support. (well, almost) Font-Color extraction Shape fill and stroke color extraction CJK languages and vertical writing scripts support. Various font types (Type1, TrueType, Type3, and CID) support. Basic encryption (RC4) support. Note : We took out the layout-analysis process in this version (there is no more LTTexLine truncated
+2 -2
View File
@@ -2,12 +2,12 @@
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url> <url>
<loc>None</loc> <loc>None</loc>
<lastmod>2018-11-04</lastmod> <lastmod>2019-01-11</lastmod>
<changefreq>daily</changefreq> <changefreq>daily</changefreq>
</url> </url>
<url> <url>
<loc>None</loc> <loc>None</loc>
<lastmod>2018-11-04</lastmod> <lastmod>2019-01-11</lastmod>
<changefreq>daily</changefreq> <changefreq>daily</changefreq>
</url> </url>
</urlset> </urlset>
Binary file not shown.
+60
View File
@@ -0,0 +1,60 @@
import struct
from io import BytesIO
from ..parser.PDFStream import LITERALS_DCT_DECODE, PDFStream
def align32(x):
return ((x+3)//4)*4
## BMPWriter
##
class BMPWriter(object):
@classmethod
def write_all(cls, fp, stream: PDFStream, bits: int, width: float, height: float, step: float):
bmp = cls(fp, 1, width, height)
data = stream.get_data()
i = 0
for y in range(height):
bmp.write_line(y, data[i:i+step])
i += step
def __init__(self, fp, bits, width, height):
self.fp = fp
self.bits = bits
self.width = width
self.height = height
if bits == 1:
ncols = 2
elif bits == 8:
ncols = 256
elif bits == 24:
ncols = 0
else:
raise ValueError(bits)
self.linesize = align32((self.width*self.bits+7)//8)
self.datasize = self.linesize * self.height
headersize = 14+40+ncols*4
info = struct.pack('<IiiHHIIIIII', 40, self.width, self.height, 1, self.bits, 0, self.datasize, 0, 0, ncols, 0)
assert len(info) == 40, str(len(info))
header = struct.pack('<ccIHHI', b'B', b'M', headersize+self.datasize, 0, 0, headersize)
assert len(header) == 14, str(len(header))
self.fp.write(header)
self.fp.write(info)
if ncols == 2:
# B&W color table
for i in (0, 255):
self.fp.write(struct.pack('BBBx', i, i, i))
elif ncols == 256:
# grayscale color table
for i in range(256):
self.fp.write(struct.pack('BBBx', i, i, i))
self.pos0 = self.fp.tell()
self.pos1 = self.pos0 + self.datasize
return
def write_line(self, y, data):
self.fp.seek(self.pos1 - (y+1)*self.linesize)
self.fp.write(data)
return
+6 -6
View File
@@ -16,15 +16,15 @@
</tr> </tr>
</table> </table>
PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. The ultimate goal of this project is to provide a simpler, faster and more functional library that both casual and low-level developers could build upon. PDF-Major is a **complete re-write** of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. The ultimate goal of this project is to provide a simpler, faster and more functional library that both casual and low-level developers could build upon.
## Documentation ## Documentation
See documentation [here](https://asosnovsky.github.io/pdfmajor/). See [full API documentation here](https://asosnovsky.github.io/pdfmajor/api/).
## Basic Example ## Basic Example
```py ```py
from pdfmajor.interpreter import PageInterpreter from pdfmajor.interpreter import PDFInterpreter
for page in PDFInterpreter("/path/to/pdf.pdf"): for page in PDFInterpreter("/path/to/pdf.pdf"):
print("page start", page.page_num) print("page start", page.page_num)
@@ -34,9 +34,9 @@ for page in PDFInterpreter("/path/to/pdf.pdf"):
``` ```
## Why Another Fork? ## Why a re-write?
`PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. However, after attempting to build and extend upon `PDFMiner.six` I found it rather difficult to extend and identify additional details regarding the items encoded in the pdf. This library will attempt to expose as much information as possible to end-users, without having them solely rely on to-xml or to-json exports. `PDFMiner` was designed to run in python 2.7, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. However, after attempting to build and extend upon `PDFMiner.six` I found it rather difficult to extend and identify additional details regarding the items encoded in the pdf. This library will attempt to expose as much information as possible to end-users, without having them solely rely on to-xml or to-json exports.
## Features ## Features
@@ -97,7 +97,7 @@ OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
""" """
__version__ = (1, 3, 1) __version__ = (1, 3, 9)
if __name__ == '__main__': if __name__ == '__main__':
print( print(
+23 -8
View File
@@ -2,33 +2,34 @@ import io
import os import os
import re import re
from typing import List from typing import List, Optional
from .html import convert_to_html from .html import convert_to_html
from .xml import convert_to_xml from .xml import convert_to_xml
from .json import convert_to_json from .json import convert_to_json
from .text import convert_to_text from .text import convert_to_text
from .yaml import convert_to_yaml
from ..utils import logging from ..utils import logging
def convert_file( def convert_file(
input_file: str, input_file: str,
output_file: str = None, output_file: Optional[str] = None,
image_folder_path: str = None, image_folder_path: Optional[str] = None,
codec: str = 'utf-8', codec: str = 'utf-8',
maxpages: int = 0, maxpages: int = 0,
password: str = None, password: Optional[str] = None,
caching: bool = True, caching: bool = True,
check_extractable: bool = True, check_extractable: bool = True,
pagenos: List[int] = None, pagenos: Optional[List[int]] = None,
dont_export_images: bool = False, dont_export_images: bool = False,
debug_level: int = logging.WARNING, debug_level: int = logging.WARNING,
out_type: str = 'html', out_type: str = 'html',
): ):
if output_file is None: if output_file is None:
output_file = os.path.join( output_file = os.path.join(
os.path.dirname(input_file.name), os.path.dirname(input_file),
os.path.basename( os.path.basename(
re.sub(r"\.\w+$", "."+out_type, input_file.name) re.sub(r"\.\w+$", "."+out_type, input_file)
) )
) )
if re.search(re.escape(out_type) + r'$', output_file ) is None: if re.search(re.escape(out_type) + r'$', output_file ) is None:
@@ -75,6 +76,20 @@ def convert_file(
dont_export_images=dont_export_images, dont_export_images=dont_export_images,
debug_level=debug_level, debug_level=debug_level,
) )
elif out_type == 'yaml' or out_type == 'yml':
return convert_to_yaml(
input_file_path=input_file,
output_file_path=output_file,
image_folder_path=image_folder_path,
codec=codec,
maxpages=maxpages,
password=password,
caching=caching,
check_extractable=check_extractable,
pagenos=pagenos,
dont_export_images=dont_export_images,
debug_level=debug_level,
)
elif out_type == 'text': elif out_type == 'text':
return convert_to_text( return convert_to_text(
input_file_path=input_file, input_file_path=input_file,
@@ -89,4 +104,4 @@ def convert_file(
dont_export_images=dont_export_images, dont_export_images=dont_export_images,
debug_level=debug_level, debug_level=debug_level,
) )
else: raise Exception("Please specify out_type as 'html' or 'xml' or 'json' or 'text'") else: raise Exception("Please specify out_type as 'html' or 'xml' or 'json' or 'text' or 'yaml'")
+130 -99
View File
@@ -1,18 +1,20 @@
from typing import List from typing import List, Optional
from .writers.html import HTMLMaker from .writers.html import HTMLMaker
from ..interpreter import PDFInterpreter, PageInterpreter, logging from ..interpreter import PDFInterpreter, PageInterpreter, logging
from ..interpreter import LTImage, LTTextBlock, LTCharBlock, LTChar, LTCurve, LTXObject from ..interpreter import LTImage, LTTextBlock, LTCharBlock, LTChar, LTCurve, LTXObject
from ..interpreter.commands import LTItem from ..interpreter.commands import LTItem
from ..interpreter.commands.state import CurvePath, PDFColor from ..interpreter.commands.state import CurvePath, PDFColor
INF = (1<<31) - 1
def convert_to_html( def convert_to_html(
input_file_path: str, input_file_path: str,
output_file_path: str, output_file_path: str,
image_folder_path: str = None, image_folder_path: Optional[str] = None,
dont_export_images: bool = False, dont_export_images: bool = False,
codec: str = 'utf-8', codec: str = 'utf-8',
maxpages: int = 0, maxpages: int = 0,
password: str = None, password: Optional[str] = None,
caching: bool = True, caching: bool = True,
check_extractable: bool = True, check_extractable: bool = True,
pagenos: List[int] = None, pagenos: List[int] = None,
@@ -43,19 +45,21 @@ def convert_to_html(
render_page(html, page) render_page(html, page)
def render_page(html: HTMLMaker, ltpage: PageInterpreter): def render_page(html: HTMLMaker, ltpage: PageInterpreter):
def render(item: LTItem): def render(item: LTItem, block = True):
if isinstance(item, LTCurve): if isinstance(item, LTCurve):
render_curve(html, item) render_curve(html, ltpage, item)
elif isinstance(item, LTXObject): elif isinstance(item, LTXObject):
with html.elm('div', { "class": 'xobject' }, { with html.elm('div', { "class": 'xobject' }, {
'position': 'absolute', # 'position': 'absolute',
'left': f'{item.x0}px', 'left': f'{item.x0}px',
'right': f'{item.x1}px',
'bottom': f'{item.y0}px', 'bottom': f'{item.y0}px',
'top': f'{item.y1}px',
'width': f'{item.width}px', 'width': f'{item.width}px',
'height': f'{item.height}px', 'height': f'{item.height}px',
}): }):
for child in item: for child in item:
render(child) render(child, False)
elif isinstance(item, LTImage): elif isinstance(item, LTImage):
name = item.name name = item.name
# if html.imagewriter is not None: # if html.imagewriter is not None:
@@ -68,42 +72,42 @@ def render_page(html: HTMLMaker, ltpage: PageInterpreter):
}) })
elif isinstance(item, LTChar): elif isinstance(item, LTChar):
with html.elm('span', { 'class': 'char' }, { with html.elm('span', { 'class': 'char' }, {
'width': item.width, 'width': str(item.width) + "px",
'min-width': item.width, 'min-width': str(item.width) + "px",
'max-width': item.width, 'max-width': str(item.width) + "px",
'height': item.height, 'height': str(item.height) + "px",
'min-height': item.height, 'min-height': str(item.height) + "px",
'max-height': item.height, 'max-height': str(item.height) + "px",
}): }):
html.write(item.get_text()) html.write(item.get_text())
elif isinstance(item, LTTextBlock): elif isinstance(item, LTTextBlock):
for char in item: with html.elm('span', {'class': 'text-block'}):
render(char) for char in item:
render(char)
elif isinstance(item, LTCharBlock): elif isinstance(item, LTCharBlock):
with html.elm('span', {'class': 'char-block'}): with html.elm('span', { 'class': 'char-block' }, {
with html.elm('p', { 'class': 'text-block' }, { 'position': 'absolute',
'position': 'absolute', 'left': f'{item.x0}',
'left': f'{item.x0}', 'bottom': f'{item.y0}',
'bottom': f'{item.y0}', 'font-size': f'{item.size}px',
'font-size': f'{item.size}px', 'font-family': item.fontname,
'font-family': item.fontname, 'font-weight': item.font.font_weight,
'font-weight': item.font.font_weight, 'color': get_color(item.color, default="black"),
'color': get_color(item.color, default="black"), 'width': str(item.width) + "px",
'width': item.width, 'min-width': str(item.width) + "px",
'min-width': item.width, 'max-width': str(item.width) + "px",
'max-width': item.width, 'height': str(item.height) + "px",
'height': item.height, 'min-height': str(item.height) + "px",
'min-height': item.height, 'max-height': str(item.height) + "px",
'max-height': item.height, 'transform': f'skew({item.font.italic_angle}deg, 0deg)',
'transform': f'skew({item.font.italic_angle}deg, 0deg)', 'text-align': 'center',
'text-align': 'center', 'display': 'flex',
'display': 'flex', 'flex-flow': 'row nowrap',
'flex-flow': 'row nowrap', 'justify-content': 'space-between',
'justify-content': 'space-between', # 'padding-left': item.font.leading
# 'padding-left': item.font.leading }):
}): for char in item:
for char in item: render(char)
render(char)
with html.elm('div', { "class": 'page', "id": f"page-{ltpage.page_num}" }, { with html.elm('div', { "class": 'page', "id": f"page-{ltpage.page_num}" }, {
'width': f'{ltpage.width}px', 'width': f'{ltpage.width}px',
@@ -125,69 +129,96 @@ def get_color(col: PDFColor, default: str = ""):
else: else:
return default return default
def render_curve(html: HTMLMaker, item: LTCurve): def render_curve(html: HTMLMaker, ltpage: PageInterpreter, item: LTCurve):
css = { path, svg_attr = create_svg_path(item)
'position': 'absolute',
'left': f'{item.x0}px',
'bottom': f'{item.y0}px',
'width': f'{item.width}px',
'height': f'{item.height}px',
}
path = ""
svg_attr = {
'class': 'curve',
'z-index': 2
}
path_css = {
} stroke_col = get_color(item.stroke)
fill_col = get_color(item.fill)
path_css = {}
if fill_col:
path_css['fill'] = fill_col
if stroke_col:
path_css['stroke'] = stroke_col
# Compute ViewBox with html.elm('svg', svg_attr, css={
p_vals = { "x": [], "y": [] } 'border': '1px solid black',
'position': 'absolute',
for p in item.paths: 'left': f'{item.x0}px',
for pt in p.points: 'bottom': f'{ltpage.height - item.y0}px',
p_vals['x'].append(pt.x) 'top': f'{ltpage.height - item.y1}px',
p_vals['y'].append(pt.y) 'right': f'{item.x1}px',
'width': f'{item.width}px',
mx_x = max([*p_vals['x'], 1]) 'height': f'{item.height}px',
mn_x = min([*p_vals['x'], 0]) }):
mx_y = max([*p_vals['y'], 1]) html.singleton('path', {
mn_y = min([*p_vals['y'], 0]) "d": path,
**path_css
if mx_x == mn_x: }, {
mn_x -= 1 **path_css
if mx_y == mn_y:
mn_y -= 1
dx = 1/( mx_x - mn_x )
dy = (item.height/item.width)/( mx_y - mn_y )
ROUND_VAL = 5
svg_attr.update({
# "viewBox": f"{min(p_vals['x'])} {min(p_vals['y'])} {max(p_vals['x'])} {max(p_vals['y'])}",
"viewBox": f"{round(mn_x*dx, ROUND_VAL)} {round(mn_y*dy, ROUND_VAL)} {round(mx_x*dx , ROUND_VAL)} {round(mx_y*dy, ROUND_VAL)}",
}) })
# Compute shape def create_svg_path(item: LTCurve):
for p in item.paths: path = ""
if p.method == CurvePath.METHOD.MOVE_TO: svg_attr = {
path += f'M{" ".join( f"{round(dx*pt.x, ROUND_VAL)} {round(dy*pt.y, ROUND_VAL)}" for pt in p.points )} ' 'class': 'curve',
if p.method == CurvePath.METHOD.LINE_TO: 'z-index': 2
path += f'L{" ".join( f"{round(dx*pt.x, ROUND_VAL)} {round(dy*pt.y, ROUND_VAL)}" for pt in p.points )} ' }
if p.method == CurvePath.METHOD.CLOSE_PATH:
path += 'Z'
stroke_col = get_color(item.stroke) if item.height > 0:
fill_col = get_color(item.fill) dy = 1.0/item.height
if fill_col: dx = (item.width/item.height)/max([(item.x1-item.x0),0.01])
path_css['fill'] = fill_col svg_attr.update({
if stroke_col: "viewBox": "0 0 %s %s" % ( item.width/item.height, 1 )
path_css['stroke'] = stroke_col })
else:
dx = 1.0/(item.x1-item.x0)
dy = 1.0
svg_attr.update({
"viewBox": "0 0 1 1"
})
with html.elm('svg', svg_attr, css): if item.height > 1:
html.singleton('path', { svg_attr.update({
"d": path, 'height': f'{item.height}px',
**path_css })
}, { else:
**path_css svg_attr.update({
}) 'height': '1px',
})
if item.width > 1:
svg_attr.update({
'width': f'{item.width}px',
})
else:
svg_attr.update({
'width': '1px',
})
def pX(val: float) -> float:
return ( val-item.x0 )*dx
def pY(val: float) -> float:
return ( item.height- (val-item.y0) )*dy
for p in item.paths:
if p.method == CurvePath.METHOD.MOVE_TO:
path += f'M{" ".join( f"{pX(p.x)} {pY(p.y)}" for p in p.points )} '
elif p.method == CurvePath.METHOD.LINE_TO:
path += f'L{" ".join( f"{pX(p.x)} {pY(p.y)}" for p in p.points )} '
elif p.method == CurvePath.METHOD.CLOSE_PATH:
path += 'Z'
elif p.method == CurvePath.METHOD.CURVE_BOTH_TO:
path += f'C{" ".join( f"{pX(pt.x)} {pY(pt.y)}" for pt in p.points )} '
return path, svg_attr
# def render_textblock(html: HTMLMaker, ltpage: PageInterpreter, item: LTTextBlock):
# (x0, y0, x1, y1) = (INF, INF, -INF, -INF)
# for c_block in item:
# x0 = min(x0, c_block.x0)
# y0 = min(y0, c_block.y0)
# x1 = max(x1, c_block.x1)
# y1 = max(y1, c_block.y1)
# with html.elm('span')
+4 -4
View File
@@ -1,4 +1,4 @@
from typing import List from typing import List, Optional
from .writers.json import JSONMaker, JSONMakerObject, JSONMakerArray from .writers.json import JSONMaker, JSONMakerObject, JSONMakerArray
from ..interpreter import PDFInterpreter, PageInterpreter, logging from ..interpreter import PDFInterpreter, PageInterpreter, logging
@@ -21,14 +21,14 @@ def normalize_color_values(values: list):
def convert_to_json( def convert_to_json(
input_file_path: str, input_file_path: str,
output_file_path: str, output_file_path: str,
image_folder_path: str = None, image_folder_path: Optional[str] = None,
dont_export_images: bool = False, dont_export_images: bool = False,
codec: str = 'utf-8', codec: str = 'utf-8',
maxpages: int = 0, maxpages: int = 0,
password: str = None, password: str = Optional[None],
caching: bool = True, caching: bool = True,
check_extractable: bool = True, check_extractable: bool = True,
pagenos: List[int] = None, pagenos: Optional[List[int]] = None,
debug_level: int = logging.WARNING, debug_level: int = logging.WARNING,
): ):
intepreter = PDFInterpreter(input_file_path, intepreter = PDFInterpreter(input_file_path,
+7 -7
View File
@@ -1,3 +1,4 @@
from typing import Optional
from io import TextIOWrapper from io import TextIOWrapper
from contextlib import contextmanager from contextlib import contextmanager
@@ -6,7 +7,7 @@ class HTMLMaker:
def __init__(self, file_path: str, codec: str = 'utf-8'): def __init__(self, file_path: str, codec: str = 'utf-8'):
self.file_path = file_path self.file_path = file_path
self.codec = codec self.codec = codec
self.outfile: TextIOWrapper = None self.outfile: Optional[TextIOWrapper] = None
def __enter__(self): def __enter__(self):
self.outfile = open(self.file_path, 'wb') self.outfile = open(self.file_path, 'wb')
@@ -42,21 +43,20 @@ class HTMLMaker:
f'{name}: {value}' f'{name}: {value}'
for name, value in css.items() for name, value in css.items()
]) ])
attrs = " ".join([ attrs_str = " ".join([
f'{name}="{value}"' f'{name}="{value}"'
for name, value in attrs.items() for name, value in attrs.items()
]) ])
if not singleton: if not singleton:
self.write(f"<{tag_name} {attrs}>", lineend=lineend) self.write(f"<{tag_name} {attrs_str}>", lineend=lineend)
self.__levels_deep += 1 self.__levels_deep += 1
yield yield
self.__levels_deep -= 1 self.__levels_deep -= 1
self.write(f"</{tag_name}>", deep_space=deep_space, lineend='\n' if not nolineend else '') self.write(f"</{tag_name}>", deep_space=deep_space, lineend='\n' if not nolineend else '')
else: else:
self.write(f"<{tag_name} {attrs} />", lineend=lineend, deep_space=deep_space) self.write(f"<{tag_name} {attrs_str} />", lineend=lineend, deep_space=deep_space)
yield yield
def singleton(self, tag_name: str, attrs: dict = None, css: dict = None): def singleton(self, tag_name: str, attrs: dict = None, css: dict = None, singleton: bool = True):
with self.elm(tag_name, attrs, css, singleton=True): with self.elm(tag_name, attrs, css, singleton=singleton):
pass pass
+6 -4
View File
@@ -1,3 +1,4 @@
from typing import Set
from json import dumps as json_dump from json import dumps as json_dump
from io import TextIOWrapper from io import TextIOWrapper
from contextlib import contextmanager from contextlib import contextmanager
@@ -8,14 +9,13 @@ def JSONMaker(file_path: str, codec: str = 'utf-8'):
with JSONMakerObject(outf, codec=codec) as obj: with JSONMakerObject(outf, codec=codec) as obj:
yield obj yield obj
class JSONMakerObject: class JSONMakerObject:
class AccessError(Exception): pass class AccessError(Exception): pass
def __init__(self, outfile: TextIOWrapper, codec: str = 'utf-8', levels_deep: int = 0): def __init__(self, outfile: TextIOWrapper, codec: str = 'utf-8', levels_deep: int = 0):
self.outfile: TextIOWrapper = outfile self.outfile: TextIOWrapper = outfile
self.codec = codec self.codec = codec
self.levels_deep = levels_deep self.levels_deep = levels_deep
self.keys = set() self.keys: Set[str] = set()
def __enter__(self): def __enter__(self):
self.write_raw('{') self.write_raw('{')
@@ -50,7 +50,8 @@ class JSONMakerObject:
self.write_prop(key, val) self.write_prop(key, val)
def string(self, key: str, val: str): def string(self, key: str, val: str):
self.write_prop(key, f'"{val}"') val_sant = str(val).replace('"', '\\"')
self.write_prop(key, f'"{val_sant}"')
@contextmanager @contextmanager
def object(self, key: str): def object(self, key: str):
@@ -109,7 +110,8 @@ class JSONMakerArray:
self.write_prop(val) self.write_prop(val)
def string(self, val: str): def string(self, val: str):
self.write_prop(f'"{val}"') val_sant = str(val).replace('"', '\\"')
self.write_prop(f'"{val_sant}"')
@contextmanager @contextmanager
def object(self): def object(self):
+117
View File
@@ -0,0 +1,117 @@
from json import dumps as json_dump
from io import TextIOWrapper
from contextlib import contextmanager
@contextmanager
def YAMLMaker(file_path: str, codec: str = 'utf-8'):
with open(file_path, 'wb') as outf:
with YAMLMakerObject(outf, codec=codec) as obj:
yield obj
class YAMLMakerWriter:
class AccessError(Exception): pass
def __init__(self, outfile: TextIOWrapper, codec: str = 'utf-8', levels_deep: int = -1):
self.outfile: TextIOWrapper = outfile
self.codec = codec
self.levels_deep = levels_deep
def __enter__(self):
if self.levels_deep > 0:
self.__write_raw('')
self.levels_deep += 1
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.levels_deep -= 1
def __write_raw(self, text):
if self.outfile is not None:
if self.codec:
text = text.encode(self.codec)
self.outfile.write(text)
else:
raise self.AccessError("Attempting to write to file without an open connection")
def _write(self, text: str, append_deep: bool = True, deep_space: str = ' ', prefix:str = ''):
if not append_deep:
deep_space = ""
text = prefix + (deep_space*self.levels_deep) + text
self.__write_raw(text)
class YAMLMakerObject(YAMLMakerWriter):
def __init__(self, outfile: TextIOWrapper, codec: str = 'utf-8', levels_deep: int = -1, parent_is_array: bool = False):
YAMLMakerWriter.__init__(self, outfile, codec, levels_deep)
self.keys = set()
self.parent_is_array = parent_is_array
def write(self, key: str, value: str):
if key not in self.keys:
self.keys.add(key)
if len(self.keys) > 1:
self._write(f'{key}: {value}', prefix='\n')
else:
self._write(f'{key}: {value}', prefix='',
append_deep=not self.parent_is_array
)
@contextmanager
def object(self, key: str):
if key not in self.keys:
self.keys.add(key)
if len(self.keys) > 1:
self._write(f'{key}:\n', prefix='\n')
else:
self._write(f'{key}:\n', prefix='', append_deep=False)
with YAMLMakerObject(self.outfile, codec=self.codec, levels_deep=self.levels_deep) as obj:
yield obj
@contextmanager
def array(self, key: str):
if key not in self.keys:
self.keys.add(key)
if len(self.keys) > 1:
self._write(f'{key}:\n', prefix='\n')
else:
self._write(f'{key}:\n', prefix='')
with YAMLMakerArray(self.outfile, codec=self.codec, levels_deep=self.levels_deep) as obj:
yield obj
def place_object(self, key: str, obj: dict):
with self.object(key) as child:
for key, val in obj.items():
child.write(key, val)
def place_array(self, key: str, arr: list):
with self.array(key) as child:
for val in arr:
child.write(val)
class YAMLMakerArray(YAMLMakerWriter):
def __init__(self, outfile: TextIOWrapper, codec: str = 'utf-8', levels_deep: int = -1):
YAMLMakerWriter.__init__(self, outfile, codec, levels_deep)
self.childrencount = 0
def write(self, value: str = ""):
self.childrencount += 1
if self.childrencount > 1:
self._write(f'- {value}', prefix='\n')
else:
self._write(f'- {value}', prefix='')
@contextmanager
def object(self):
self.childrencount += 1
if self.childrencount > 1:
self._write(f'- ', prefix='\n')
else:
self._write(f'- ', prefix='')
with YAMLMakerObject(self.outfile, codec=self.codec, levels_deep=self.levels_deep, parent_is_array=True) as obj:
yield obj
def place_object(self, obj: dict):
with self.object() as child:
for key, val in obj.items():
child.write(key, val)
+144
View File
@@ -0,0 +1,144 @@
from typing import List
from .writers.yaml import YAMLMaker, YAMLMakerObject, YAMLMakerArray
from ..interpreter import PDFInterpreter, PageInterpreter, logging
from ..interpreter import LTImage, LTTextBlock, LTCharBlock, LTCurve, LTXObject
from ..interpreter.commands import LTItem
from ..interpreter.commands import LTRect, LTLine
from ..interpreter.commands.state import CurvePath, PDFColor
from ..parser.PSStackParser import PSLiteral
from ..utils import enc
def normalize_color_values(values: list):
ret = []
for v in values:
if isinstance(v, PSLiteral):
ret.append( enc(str(v.name)).decode() )
else:
ret.append( enc(str(v)).decode() )
return ret
def convert_to_yaml(
input_file_path: str,
output_file_path: str,
image_folder_path: str = None,
dont_export_images: bool = False,
codec: str = 'utf-8',
maxpages: int = 0,
password: str = None,
caching: bool = True,
check_extractable: bool = True,
pagenos: List[int] = None,
debug_level: int = logging.WARNING,
):
intepreter = PDFInterpreter(input_file_path,
maxpages=maxpages,
password=password,
caching=caching,
check_extractable=check_extractable,
pagenos=pagenos,
debug_level=debug_level
)
with YAMLMaker(output_file_path, codec=codec) as yaml:
with yaml.array("pages") as arr:
for page in intepreter:
with arr.object() as page_obj:
render_page(page_obj, page)
def render_page(yaml: YAMLMakerObject, ltpage: PageInterpreter):
yaml.write("id", ltpage.page_num)
yaml.write("width", ltpage.width)
yaml.write("height", ltpage.height)
with yaml.array("children") as chil_arr:
for child in ltpage:
with chil_arr.object() as chil_obj:
render(chil_obj, child)
def render(yaml: YAMLMakerObject, item: LTItem):
if isinstance(item, LTCurve):
place_curve(yaml, item)
elif isinstance(item, LTXObject):
yaml.write("type", 'xobject')
yaml.write("x0", item.x0)
yaml.write("x1", item.x1)
yaml.write("y0", item.y0)
yaml.write("y1", item.y1)
with yaml.array("children") as chil_arr:
for child in item:
with chil_arr.object() as chil_obj:
render(chil_obj, child)
elif isinstance(item, LTImage):
# if yaml.imagewriter is not None:
# name = yaml.imagewriter.export_image(item)
yaml.write("type", 'img')
yaml.write("src", enc(item.name, None))
yaml.write("x0", item.x0)
yaml.write("x1", item.x1)
yaml.write("y0", item.y0)
yaml.write("y1", item.y1)
elif isinstance(item, LTTextBlock):
place_text_block(yaml, item)
def place_text_block(yaml: YAMLMakerObject, text_block: LTTextBlock):
yaml.write("type", 'text-block')
yaml.write("x0", text_block.x0)
yaml.write("x1", text_block.x1)
yaml.write("y0", text_block.y0)
yaml.write("y1", text_block.y1)
with yaml.array("char_blocks") as cb_arr:
for cb in text_block:
with cb_arr.object() as cb_obj:
place_char_block(cb_obj, cb)
def place_char_block(yaml: YAMLMakerObject, char_block: LTCharBlock):
yaml.write("size", char_block.size)
yaml.write("x0", char_block.x0)
yaml.write("x1", char_block.x1)
yaml.write("y0", char_block.y0)
yaml.write("y1", char_block.y1)
if char_block.color.color_space is not None:
yaml.place_object("color", {
"type": enc(char_block.color.color_space.name).decode(),
"value": normalize_color_values(char_block.color.values)
})
for key, value in char_block.font.descriptor.items():
if key != "Type" and "FontFile" not in key:
yaml.write(key, value)
yaml.write("text",
"".join([
char.get_text()
for char in char_block
])
)
def place_curve(yaml: YAMLMakerObject, item: LTCurve):
if isinstance(item, LTRect):
yaml.write("type", 'rect')
elif isinstance(item, LTLine):
yaml.write("type", 'line')
else:
yaml.write("type", 'curve')
with yaml.array('paths') as paths_arr:
for path in item.paths:
paths_arr.place_object({
"type": path.method.name,
"points": [
{"x": point.x, "y": point.y}
for point in path.points
]
})
yaml.write("x0", item.x0)
yaml.write("x1", item.x1)
yaml.write("y0", item.y0)
yaml.write("y1", item.y1)
if item.fill.color_space is not None:
yaml.place_object("fill", {
"type": enc(item.fill.color_space.name).decode(),
"value": normalize_color_values(item.fill.values)
})
if item.stroke.color_space is not None:
yaml.place_object("stroke", {
"type": enc(item.stroke.color_space.name).decode(),
"value": normalize_color_values(item.stroke.values)
})
-127
View File
@@ -1,127 +0,0 @@
import struct
import os
import os.path
from io import BytesIO
from ..interpreter.PDFStream import LITERALS_DCT_DECODE
from ..interpreter.PDFColorSpace import LITERAL_DEVICE_GRAY
from ..interpreter.PDFColorSpace import LITERAL_DEVICE_RGB
from ..interpreter.PDFColorSpace import LITERAL_DEVICE_CMYK
from ..layouts import LTImage
def align32(x):
return ((x+3)//4)*4
## BMPWriter
##
class BMPWriter(object):
def __init__(self, fp, bits, width, height):
self.fp = fp
self.bits = bits
self.width = width
self.height = height
if bits == 1:
ncols = 2
elif bits == 8:
ncols = 256
elif bits == 24:
ncols = 0
else:
raise ValueError(bits)
self.linesize = align32((self.width*self.bits+7)//8)
self.datasize = self.linesize * self.height
headersize = 14+40+ncols*4
info = struct.pack('<IiiHHIIIIII', 40, self.width, self.height, 1, self.bits, 0, self.datasize, 0, 0, ncols, 0)
assert len(info) == 40, str(len(info))
header = struct.pack('<ccIHHI', b'B', b'M', headersize+self.datasize, 0, 0, headersize)
assert len(header) == 14, str(len(header))
self.fp.write(header)
self.fp.write(info)
if ncols == 2:
# B&W color table
for i in (0, 255):
self.fp.write(struct.pack('BBBx', i, i, i))
elif ncols == 256:
# grayscale color table
for i in range(256):
self.fp.write(struct.pack('BBBx', i, i, i))
self.pos0 = self.fp.tell()
self.pos1 = self.pos0 + self.datasize
return
def write_line(self, y, data):
self.fp.seek(self.pos1 - (y+1)*self.linesize)
self.fp.write(data)
return
## ImageWriter
##
class ImageWriter(object):
def __init__(self, outdir: str = None):
if outdir is None:
self.inactive = True
else:
self.inactive = False
self.outdir = outdir
if not os.path.exists(self.outdir):
os.makedirs(self.outdir)
def export_image(self, image: LTImage) -> str:
if self.inactive:
return "N/A"
stream = image.stream
filters = stream.get_filters()
(width, height) = image.srcsize
if len(filters) == 1 and filters[0][0] in LITERALS_DCT_DECODE:
ext = '.jpg'
elif (image.bits == 1 or
image.bits == 8 and image.colorspace in (LITERAL_DEVICE_RGB, LITERAL_DEVICE_GRAY)):
ext = '.%dx%d.bmp' % (width, height)
else:
ext = '.%d.%dx%d.img' % (image.bits, width, height)
name = image.name+ext
path = os.path.join(self.outdir, name)
fp=open(path, 'wb')
if ext == '.jpg':
raw_data = stream.get_rawdata()
if LITERAL_DEVICE_CMYK in image.colorspace:
from PIL import Image
from PIL import ImageChops
ifp = BytesIO(raw_data)
i = Image.open(ifp)
i = ImageChops.invert(i)
i = i.convert('RGB')
i.save(fp, 'JPEG')
else:
fp.write(raw_data)
elif image.bits == 1:
bmp = BMPWriter(fp, 1, width, height)
data = stream.get_data()
i = 0
width = (width+7)//8
for y in range(height):
bmp.write_line(y, data[i:i+width])
i += width
elif image.bits == 8 and image.colorspace is LITERAL_DEVICE_RGB:
bmp = BMPWriter(fp, 24, width, height)
data = stream.get_data()
i = 0
width = width*3
for y in range(height):
bmp.write_line(y, data[i:i+width])
i += width
elif image.bits == 8 and image.colorspace is LITERAL_DEVICE_GRAY:
bmp = BMPWriter(fp, 8, width, height)
data = stream.get_data()
i = 0
for y in range(height):
bmp.write_line(y, data[i:i+width])
i += width
else:
fp.write(stream.get_data())
fp.close()
return name
+2 -4
View File
@@ -1,7 +1,5 @@
from typing import List from typing import List
from tqdm import tqdm
from ..parser.PDFPage import PDFPage from ..parser.PDFPage import PDFPage
from ..utils import set_log_level, get_logger, logging from ..utils import set_log_level, get_logger, logging
from .commands.state import PDFStateStack, PDFColorSpace, PREDEFINED_COLORSPACE from .commands.state import PDFStateStack, PDFColorSpace, PREDEFINED_COLORSPACE
@@ -53,7 +51,7 @@ class PDFInterpreter:
caching=self.caching, caching=self.caching,
check_extractable=self.check_extractable check_extractable=self.check_extractable
) )
for page_num, page in tqdm(enumerate(pages), desc="Reading pages...", disable=self.debug_level > logging.INFO): for page_num, page in enumerate(pages):
self.__pages.append(PageInterpreter(page, page_num, font_cache)) self.__pages.append(PageInterpreter(page, page_num, font_cache))
yield self.__pages[-1] yield self.__pages[-1]
log.info(f"Done Reading {len(self.__pages)} pages.") log.info(f"Done Reading {len(self.__pages)} pages.")
@@ -64,7 +62,7 @@ class PDFInterpreter:
def __iter__(self): def __iter__(self):
if len(self.__pages) > 0: if len(self.__pages) > 0:
set_log_level(self.debug_level) set_log_level(self.debug_level)
for page in tqdm(self.__pages, desc="Reading pages from store...", disable=self.debug_level > logging.INFO): for page in self.__pages:
yield page yield page
set_log_level(logging.WARNING) set_log_level(logging.WARNING)
else: else:
+13 -4
View File
@@ -20,15 +20,19 @@ log = get_logger('process_command_stream')
class CommandProcessorError(Exception): pass class CommandProcessorError(Exception): pass
def process_command_stream(streams: List[PDFStream], font_cache: dict = None, state: PDFStateStack = None): def process_command_stream(streams: List[PDFStream], font_cache: dict = None, state: PDFStateStack = None):
if state is None:
state = PDFStateStack()
if font_cache is None: if font_cache is None:
font_cache = {} font_cache = {}
parser = PDFContentParser(streams) parser = PDFContentParser(streams)
# from .cmd_record import CmdRecord
# history = CmdRecord()
for obj in parser: for obj in parser:
if isinstance(obj, PSKeyword): if isinstance(obj, PSKeyword):
name = keyword_name(obj) name = keyword_name(obj)
method = name.replace('*', '_a').replace('"', '_w').replace("'", '_q') method = name.replace('*', '_a').replace('"', '_w').replace("'", '_q')
func = None
args = []
if method == "Do":
pass
if method in PDFCommands.commands.keys(): if method in PDFCommands.commands.keys():
func = PDFCommands.commands.get(method) func = PDFCommands.commands.get(method)
nargs = func.__code__.co_argcount-1 nargs = func.__code__.co_argcount-1
@@ -41,18 +45,23 @@ def process_command_stream(streams: List[PDFStream], font_cache: dict = None, st
else: else:
func(state) func(state)
else: else:
log.warning('Unknown operator: %r (i.e. %r)' % (name, method)) log.debug('Unknown operator: %r (i.e. %r)' % (name, method))
# history.write(name,
# None if func is None else func.__name__
# , args)
else: else:
state.argstack.append(obj) state.argstack.append(obj)
# history.write('arg', None, obj)
for complete_item in state.complete_layout_items: for complete_item in state.complete_layout_items:
if isinstance(complete_item, LTXObject): if isinstance(complete_item, LTXObject):
xobj_state = prep_state( xobj_state = prep_state(
state=PDFGraphicState(), state=PDFStateStack(),
ctm=complete_item.t_matrix, ctm=complete_item.t_matrix,
resources=complete_item.resources, resources=complete_item.resources,
font_cache=font_cache font_cache=font_cache
) )
xobj_state.graphics = state.graphics.copy()
for item in process_command_stream( for item in process_command_stream(
[complete_item.stream], [complete_item.stream],
font_cache=font_cache, font_cache=font_cache,
@@ -0,0 +1,9 @@
class CmdRecord:
def __init__(self, file_name: str = "./cmd_record"):
self.file = open(file_name, 'w')
self.file.write("c_name,func_name,args\n")
def write(self, c_name, func_name, args):
self.file.write(f"{c_name},{func_name},{args}\n")
def close(self):
self.file.close()
+96 -86
View File
@@ -11,35 +11,45 @@ from .state import make_char_block, make_curve, make_image, make_xobject
log = get_logger('commands') log = get_logger('commands')
class CommandError(Exception): pass
# No Support # No Support
# @PDFCommands.add('W','W_a', 'sh', 'ET', 'BX', 'EX' ,'BI', 'ID', 'gs') # @PDFCommands.add('W','W_a', 'sh', 'ET', 'BX', 'EX' ,'BI', 'ID', 'gs')
# def do_no_support(stack: PDFStateStack) -> PDFStateStack: # def do_no_support(stack: PDFStateStack) -> PDFStateStack:
# # log.warning("Unsupported command") # # log.debug("Unsupported command")
# return stack # return stack
# marked content operators # marked content operators
@PDFCommands.add('MP', 'DP') # @PDFCommands.add('MP')
def do_MP(stack: PDFStateStack) -> PDFStateStack: # def do_MP(stack: PDFStateStack, tag) -> PDFStateStack:
log.warning("Unsupported command - do_tag") # log.debug(f"Unsupported command - do_tag {[tag]}")
# self.device.do_tag(tag) # # self.device.do_tag(tag)
return stack # return stack
@PDFCommands.add('BMC', 'BDC') # @PDFCommands.add('DP')
def do_BMC(stack: PDFStateStack) -> PDFStateStack: # def do_DP(stack: PDFStateStack, tag, props=None) -> PDFStateStack:
log.warning("Unsupported command - begin_tag") # log.debug(f"Unsupported command - do_tag {[tag.name, type(tag), props]}")
# self.device.begin_tag(tag) # # self.device.do_tag(tag)
return stack # return stack
@PDFCommands.add('EMC') # @PDFCommands.add('BMC')
def do_EMC(stack: PDFStateStack) -> PDFStateStack: # def do_BMC(stack: PDFStateStack, tag) -> PDFStateStack:
# self.device.end_tag(tag) # log.debug(f"Unsupported command - begin_tag {[tag.name, type(tag)]}")
log.warning("Unsupported command - end_tag") # # self.device.begin_tag(tag)
return stack # return stack
# @PDFCommands.add('BDC')
# def do_BDC(stack: PDFStateStack, tag, props=None) -> PDFStateStack:
# log.debug(f"Unsupported command - begin_tag {[tag.name, type(tag), props]}")
# # self.device.begin_tag(tag)
# return stack
# @PDFCommands.add('EMC')
# def do_EMC(stack: PDFStateStack) -> PDFStateStack:
# # self.device.end_tag(tag)
# log.debug("Unsupported command - end_tag")
# return stack
@PDFCommands.add('q') @PDFCommands.add('q')
def do_q(stack: PDFStateStack) -> PDFStateStack: def save_state(stack: PDFStateStack) -> PDFStateStack:
stack.gstack.append([ stack.gstack.append([
stack.t_matrix, stack.t_matrix,
stack.text.copy(), stack.text.copy(),
@@ -48,136 +58,136 @@ def do_q(stack: PDFStateStack) -> PDFStateStack:
return stack return stack
@PDFCommands.add('Q') @PDFCommands.add('Q')
def do_Q(stack: PDFStateStack) -> PDFStateStack: def restore_state(stack: PDFStateStack) -> PDFStateStack:
if len(stack.gstack) > 0: if len(stack.gstack) > 0:
stack.t_matrix, stack.text, stack.graphicstate = stack.gstack.pop() stack.t_matrix, stack.text, stack.graphicstate = stack.gstack.pop()
return stack return stack
# concat-matrix # concat-matrix
@PDFCommands.add('cm') @PDFCommands.add('cm')
def do_cm(stack: PDFStateStack, a1, b1, c1, d1, e1, f1) -> PDFStateStack: def concat_transition_matrix(stack: PDFStateStack, a1, b1, c1, d1, e1, f1) -> PDFStateStack:
stack.t_matrix = mult_matrix((a1, b1, c1, d1, e1, f1), stack.t_matrix) stack.t_matrix = mult_matrix((a1, b1, c1, d1, e1, f1), stack.t_matrix)
return stack return stack
# setlinewidth # setlinewidth
@PDFCommands.add('w') @PDFCommands.add('w')
def do_w(stack: PDFStateStack, linewidth) -> PDFStateStack: def set_line_width(stack: PDFStateStack, linewidth) -> PDFStateStack:
stack.graphics.linewidth = linewidth stack.graphics.linewidth = linewidth
return stack return stack
# setlinecap # setlinecap
@PDFCommands.add('J') @PDFCommands.add('J')
def do_J(stack: PDFStateStack, linecap) -> PDFStateStack: def set_linecap(stack: PDFStateStack, linecap) -> PDFStateStack:
stack.graphics.linecap = linecap stack.graphics.linecap = linecap
return stack return stack
# setlinejoin # setlinejoin
@PDFCommands.add('j') @PDFCommands.add('j')
def do_j(stack: PDFStateStack, linejoin) -> PDFStateStack: def set_linejoin(stack: PDFStateStack, linejoin) -> PDFStateStack:
stack.graphics.linejoin = linejoin stack.graphics.linejoin = linejoin
return stack return stack
# setmiterlimit # setmiterlimit
@PDFCommands.add('M') @PDFCommands.add('M')
def do_M(stack: PDFStateStack, miterlimit) -> PDFStateStack: def set_miterlimit(stack: PDFStateStack, miterlimit) -> PDFStateStack:
stack.graphics.miterlimit = miterlimit stack.graphics.miterlimit = miterlimit
return stack return stack
# setdash # setdash
@PDFCommands.add('d') @PDFCommands.add('d')
def do_d(stack: PDFStateStack, dash, phase) -> PDFStateStack: def set_dash(stack: PDFStateStack, dash, phase) -> PDFStateStack:
stack.graphics.dash = (dash, phase) stack.graphics.dash = (dash, phase)
return stack return stack
# setintent # setintent
@PDFCommands.add('ri') @PDFCommands.add('ri')
def do_ri(stack: PDFStateStack, intent) -> PDFStateStack: def set_intent(stack: PDFStateStack, intent) -> PDFStateStack:
stack.graphics.intent = intent stack.graphics.intent = intent
return stack return stack
# setflatness # setflatness
@PDFCommands.add('i') @PDFCommands.add('i')
def do_i(stack: PDFStateStack, flatness) -> PDFStateStack: def set_flatness(stack: PDFStateStack, flatness) -> PDFStateStack:
stack.graphics.flatness = flatness stack.graphics.flatness = flatness
return stack return stack
# moveto # moveto
@PDFCommands.add('m') @PDFCommands.add('m')
def do_m(stack: PDFStateStack, x: float, y: float) -> PDFStateStack: def curve_moveto(stack: PDFStateStack, x: float, y: float) -> PDFStateStack:
stack.curvestacks.append(CurvePath( stack.curvestacks.append(CurvePath(
CurvePath.METHOD.MOVE_TO, CurvePath.METHOD.MOVE_TO,
CurvePoint(x, y) CurvePoint(stack.t_matrix, x, y)
)) ))
return stack return stack
# lineto # lineto
@PDFCommands.add('l') @PDFCommands.add('l')
def do_l(stack: PDFStateStack, x: float, y: float) -> PDFStateStack: def curve_lineto(stack: PDFStateStack, x: float, y: float) -> PDFStateStack:
stack.curvestacks.append(CurvePath( stack.curvestacks.append(CurvePath(
CurvePath.METHOD.LINE_TO, CurvePath.METHOD.LINE_TO,
CurvePoint(x, y) CurvePoint(stack.t_matrix, x, y)
)) ))
return stack return stack
# curveto # curveto
@PDFCommands.add('c') @PDFCommands.add('c')
def do_c(stack: PDFStateStack, def curve_curveto(stack: PDFStateStack,
x1: float, y1: float, x1: float, y1: float,
x2: float, y2: float, x2: float, y2: float,
x3: float, y3: float ) -> PDFStateStack: x3: float, y3: float ) -> PDFStateStack:
stack.curvestacks.append(CurvePath( stack.curvestacks.append(CurvePath(
CurvePath.METHOD.CURVE_BOTH_TO, CurvePath.METHOD.CURVE_BOTH_TO,
CurvePoint(x1, y1), CurvePoint(stack.t_matrix, x1, y1),
CurvePoint(x2, y2), CurvePoint(stack.t_matrix, x2, y2),
CurvePoint(x3, y3), CurvePoint(stack.t_matrix, x3, y3),
)) ))
return stack return stack
# urveto # urveto
@PDFCommands.add('v') @PDFCommands.add('v')
def do_v(stack: PDFStateStack, def curve_urveto(stack: PDFStateStack,
x2: float, y2: float, x2: float, y2: float,
x3: float, y3: float) -> PDFStateStack: x3: float, y3: float) -> PDFStateStack:
stack.curvestacks.append(CurvePath( stack.curvestacks.append(CurvePath(
CurvePath.METHOD.CURVE_NEXT_TO, CurvePath.METHOD.CURVE_NEXT_TO,
CurvePoint(x2, y2), CurvePoint(stack.t_matrix, x2, y2),
CurvePoint(x3, y3), CurvePoint(stack.t_matrix, x3, y3),
)) ))
return stack return stack
# rveto # rveto
@PDFCommands.add('y') @PDFCommands.add('y')
def do_y(stack: PDFStateStack, def curve_rveto(stack: PDFStateStack,
x1: float, y1: float, x1: float, y1: float,
x3: float, y3: float) -> PDFStateStack: x3: float, y3: float) -> PDFStateStack:
stack.curvestacks.append(CurvePath( stack.curvestacks.append(CurvePath(
CurvePath.METHOD.CURVE_FIRST_TO, CurvePath.METHOD.CURVE_FIRST_TO,
CurvePoint(x1, y1), CurvePoint(stack.t_matrix, x1, y1),
CurvePoint(x3, y3) CurvePoint(stack.t_matrix, x3, y3)
)) ))
return stack return stack
# closepath # closepath
@PDFCommands.add('h') @PDFCommands.add('h')
def do_h(stack: PDFStateStack) -> PDFStateStack: def curve_close(stack: PDFStateStack) -> PDFStateStack:
stack.curvestacks.append(CurvePath(CurvePath.METHOD.CLOSE_PATH)) stack.curvestacks.append(CurvePath(CurvePath.METHOD.CLOSE_PATH))
return stack return stack
# rectangle # rectangle
@PDFCommands.add('re') @PDFCommands.add('re')
def do_re(stack: PDFStateStack, x, y, w, h) -> PDFStateStack: def curve_rect(stack: PDFStateStack, x, y, w, h) -> PDFStateStack:
stack.curvestacks.extend([ stack.curvestacks.extend([
CurvePath(CurvePath.METHOD.MOVE_TO, CurvePoint(x, y) ), CurvePath(CurvePath.METHOD.MOVE_TO, CurvePoint(stack.t_matrix, x, y) ),
CurvePath(CurvePath.METHOD.LINE_TO, CurvePoint(x+w, y) ), CurvePath(CurvePath.METHOD.LINE_TO, CurvePoint(stack.t_matrix, x+w, y) ),
CurvePath(CurvePath.METHOD.LINE_TO, CurvePoint(x+w, y+h) ), CurvePath(CurvePath.METHOD.LINE_TO, CurvePoint(stack.t_matrix, x+w, y+h) ),
CurvePath(CurvePath.METHOD.LINE_TO, CurvePoint(x, y+h) ), CurvePath(CurvePath.METHOD.LINE_TO, CurvePoint(stack.t_matrix, x, y+h) ),
CurvePath(CurvePath.METHOD.CLOSE_PATH) CurvePath(CurvePath.METHOD.CLOSE_PATH)
]) ])
return stack return stack
# stroke, fill # stroke, fill
@PDFCommands.add('S', 'f', 'F', 'B') @PDFCommands.add('S', 'f', 'F', 'B')
def do_complete_path(stack: PDFStateStack) -> PDFStateStack: def curve_complete_path(stack: PDFStateStack) -> PDFStateStack:
stack.complete_layout_items.append(make_curve( stack.complete_layout_items.append(make_curve(
stack.t_matrix, stack.t_matrix,
stack.graphics.copy(), stack.graphics.copy(),
@@ -189,7 +199,7 @@ def do_complete_path(stack: PDFStateStack) -> PDFStateStack:
# sroke, fill-even-odd # sroke, fill-even-odd
@PDFCommands.add('f_a', 'B_a') @PDFCommands.add('f_a', 'B_a')
def do_complete_path_evenodd(stack: PDFStateStack) -> PDFStateStack: def curve_complete_path_evenodd(stack: PDFStateStack) -> PDFStateStack:
stack.complete_layout_items.append(make_curve( stack.complete_layout_items.append(make_curve(
stack.t_matrix, stack.t_matrix,
stack.graphics.copy(), stack.graphics.copy(),
@@ -201,7 +211,7 @@ def do_complete_path_evenodd(stack: PDFStateStack) -> PDFStateStack:
# close-and-stroke (do_h, do_S) # close-and-stroke (do_h, do_S)
@PDFCommands.add('s', 'b') @PDFCommands.add('s', 'b')
def do_close_complete(stack: PDFStateStack) -> PDFStateStack: def curve_close_complete(stack: PDFStateStack) -> PDFStateStack:
stack.curvestacks.append(CurvePath( stack.curvestacks.append(CurvePath(
CurvePath.METHOD.CLOSE_PATH CurvePath.METHOD.CLOSE_PATH
)) ))
@@ -216,7 +226,7 @@ def do_close_complete(stack: PDFStateStack) -> PDFStateStack:
# close-fill-and-stroke-even-odd # close-fill-and-stroke-even-odd
@PDFCommands.add('b_a') @PDFCommands.add('b_a')
def do_close_complete_evenodd(stack: PDFStateStack) -> PDFStateStack: def curve_close_complete_evenodd(stack: PDFStateStack) -> PDFStateStack:
stack.curvestacks.append(CurvePath( stack.curvestacks.append(CurvePath(
CurvePath.METHOD.CLOSE_PATH CurvePath.METHOD.CLOSE_PATH
)) ))
@@ -231,14 +241,14 @@ def do_close_complete_evenodd(stack: PDFStateStack) -> PDFStateStack:
# close-only (empty path) # close-only (empty path)
@PDFCommands.add('n') @PDFCommands.add('n')
def do_n(stack: PDFStateStack) -> PDFStateStack: def curve_close_only(stack: PDFStateStack) -> PDFStateStack:
stack.curvestacks = [] stack.curvestacks = []
return stack return stack
# setcolorspace-stroking # setcolorspace-stroking
@PDFCommands.add('CS') @PDFCommands.add('CS')
def do_CS(stack: PDFStateStack, name: str) -> PDFStateStack: def set_stroke_colorspace(stack: PDFStateStack, name: str) -> PDFStateStack:
try: try:
stack.graphics.scolspace = stack.colorspace_map[literal_name(name)] stack.graphics.scolspace = stack.colorspace_map[literal_name(name)]
return stack return stack
@@ -247,7 +257,7 @@ def do_CS(stack: PDFStateStack, name: str) -> PDFStateStack:
# setcolorspace-non-strokine # setcolorspace-non-strokine
@PDFCommands.add('cs') @PDFCommands.add('cs')
def do_cs(stack: PDFStateStack, name: str) -> PDFStateStack: def set_nonstroke_colorspace(stack: PDFStateStack, name: str) -> PDFStateStack:
try: try:
stack.graphics.ncolspace = stack.colorspace_map[literal_name(name)] stack.graphics.ncolspace = stack.colorspace_map[literal_name(name)]
return stack return stack
@@ -256,7 +266,7 @@ def do_cs(stack: PDFStateStack, name: str) -> PDFStateStack:
# setgray-stroking # setgray-stroking
@PDFCommands.add('G') @PDFCommands.add('G')
def do_G(stack: PDFStateStack, gray: float) -> PDFStateStack: def set_stroke_gray(stack: PDFStateStack, gray: float) -> PDFStateStack:
stack.graphics.set_stroke_color( stack.graphics.set_stroke_color(
stack.colorspace_map['DeviceGray'], stack.colorspace_map['DeviceGray'],
gray gray
@@ -265,7 +275,7 @@ def do_G(stack: PDFStateStack, gray: float) -> PDFStateStack:
# setgray-non-stroking # setgray-non-stroking
@PDFCommands.add('g') @PDFCommands.add('g')
def do_g(stack: PDFStateStack, gray: float) -> PDFStateStack: def set_nonstroke_gray(stack: PDFStateStack, gray: float) -> PDFStateStack:
stack.graphics.set_nostroke_color( stack.graphics.set_nostroke_color(
stack.colorspace_map['DeviceGray'], stack.colorspace_map['DeviceGray'],
gray gray
@@ -275,7 +285,7 @@ def do_g(stack: PDFStateStack, gray: float) -> PDFStateStack:
# setrgb-stroking # setrgb-stroking
@PDFCommands.add('RG') @PDFCommands.add('RG')
def do_RG(stack: PDFStateStack, r, g, b) -> PDFStateStack: def set_stroke_rgb(stack: PDFStateStack, r, g, b) -> PDFStateStack:
stack.graphics.set_stroke_color( stack.graphics.set_stroke_color(
stack.colorspace_map['DeviceRGB'], stack.colorspace_map['DeviceRGB'],
round(r*255), round(r*255),
@@ -286,7 +296,7 @@ def do_RG(stack: PDFStateStack, r, g, b) -> PDFStateStack:
# setrgb-non-stroking # setrgb-non-stroking
@PDFCommands.add('rg') @PDFCommands.add('rg')
def do_rg(stack: PDFStateStack, r, g, b) -> PDFStateStack: def set_nonstroke_rgb(stack: PDFStateStack, r, g, b) -> PDFStateStack:
stack.graphics.set_nostroke_color( stack.graphics.set_nostroke_color(
stack.colorspace_map['DeviceRGB'], stack.colorspace_map['DeviceRGB'],
round(r*255), round(r*255),
@@ -297,7 +307,7 @@ def do_rg(stack: PDFStateStack, r, g, b) -> PDFStateStack:
# setcmyk-stroking # setcmyk-stroking
@PDFCommands.add('K') @PDFCommands.add('K')
def do_K(stack: PDFStateStack, c, m, y, k) -> PDFStateStack: def set_stroke_cmyk(stack: PDFStateStack, c, m, y, k) -> PDFStateStack:
stack.graphics.set_stroke_color( stack.graphics.set_stroke_color(
stack.colorspace_map['DeviceCMYK'], stack.colorspace_map['DeviceCMYK'],
c, m, y, k c, m, y, k
@@ -306,7 +316,7 @@ def do_K(stack: PDFStateStack, c, m, y, k) -> PDFStateStack:
# setcmyk-non-stroking # setcmyk-non-stroking
@PDFCommands.add('k') @PDFCommands.add('k')
def do_k(stack: PDFStateStack, c, m, y, k) -> PDFStateStack: def set_nonstroke_cmyk(stack: PDFStateStack, c, m, y, k) -> PDFStateStack:
stack.graphics.set_stroke_color( stack.graphics.set_stroke_color(
stack.colorspace_map['DeviceCMYK'], stack.colorspace_map['DeviceCMYK'],
c, m, y, k c, m, y, k
@@ -314,7 +324,7 @@ def do_k(stack: PDFStateStack, c, m, y, k) -> PDFStateStack:
return stack return stack
@PDFCommands.add('SCN','SC') @PDFCommands.add('SCN','SC')
def do_SCN(stack: PDFStateStack) -> PDFStateStack: def set_stroke_colorspace_custom(stack: PDFStateStack) -> PDFStateStack:
if stack.graphics.scolspace is None: if stack.graphics.scolspace is None:
raise PDFCommands.InvalidOperation('No colorspace specified!') raise PDFCommands.InvalidOperation('No colorspace specified!')
components = stack.pop(stack.graphics.scolspace.ncomponents) components = stack.pop(stack.graphics.scolspace.ncomponents)
@@ -325,7 +335,7 @@ def do_SCN(stack: PDFStateStack) -> PDFStateStack:
return stack return stack
@PDFCommands.add('scn','sc') @PDFCommands.add('scn','sc')
def do_scn(stack: PDFStateStack) -> PDFStateStack: def set_nonstroke_colorspace_custom(stack: PDFStateStack) -> PDFStateStack:
if stack.graphics.ncolspace is None: if stack.graphics.ncolspace is None:
raise PDFCommands.InvalidOperation('No colorspace specified!') raise PDFCommands.InvalidOperation('No colorspace specified!')
components = stack.pop(stack.graphics.ncolspace.ncomponents) components = stack.pop(stack.graphics.ncolspace.ncomponents)
@@ -337,7 +347,7 @@ def do_scn(stack: PDFStateStack) -> PDFStateStack:
# begin-text # begin-text
@PDFCommands.add('BT') @PDFCommands.add('BT')
def do_BT(stack: PDFStateStack) -> PDFStateStack: def begin_text(stack: PDFStateStack) -> PDFStateStack:
stack.text.matrix = MATRIX_IDENTITY stack.text.matrix = MATRIX_IDENTITY
stack.text.linematrix = [0, 0] stack.text.linematrix = [0, 0]
if stack.current_textblock is None: if stack.current_textblock is None:
@@ -347,38 +357,38 @@ def do_BT(stack: PDFStateStack) -> PDFStateStack:
return stack return stack
@PDFCommands.add('ET') @PDFCommands.add('ET')
def do_ET(stack: PDFStateStack) -> PDFStateStack: def end_text(stack: PDFStateStack) -> PDFStateStack:
stack.complete_layout_items.append(stack.current_textblock) stack.complete_layout_items.append(stack.current_textblock)
stack.current_textblock = None stack.current_textblock = None
return stack return stack
# setcharspace # setcharspace
@PDFCommands.add('Tc') @PDFCommands.add('Tc')
def do_Tc(stack: PDFStateStack, charspace) -> PDFStateStack: def set_charspace(stack: PDFStateStack, charspace) -> PDFStateStack:
stack.text.charspace = charspace stack.text.charspace = charspace
return stack return stack
# setwordspace # setwordspace
@PDFCommands.add('Tw') @PDFCommands.add('Tw')
def do_Tw(stack: PDFStateStack, wordspace) -> PDFStateStack: def set_wordspace(stack: PDFStateStack, wordspace) -> PDFStateStack:
stack.text.wordspace = wordspace stack.text.wordspace = wordspace
return stack return stack
# textscale # textscale
@PDFCommands.add('Tz') @PDFCommands.add('Tz')
def do_Tz(stack: PDFStateStack, scaling) -> PDFStateStack: def set_textscale(stack: PDFStateStack, scaling) -> PDFStateStack:
stack.text.scaling = scaling stack.text.scaling = scaling
return stack return stack
# setleading # setleading
@PDFCommands.add('TL') @PDFCommands.add('TL')
def do_TL(stack: PDFStateStack, leading) -> PDFStateStack: def set_text_leading(stack: PDFStateStack, leading) -> PDFStateStack:
stack.text.leading = -leading stack.text.leading = -leading
return stack return stack
# selectfont # selectfont
@PDFCommands.add('Tf') @PDFCommands.add('Tf')
def do_Tf(stack: PDFStateStack, fontid, fontsize) -> PDFStateStack: def set_font(stack: PDFStateStack, fontid, fontsize) -> PDFStateStack:
stack.text.fontsize = fontsize stack.text.fontsize = fontsize
try: try:
stack.text.font = stack.fontmap[literal_name(fontid)] stack.text.font = stack.fontmap[literal_name(fontid)]
@@ -388,19 +398,19 @@ def do_Tf(stack: PDFStateStack, fontid, fontsize) -> PDFStateStack:
# setrendering # setrendering
@PDFCommands.add('Tr') @PDFCommands.add('Tr')
def do_Tr(stack: PDFStateStack, render) -> PDFStateStack: def set_text_rendering(stack: PDFStateStack, render) -> PDFStateStack:
stack.text.render = render stack.text.render = render
return stack return stack
# settextrise # settextrise
@PDFCommands.add('Ts') @PDFCommands.add('Ts')
def do_Ts(stack: PDFStateStack, rise) -> PDFStateStack: def set_textrise(stack: PDFStateStack, rise) -> PDFStateStack:
stack.text.rise = rise stack.text.rise = rise
return stack return stack
# text-move # text-move
@PDFCommands.add('Td') @PDFCommands.add('Td')
def do_Td(stack: PDFStateStack, tx, ty) -> PDFStateStack: def move_text_x(stack: PDFStateStack, tx, ty) -> PDFStateStack:
(a, b, c, d, e, f) = stack.text.matrix (a, b, c, d, e, f) = stack.text.matrix
stack.text.matrix = (a, b, c, d, tx*a+ty*c+e, tx*b+ty*d+f) stack.text.matrix = (a, b, c, d, tx*a+ty*c+e, tx*b+ty*d+f)
stack.text.linematrix = [0, 0] stack.text.linematrix = [0, 0]
@@ -408,7 +418,7 @@ def do_Td(stack: PDFStateStack, tx, ty) -> PDFStateStack:
# text-move # text-move
@PDFCommands.add('TD') @PDFCommands.add('TD')
def do_TD(stack: PDFStateStack, tx, ty) -> PDFStateStack: def move_text_y(stack: PDFStateStack, tx, ty) -> PDFStateStack:
(a, b, c, d, e, f) = stack.text.matrix (a, b, c, d, e, f) = stack.text.matrix
stack.text.matrix = (a, b, c, d, tx*a+ty*c+e, tx*b+ty*d+f) stack.text.matrix = (a, b, c, d, tx*a+ty*c+e, tx*b+ty*d+f)
stack.text.leading = ty stack.text.leading = ty
@@ -417,14 +427,14 @@ def do_TD(stack: PDFStateStack, tx, ty) -> PDFStateStack:
# textmatrix # textmatrix
@PDFCommands.add('Tm') @PDFCommands.add('Tm')
def do_Tm(stack: PDFStateStack, a, b, c, d, e, f) -> PDFStateStack: def set_text_matrix(stack: PDFStateStack, a, b, c, d, e, f) -> PDFStateStack:
stack.text.matrix = (a, b, c, d, e, f) stack.text.matrix = (a, b, c, d, e, f)
stack.text.linematrix = [0, 0] stack.text.linematrix = [0, 0]
return stack return stack
# nextline # nextline
@PDFCommands.add('T_a') @PDFCommands.add('T_a')
def do_T_a(stack: PDFStateStack) -> PDFStateStack: def nextline(stack: PDFStateStack) -> PDFStateStack:
(a, b, c, d, e, f) = stack.text.matrix (a, b, c, d, e, f) = stack.text.matrix
stack.text.matrix = (a, b, c, d, stack.text.leading*c+e, stack.text.leading*d+f) stack.text.matrix = (a, b, c, d, stack.text.leading*c+e, stack.text.leading*d+f)
stack.text.linematrix = [0, 0] stack.text.linematrix = [0, 0]
@@ -432,7 +442,7 @@ def do_T_a(stack: PDFStateStack) -> PDFStateStack:
# show-pos # show-pos
@PDFCommands.add('TJ') @PDFCommands.add('TJ')
def do_TJ(stack: PDFStateStack, seq: bytearray) -> PDFStateStack: def add_char_blocks(stack: PDFStateStack, seq: bytearray) -> PDFStateStack:
if stack.text.font is None: if stack.text.font is None:
raise PDFCommands.InvalidOperation("No Font Specified") raise PDFCommands.InvalidOperation("No Font Specified")
if stack.current_textblock is None: if stack.current_textblock is None:
@@ -447,32 +457,32 @@ def do_TJ(stack: PDFStateStack, seq: bytearray) -> PDFStateStack:
# show # show
@PDFCommands.add('Tj') @PDFCommands.add('Tj')
def do_Tj(stack: PDFStateStack, b: bytes) -> PDFStateStack: def add_char_block(stack: PDFStateStack, b: bytes) -> PDFStateStack:
return do_TJ(stack, [b]) return add_char_blocks(stack, [b])
# quote # quote
@PDFCommands.add('_q') @PDFCommands.add('_q')
def do__q(stack: PDFStateStack, b: bytes) -> PDFStateStack: def add_quoted_char_blocks(stack: PDFStateStack, b: bytes) -> PDFStateStack:
return do_TJ(do_T_a(stack), [b]) return add_char_blocks(nextline(stack), [b])
# doublequote # doublequote
@PDFCommands.add('_w') @PDFCommands.add('_w')
def do__w(stack: PDFStateStack, wordspace, charspace, b) -> PDFStateStack: def add_dbquoted_char_blocks(stack: PDFStateStack, wordspace, charspace, b) -> PDFStateStack:
stack.text.wordspace = wordspace stack.text.wordspace = wordspace
stack.text.charspace = charspace stack.text.charspace = charspace
return do_TJ(stack, [b]) return add_char_blocks(stack, [b])
@PDFCommands.add('EI') @PDFCommands.add('EI')
def do_EI(stack: PDFStateStack, obj) -> PDFStateStack: def insert_image(stack: PDFStateStack, obj) -> PDFStateStack:
if 'W' in obj and 'H' in obj: if 'W' in obj and 'H' in obj:
stack.complete_layout_items.append(make_image( stack.complete_layout_items.append(make_image(
obj, stack.ctm obj, stack.t_matrix
)) ))
return stack return stack
# invoke an XObject # invoke an XObject
@PDFCommands.add('Do') @PDFCommands.add('Do')
def do_Do(stack: PDFStateStack, xobjid) -> PDFStateStack: def insert_xobject(stack: PDFStateStack, xobjid) -> PDFStateStack:
xobjid = literal_name(xobjid) xobjid = literal_name(xobjid)
try: try:
xobj = PDFStream.validated_stream(stack.xobjmap[xobjid]) xobj = PDFStream.validated_stream(stack.xobjmap[xobjid])
@@ -1,8 +1,14 @@
from enum import Enum from enum import Enum
from typing import List from typing import List
from pdfmajor.utils import apply_matrix_pt
class CurvePoint: class CurvePoint:
def __init__(self, x: float, y: float): def __init__(self, ctm: tuple, x: float, y: float):
# old = (x,y)
# (x, y) = apply_matrix_pt(ctm, (x, y))
# if old != (x,y):
# print("CurvePoint", ctm, old)
# print(" >", x, y)
self.x = x self.x = x
self.y = y self.y = y
@@ -1,10 +1,10 @@
from typing import List from typing import Tuple, Optional
from .PDFColorSpace import PDFColorSpace from .PDFColorSpace import PDFColorSpace
class PDFColor: class PDFColor:
def __init__(self, color_space: PDFColorSpace, *values: List[float]): def __init__(self, color_space: Optional[PDFColorSpace], *values: float):
self.color_space: PDFColorSpace = color_space self.color_space: Optional[PDFColorSpace] = color_space
self.values: list = values self.values: Tuple[float, ...] = values
def copy(self): def copy(self):
return self.__class__( return self.__class__(
@@ -13,4 +13,6 @@ class PDFColor:
) )
def __repr__(self): def __repr__(self):
return f"<PDFColor space={self.color_space} values={self.values}/>" return f"""<PDFColor space="{
None if self.color_space is None else self.color_space.name
}" values={self.values}/>"""
@@ -1,3 +1,4 @@
from typing import Dict
from collections import OrderedDict from collections import OrderedDict
class PDFColorSpace(object): class PDFColorSpace(object):
@@ -5,13 +6,12 @@ class PDFColorSpace(object):
def __init__(self, name: str, ncomponents: int): def __init__(self, name: str, ncomponents: int):
self.name: str = name self.name: str = name
self.ncomponents: int = ncomponents self.ncomponents: int = ncomponents
return
def __repr__(self): def __repr__(self):
return '<PDFColorSpace: %s, ncomponents=%d>' % (self.name, self.ncomponents) return '<PDFColorSpace: %s, ncomponents=%d>' % (self.name, self.ncomponents)
PREDEFINED_COLORSPACE = OrderedDict() PREDEFINED_COLORSPACE: Dict[str, PDFColorSpace] = OrderedDict()
for (name, n) in [ for (name, n) in [
('DeviceGray', 1), # default value first ('DeviceGray', 1), # default value first
@@ -24,6 +24,14 @@ class LTCurve(LTComponent):
self.stroke = stroke self.stroke = stroke
self.fill = fill self.fill = fill
def __repr__(self):
return ('<%s [%s] fill=%s stroke=%s>' % (
self.__class__.__name__,
str(self.bbox),
self.fill,
self.stroke
))
## LTLine ## LTLine
## ##
class LTLine(LTCurve): class LTLine(LTCurve):
@@ -1,7 +1,19 @@
import os
from io import BytesIO
import PIL as pillow
from pdfmajor.BMPWriter import BMPWriter
from pdfmajor.utils import Bbox from pdfmajor.utils import Bbox
from pdfmajor.parser.PDFStream import PDFStream from pdfmajor.parser.PDFStream import PDFStream, LITERALS_DCT_DECODE
from pdfmajor.parser.constants import LIT
from ._base import LTComponent from ._base import LTComponent
LITERAL_DEVICE_GRAY = LIT('DeviceGray')
LITERAL_DEVICE_RGB = LIT('DeviceRGB')
LITERAL_DEVICE_CMYK = LIT('DeviceCMYK')
## LTImage ## LTImage
## ##
class LTImage(LTComponent): class LTImage(LTComponent):
@@ -23,3 +35,45 @@ class LTImage(LTComponent):
return ('<%s(%s) %s %r>' % return ('<%s(%s) %s %r>' %
(self.__class__.__name__, self.name, (self.__class__.__name__, self.name,
str(self.bbox), self.srcsize)) str(self.bbox), self.srcsize))
def save_image(self, outdir: str, file_name: str = None):
stream = self.stream
filters = stream.get_filters()
width = self.bbox.width
height = self.bbox.height
bits = self.bits
if len(filters) == 1 and filters[0][0] in LITERALS_DCT_DECODE:
ext = '.jpg'
elif (bits == 1 or
bits == 8 and self.colorspace in (LITERAL_DEVICE_RGB, LITERAL_DEVICE_GRAY)):
ext = '.w%dh%d.bmp' % (width, height)
else:
ext = '.b%dw%dh%d.img' % (bits, width, height)
if file_name is None:
file_name = self.name + ext
else:
file_name += ext
path = os.path.join(outdir, file_name)
with open(path, 'wb') as fp:
if ext == '.jpg':
raw_data = stream.get_rawdata()
if LITERAL_DEVICE_CMYK in self.colorspace:
ifp = BytesIO(raw_data)
i = pillow.Image.open(ifp)
i = pillow.ImageChops.invert(i)
i = i.convert('RGB')
i.save(fp, 'JPEG')
else:
fp.write(raw_data)
elif bits == 1:
BMPWriter.write_all(fp, stream, 1, width, height, (width+7)//8)
elif bits == 8 and self.colorspace is LITERAL_DEVICE_RGB:
BMPWriter.write_all(fp, stream, 24, width, height, width*3)
elif bits == 8 and self.colorspace is LITERAL_DEVICE_GRAY:
BMPWriter.write_all(fp, stream, 8, width, height, width)
else:
fp.write(stream.get_data())
return path
@@ -90,9 +90,9 @@ class LTComponent(LTItem):
## ##
class LTContainer(LTComponent): class LTContainer(LTComponent):
def __init__(self, bbox: Bbox, objs: List[LTItem] = []): def __init__(self, bbox: Bbox, objs: List[LTItem] = None):
LTComponent.__init__(self, bbox) LTComponent.__init__(self, bbox)
self._objs: List[LTItem] = objs self._objs: List[LTItem] = [] if objs is None else objs
return return
def __iter__(self): def __iter__(self):
-35
View File
@@ -1,35 +0,0 @@
import collections
from .PSStackParser import LIT
## PDFColorSpace
##
LITERAL_DEVICE_GRAY = LIT('DeviceGray')
LITERAL_DEVICE_RGB = LIT('DeviceRGB')
LITERAL_DEVICE_CMYK = LIT('DeviceCMYK')
class PDFColorSpace(object):
def __init__(self, name: str, ncomponents: int):
self.name: str = name
self.ncomponents: int = ncomponents
return
def __repr__(self):
return '<PDFColorSpace: %s, ncomponents=%d>' % (self.name, self.ncomponents)
PREDEFINED_COLORSPACE = collections.OrderedDict()
for (name, n) in [
('DeviceGray', 1), # default value first
('CalRGB', 3),
('CalGray', 1),
('Lab', 3),
('DeviceRGB', 3),
('DeviceCMYK', 4),
('Separation', 1),
('Indexed', 1),
('Pattern', 1),
]:
PREDEFINED_COLORSPACE[name]=PDFColorSpace(name, n)
+4 -2
View File
@@ -1,5 +1,6 @@
import re import re
from logging import getLogger from logging import getLogger
from typing import Tuple, Optional, Dict
from ...utils import settings, choplist, nunpack from ...utils import settings, choplist, nunpack
from ..PSStackParser import PSEOF from ..PSStackParser import PSEOF
@@ -13,6 +14,7 @@ from .exceptions import PDFNoValidXRef, PDFSyntaxError
log = getLogger(__name__) log = getLogger(__name__)
Pos = Tuple[Optional[int], int, int]
## XRefs ## XRefs
## ##
@@ -27,7 +29,7 @@ class PDFBaseXRef(object):
# Must return # Must return
# (strmid, index, genno) # (strmid, index, genno)
# or (None, pos, genno) # or (None, pos, genno)
def get_pos(self, objid): def get_pos(self, objid) -> Pos:
raise KeyError(objid) raise KeyError(objid)
@@ -36,7 +38,7 @@ class PDFBaseXRef(object):
class PDFXRef(PDFBaseXRef): class PDFXRef(PDFBaseXRef):
def __init__(self): def __init__(self):
self.offsets = {} self.offsets: Dict[int, Pos] = {}
self.trailer = {} self.trailer = {}
return return
+6
View File
@@ -13,6 +13,12 @@ class Bbox:
self.x1 = x1 self.x1 = x1
self.y0 = y0 self.y0 = y0
self.y1 = y1 self.y1 = y1
if (x0 is not None) and (x1 is not None):
self.x0 = min([x0,x1])
self.x1 = max([x0,x1])
if (y0 is not None) and (y1 is not None):
self.y0 = min([y0,y1])
self.y1 = max([y0,y1])
@property @property
def height(self) -> float: def height(self) -> float:
Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 576 KiB

Binary file not shown.
Binary file not shown.
+10 -3
View File
@@ -10,12 +10,19 @@ import sys
with open('README.md') as readme_file: with open('README.md') as readme_file:
SETUP = { SETUP = {
"name": "pdfmajor", "name": "pdfmajor",
"version": "1.3.1", "version": "1.3.9",
"packages": find_packages(include=['pdfmajor*']), "packages": find_packages(include=['pdfmajor*']),
"install_requires": [ "install_requires": [
'pycryptodome', 'pycryptodome==3.8.2',
'chardet' 'chardet==3.0.4',
'Pillow==6.0.0'
], ],
'extras_require': {
'dev': [
'tqdm==4.32.1',
'nose==1.3.7'
]
},
"description": 'PDF parser', "description": 'PDF parser',
"long_description": readme_file.read(), "long_description": readme_file.read(),
'long_description_content_type': 'text/markdown', 'long_description_content_type': 'text/markdown',
+4
View File
@@ -0,0 +1,4 @@
from unittest import TestLoader, TextTestRunner
tests = TestLoader().discover('./tests')
result = TextTestRunner(verbosity=20).run(tests)
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
BIN
View File
Binary file not shown.
Binary file not shown.
+38
View File
@@ -0,0 +1,38 @@
import os
from unittest import TestCase, main
# from pdfmajor.extractor import extract_items_from_pdf
from pdfmajor.interpreter import LTCharBlock, PDFInterpreter, logging
from pdfmajor.interpreter import PageInterpreter
from pdfmajor.interpreter.commands import LTContainer
CUR_PATH = os.path.dirname(os.path.dirname(__file__))
INPUT_FOLDER = os.path.join(
CUR_PATH, "tests/samples/pdf"
)
FILES = os.listdir(INPUT_FOLDER)
NUM_LOOPS = 10
def count_elms(file_path: str):
count = 0
for page in PDFInterpreter(file_path):
for item in page:
if isinstance(item, LTContainer):
count += len([*item])
else:
count += 1
return count
class CachingTester(TestCase):
def test_cache(self):
for file_name in FILES:
self.subTest(name = file_name)
file_path = os.path.join(INPUT_FOLDER, file_name)
last_count = count_elms(file_path)
for _ in range(NUM_LOOPS):
new_count = count_elms(file_path)
self.assertEqual( new_count , last_count )
last_count = new_count
if __name__ == '__main__':
# Run Tests
main()
+115
View File
@@ -0,0 +1,115 @@
import os
from typing import List
from unittest import TestCase, main
# from pdfmajor.extractor import extract_items_from_pdf
from pdfmajor.interpreter import LTCharBlock, PDFInterpreter, logging
from pdfmajor.interpreter import PageInterpreter
from pdfmajor.interpreter.commands import LTCurve, PDFColor, LTXObject
from pdfmajor.interpreter.commands import PREDEFINED_COLORSPACE
CUR_PATH = os.path.dirname(os.path.dirname(__file__))
INPUT_FOLDER = os.path.join(
CUR_PATH, "tests/samples/pdf"
)
FILES = os.listdir(INPUT_FOLDER)
NUM_LOOPS = 10
class ColorTest(TestCase):
def run_test(self, file_path: str, curve_defs: List[List[PDFColor]]):
def run_once(item):
if isinstance(item, LTCurve):
t_stroke, t_fill = curve_defs.pop(0)
self.assertEqual( item.stroke.values, t_stroke.values )
self.assertEqual( item.stroke.color_space, t_stroke.color_space )
self.assertEqual( item.fill.values, t_fill.values )
self.assertEqual( item.fill.color_space, t_fill.color_space )
elif isinstance(item, LTXObject):
for c in item:
run_once(c)
for page in PDFInterpreter(file_path):
for item in page:
run_once(item)
def test_lorem_v3(self):
self.run_test(
os.path.join(INPUT_FOLDER, "lorem-v3.pdf"),
[
[
PDFColor(None),
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 255, 255, 255),
],
[
PDFColor(None),
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 114, 159, 207),
],
[
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 255, 255, 0),
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 114, 159, 207),
],
[
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 255, 255, 0),
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 114, 159, 207),
],
[
PDFColor(None),
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 255, 255, 255),
],
[
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 0, 0, 128),
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 255, 0, 0),
]
] +
([[
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 0, 0, 128),
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 255, 255, 255),
]]*24)+
([[
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 0, 0, 128),
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 0, 0, 0),
]]*11)+
([[
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 128, 128, 128),
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 0, 0, 0),
]]*5)+
[[
PDFColor(None),
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 255, 255, 255),
]]
)
def test_lorem_v4(self):
self.run_test(
os.path.join(INPUT_FOLDER, "lorem-v4.pdf"),
[
[
PDFColor(None),
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 255, 255, 255),
],
[
PDFColor(None),
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 64, 64, 64),
]
]
)
def test_lorem_v5(self):
self.run_test(
os.path.join(INPUT_FOLDER, "lorem-v5.pdf"),
[
[
PDFColor(None),
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 255, 255, 255),
],
[
PDFColor(None),
PDFColor(PREDEFINED_COLORSPACE['DeviceRGB'], 128, 128, 128),
]
]
)
if __name__ == '__main__':
# Run Tests
main()
+36 -27
View File
@@ -3,13 +3,14 @@ import re
import os import os
import logging import logging
from unittest import TestCase, main
from typing import List from typing import List
from pdfmajor.converters import convert_file from pdfmajor.converters import convert_file
from xml.etree import ElementTree from xml.etree import ElementTree
from json import load as json_load from json import load as json_load
from tqdm import tqdm from tqdm import tqdm
def rel_path(*arg: List[str]) -> str: def rel_path(*arg: str) -> str:
return os.path.normpath(os.path.join(os.path.dirname(__file__), *arg)) return os.path.normpath(os.path.join(os.path.dirname(__file__), *arg))
EXTS = ['text', 'json','xml','html'] EXTS = ['text', 'json','xml','html']
@@ -22,30 +23,38 @@ if not os.path.exists(OUTPUT_FOLDER):
files = os.listdir(INPUT_FOLDER) files = os.listdir(INPUT_FOLDER)
TOTAL = len(files)*len(EXTS) TOTAL = len(files)*len(EXTS)
waiter = tqdm(total=TOTAL)
for idx, file_name in enumerate(files):
file_name_no_ext = re.sub(r'\.pdf$',"", file_name)
input_file_path = os.path.join(INPUT_FOLDER, file_name)
for ext in EXTS: class ConversionTest(TestCase):
if not os.path.exists(os.path.join(OUTPUT_FOLDER, ext)): def test_conversion(self):
os.makedirs(os.path.join(OUTPUT_FOLDER, ext)) # with tqdm(total=TOTAL) as waiter:
output_file_path = os.path.join(OUTPUT_FOLDER, ext, file_name_no_ext + "." + ext) for idx, file_name in enumerate(files):
if len(file_name) > len(file_name_no_ext): self.subTest(i=idx);
waiter.set_description(f"Processing {file_name}") file_name_no_ext = re.sub(r'\.pdf$',"", file_name)
convert_file( input_file_path = os.path.join(INPUT_FOLDER, file_name)
input_file=input_file_path,
output_file=output_file_path, for ext in EXTS:
out_type=ext, if not os.path.exists(os.path.join(OUTPUT_FOLDER, ext)):
debug_level=logging.WARN, os.makedirs(os.path.join(OUTPUT_FOLDER, ext))
) output_file_path = os.path.join(OUTPUT_FOLDER, ext, file_name_no_ext + "." + ext)
if ext == 'xml': if len(file_name) > len(file_name_no_ext):
waiter.set_description(f" > testing format {file_name}") # waiter.set_description(f"Conversion test -- {file_name}")
ElementTree.parse(output_file_path) convert_file(
elif ext == 'json': input_file=input_file_path,
waiter.set_description(f" > testing format {file_name}") output_file=output_file_path,
json_load(open(output_file_path, 'r')) out_type=ext,
waiter.refresh() debug_level=logging.WARN,
waiter.update() )
waiter.set_description("Done") if ext == 'xml':
waiter.close() # waiter.set_description(f" > testing format {file_name}")
ElementTree.parse(output_file_path)
elif ext == 'json':
# waiter.set_description(f" > testing format {file_name}")
with open(output_file_path, 'r') as outf:
json_load(outf)
# waiter.update()
# waiter.set_description("Done")
if __name__ == '__main__':
# Run Tests
main()
+52 -30
View File
@@ -1,43 +1,60 @@
from contextlib import contextmanager
from unittest import TestCase, main
from typing import List, Tuple, Callable
from os import remove from os import remove
from tempfile import mktemp from tempfile import mktemp
from pdfmajor.converters.writers.html import HTMLMaker from pdfmajor.converters.writers.html import HTMLMaker
from tqdm import tqdm
# Make temp file # Make temp file
FILE_NAME = mktemp() FILE_NAME = mktemp()
TestFunction = Callable[[HTMLMaker], str];
TESTS: List[Tuple[TestFunction, str]] = [];
with HTMLMaker(FILE_NAME) as html: class HTMLTester(TestCase):
with html.elm("html", nolineend=True): @contextmanager
html.write("test") def get_html(self):
assert(open(FILE_NAME, 'r').read() == """<html > with HTMLMaker(FILE_NAME) as html:
yield html
def compare(self, expected: str):
with open(FILE_NAME, 'r') as f:
self.assertEqual(f.read() , expected)
def test_html_basic(self):
with self.get_html() as html:
with html.elm("html", nolineend=True):
html.write("test")
self.compare("""<html >
test test
</html>""") </html>""")
def test_html_with_attributes(self):
with HTMLMaker(FILE_NAME) as html: with self.get_html() as html:
with html.elm("html", nolineend=True): with html.elm("html", nolineend=True):
with html.elm("head"): with html.elm("head"):
html.singleton("meta", { html.singleton("meta", {
'http-equiv': "Content-Type", 'http-equiv': "Content-Type",
'content': "text/html" 'content': "text/html"
}) })
assert(open(FILE_NAME, 'r').read() == """<html > self.compare("""<html >
<head > <head >
<meta http-equiv="Content-Type" content="text/html" /> <meta http-equiv="Content-Type" content="text/html" />
</head> </head>
</html>""") </html>""")
with HTMLMaker(FILE_NAME) as html: def test_html_with_children(self):
with html.elm("html", nolineend=True): with self.get_html() as html:
with html.elm("head"): with html.elm("html", nolineend=True):
html.singleton("meta", { with html.elm("head"):
'http-equiv': "Content-Type", html.singleton("meta", {
'content': "text/html" 'http-equiv': "Content-Type",
}) 'content': "text/html"
with html.elm("body"): })
with html.elm("h1", css={"color": "red"}): with html.elm("body"):
html.write("Hello World") with html.elm("h1", css={"color": "red"}):
assert(open(FILE_NAME, 'r').read() == """<html > html.write("Hello World")
self.compare("""<html >
<head > <head >
<meta http-equiv="Content-Type" content="text/html" /> <meta http-equiv="Content-Type" content="text/html" />
</head> </head>
@@ -48,13 +65,18 @@ assert(open(FILE_NAME, 'r').read() == """<html >
</body> </body>
</html>""") </html>""")
with HTMLMaker(FILE_NAME) as html: def test_html_with_css(self):
with html.elm('style'): with self.get_html() as html:
html.write('''.page { position: relative; overflow: hidden;border: 1px black solid;}''') with html.elm('style'):
assert(open(FILE_NAME, 'r').read() == """<style > html.write('''.page { position: relative; overflow: hidden;border: 1px black solid;}''')
self.compare("""<style >
.page { position: relative; overflow: hidden;border: 1px black solid;} .page { position: relative; overflow: hidden;border: 1px black solid;}
</style> </style>
""") """)
# Clean up
remove(FILE_NAME) if __name__ == '__main__':
# Run Tests
main()
# Clean up
remove(FILE_NAME)
+41
View File
@@ -0,0 +1,41 @@
import os
from unittest import TestCase, main
from pdfmajor.interpreter import LTImage, logging
from pdfmajor.interpreter import PageInterpreter, PDFInterpreter
from pdfmajor.interpreter.commands import LTItem
from pdfmajor.interpreter.commands import LTCharBlock, LTChar
CUR_PATH = os.path.dirname(os.path.dirname(__file__))
FILES = [
"jpg",
"looseless"
]
OUTPUT_FOLDER = os.path.join(
CUR_PATH,
'samples/output/images'
)
if not os.path.exists(OUTPUT_FOLDER):
os.makedirs(OUTPUT_FOLDER)
class ImageTester(TestCase):
def test_image_extract(self):
for file_name in FILES:
self.subTest(file_name=file_name)
for page in PDFInterpreter(os.path.join(
CUR_PATH,
f"tests/samples/pdf/{file_name}.pdf"
), debug_level=logging.ERROR):
self.assertTrue( isinstance(page, PageInterpreter) )
for item_idx, item in enumerate(page):
self.assertTrue( isinstance(item, LTItem) )
if isinstance(item, LTImage):
item.save_image(
OUTPUT_FOLDER,
file_name=f'{file_name}_p{page.page_num}i{item_idx}'
)
if __name__ == '__main__':
# Run Tests
main()
+20 -12
View File
@@ -1,21 +1,29 @@
import os import os
from unittest import TestCase, main
# from pdfmajor.extractor import extract_items_from_pdf
from pdfmajor.interpreter import LTCharBlock, PDFInterpreter, logging from pdfmajor.interpreter import LTCharBlock, PDFInterpreter, logging
from pdfmajor.interpreter import PageInterpreter from pdfmajor.interpreter import PageInterpreter
from pdfmajor.interpreter.commands import LTItem from pdfmajor.interpreter.commands import LTItem
from pdfmajor.interpreter.commands import LTCharBlock, LTChar from pdfmajor.interpreter.commands import LTCharBlock, LTChar
CUR_PATH = os.path.dirname(os.path.dirname(__file__)) CUR_PATH = os.path.dirname(os.path.dirname(__file__))
FILE_NAME = os.path.join( INPUT_FOLDER = os.path.join(
CUR_PATH, CUR_PATH, "tests/samples/pdf"
"samples/pdf/tables.pdf"
) )
FILES = os.listdir(INPUT_FOLDER)
for page in PDFInterpreter(FILE_NAME, debug_level=logging.INFO): class TestInterpreter(TestCase):
assert isinstance(page, PageInterpreter) def test_pdfintp(self):
for item in page: for file_name in FILES:
assert isinstance(item, LTItem) self.subTest(file_name=file_name)
if isinstance(item, LTCharBlock): file_path = os.path.join(INPUT_FOLDER, file_name)
for char in item: for page in PDFInterpreter(file_path, debug_level=logging.ERROR):
assert(char, LTChar) self.assertTrue( isinstance(page, PageInterpreter) )
for item in page:
self.assertTrue( isinstance(item, LTItem) )
if isinstance(item, LTCharBlock):
for char in item:
self.assertTrue( isinstance(char, LTChar) )
if __name__ == '__main__':
# Run Tests
main()
-1
View File
@@ -80,7 +80,6 @@ with JSONMaker(FILE_NAME) as obj:
item.number("x1", 4.2) item.number("x1", 4.2)
item.number("y0", 231.12) item.number("y0", 231.12)
item.number("y1", 331.12) item.number("y1", 331.12)
print(open(FILE_NAME, 'r').read())
assert(open(FILE_NAME, 'r').read() == """{ assert(open(FILE_NAME, 'r').read() == """{
"pages": [ "pages": [
{ {
+44 -49
View File
@@ -2,17 +2,12 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
from nose.tools import assert_equal, assert_true, assert_false import unittest
from nose import SkipTest
import nose
import logging
from pdfmajor.parser.PDFStream.ccitt import CCITTG4Parser from pdfmajor.parser.PDFStream.ccitt import CCITTG4Parser
## Test cases ## Test cases
## ##
class TestCCITTG4Parser(): class TestCCITTG4Parser(unittest.TestCase):
def get_parser(self, bits): def get_parser(self, bits):
parser = CCITTG4Parser(len(bits)) parser = CCITTG4Parser(len(bits))
@@ -23,98 +18,98 @@ class TestCCITTG4Parser():
def test_b1(self): def test_b1(self):
parser = self.get_parser('00000') parser = self.get_parser('00000')
parser._do_vertical(0) parser._do_vertical(0)
assert_equal(parser._curpos, 0) self.assertEqual(parser._curpos, 0)
return return
def test_b2(self): def test_b2(self):
parser = self.get_parser('10000') parser = self.get_parser('10000')
parser._do_vertical(-1) parser._do_vertical(-1)
assert_equal(parser._curpos, 0) self.assertEqual(parser._curpos, 0)
return return
def test_b3(self): def test_b3(self):
parser = self.get_parser('000111') parser = self.get_parser('000111')
parser._do_pass() parser._do_pass()
assert_equal(parser._curpos, 3) self.assertEqual(parser._curpos, 3)
assert_equal(parser._get_bits(), '111') self.assertEqual(parser._get_bits(), '111')
return return
def test_b4(self): def test_b4(self):
parser = self.get_parser('00000') parser = self.get_parser('00000')
parser._do_vertical(+2) parser._do_vertical(+2)
assert_equal(parser._curpos, 2) self.assertEqual(parser._curpos, 2)
assert_equal(parser._get_bits(), '11') self.assertEqual(parser._get_bits(), '11')
return return
def test_b5(self): def test_b5(self):
parser = self.get_parser('11111111100') parser = self.get_parser('11111111100')
parser._do_horizontal(0, 3) parser._do_horizontal(0, 3)
assert_equal(parser._curpos, 3) self.assertEqual(parser._curpos, 3)
parser._do_vertical(1) parser._do_vertical(1)
assert_equal(parser._curpos, 10) self.assertEqual(parser._curpos, 10)
assert_equal(parser._get_bits(), '0001111111') self.assertEqual(parser._get_bits(), '0001111111')
return return
def test_e1(self): def test_e1(self):
parser = self.get_parser('10000') parser = self.get_parser('10000')
parser._do_vertical(0) parser._do_vertical(0)
assert_equal(parser._curpos, 1) self.assertEqual(parser._curpos, 1)
parser._do_vertical(0) parser._do_vertical(0)
assert_equal(parser._curpos, 5) self.assertEqual(parser._curpos, 5)
assert_equal(parser._get_bits(), '10000') self.assertEqual(parser._get_bits(), '10000')
return return
def test_e2(self): def test_e2(self):
parser = self.get_parser('10011') parser = self.get_parser('10011')
parser._do_vertical(0) parser._do_vertical(0)
assert_equal(parser._curpos, 1) self.assertEqual(parser._curpos, 1)
parser._do_vertical(2) parser._do_vertical(2)
assert_equal(parser._curpos, 5) self.assertEqual(parser._curpos, 5)
assert_equal(parser._get_bits(), '10000') self.assertEqual(parser._get_bits(), '10000')
return return
def test_e3(self): def test_e3(self):
parser = self.get_parser('011111') parser = self.get_parser('011111')
parser._color = 0 parser._color = 0
parser._do_vertical(0) parser._do_vertical(0)
assert_equal(parser._color, 1) self.assertEqual(parser._color, 1)
assert_equal(parser._curpos, 1) self.assertEqual(parser._curpos, 1)
parser._do_vertical(-2) parser._do_vertical(-2)
assert_equal(parser._color, 0) self.assertEqual(parser._color, 0)
assert_equal(parser._curpos, 4) self.assertEqual(parser._curpos, 4)
parser._do_vertical(0) parser._do_vertical(0)
assert_equal(parser._curpos, 6) self.assertEqual(parser._curpos, 6)
assert_equal(parser._get_bits(), '011100') self.assertEqual(parser._get_bits(), '011100')
return return
def test_e4(self): def test_e4(self):
parser = self.get_parser('10000') parser = self.get_parser('10000')
parser._do_vertical(0) parser._do_vertical(0)
assert_equal(parser._curpos, 1) self.assertEqual(parser._curpos, 1)
parser._do_vertical(-2) parser._do_vertical(-2)
assert_equal(parser._curpos, 3) self.assertEqual(parser._curpos, 3)
parser._do_vertical(0) parser._do_vertical(0)
assert_equal(parser._curpos, 5) self.assertEqual(parser._curpos, 5)
assert_equal(parser._get_bits(), '10011') self.assertEqual(parser._get_bits(), '10011')
return return
def test_e5(self): def test_e5(self):
parser = self.get_parser('011000') parser = self.get_parser('011000')
parser._color = 0 parser._color = 0
parser._do_vertical(0) parser._do_vertical(0)
assert_equal(parser._curpos, 1) self.assertEqual(parser._curpos, 1)
parser._do_vertical(3) parser._do_vertical(3)
assert_equal(parser._curpos, 6) self.assertEqual(parser._curpos, 6)
assert_equal(parser._get_bits(), '011111') self.assertEqual(parser._get_bits(), '011111')
return return
def test_e6(self): def test_e6(self):
parser = self.get_parser('11001') parser = self.get_parser('11001')
parser._do_pass() parser._do_pass()
assert_equal(parser._curpos, 4) self.assertEqual(parser._curpos, 4)
parser._do_vertical(0) parser._do_vertical(0)
assert_equal(parser._curpos, 5) self.assertEqual(parser._curpos, 5)
assert_equal(parser._get_bits(), '11111') self.assertEqual(parser._get_bits(), '11111')
return return
def test_e7(self): def test_e7(self):
@@ -122,8 +117,8 @@ class TestCCITTG4Parser():
parser._curpos = 2 parser._curpos = 2
parser._color = 1 parser._color = 1
parser._do_horizontal(2, 6) parser._do_horizontal(2, 6)
assert_equal(parser._curpos, 10) self.assertEqual(parser._curpos, 10)
assert_equal(parser._get_bits(), '1111000000') self.assertEqual(parser._get_bits(), '1111000000')
return return
def test_e8(self): def test_e8(self):
@@ -131,19 +126,19 @@ class TestCCITTG4Parser():
parser._curpos = 1 parser._curpos = 1
parser._color = 0 parser._color = 0
parser._do_vertical(0) parser._do_vertical(0)
assert_equal(parser._curpos, 2) self.assertEqual(parser._curpos, 2)
parser._do_horizontal(7, 0) parser._do_horizontal(7, 0)
assert_equal(parser._curpos, 9) self.assertEqual(parser._curpos, 9)
assert_equal(parser._get_bits(), '101111111') self.assertEqual(parser._get_bits(), '101111111')
return return
def test_m1(self): def test_m1(self):
parser = self.get_parser('10101') parser = self.get_parser('10101')
parser._do_pass() parser._do_pass()
assert_equal(parser._curpos, 2) self.assertEqual(parser._curpos, 2)
parser._do_pass() parser._do_pass()
assert_equal(parser._curpos, 4) self.assertEqual(parser._curpos, 4)
assert_equal(parser._get_bits(), '1111') self.assertEqual(parser._get_bits(), '1111')
return return
def test_m2(self): def test_m2(self):
@@ -152,7 +147,7 @@ class TestCCITTG4Parser():
parser._do_vertical(-1) parser._do_vertical(-1)
parser._do_vertical(1) parser._do_vertical(1)
parser._do_horizontal(1, 1) parser._do_horizontal(1, 1)
assert_equal(parser._get_bits(), '011101') self.assertEqual(parser._get_bits(), '011101')
return return
def test_m3(self): def test_m3(self):
@@ -161,8 +156,8 @@ class TestCCITTG4Parser():
parser._do_pass() parser._do_pass()
parser._do_vertical(1) parser._do_vertical(1)
parser._do_vertical(1) parser._do_vertical(1)
assert_equal(parser._get_bits(), '00000001') self.assertEqual(parser._get_bits(), '00000001')
return return
if __name__ == '__main__': if __name__ == '__main__':
nose.runmodule() unittest.main()
+12 -14
View File
@@ -1,9 +1,7 @@
#!/usr/bin/env python #!/usr/bin/env python
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
from nose.tools import assert_equal from unittest import TestCase, main
from nose import SkipTest
import nose
#test of various compression/encoding modules (previously in doctests): #test of various compression/encoding modules (previously in doctests):
from pdfmajor.utils.ascii85 import ascii85decode, asciihexdecode from pdfmajor.utils.ascii85 import ascii85decode, asciihexdecode
@@ -14,24 +12,24 @@ import binascii
def hex(b): return binascii.hexlify(b) #encode('hex') def hex(b): return binascii.hexlify(b) #encode('hex')
def dehex(b): return binascii.unhexlify(b) #decode('hex') def dehex(b): return binascii.unhexlify(b) #decode('hex')
class TestAscii85(): class TestAscii85(TestCase):
def test_ascii85decode(self): def test_ascii85decode(self):
#The sample string is taken from: http://en.wikipedia.org/w/index.php?title=Ascii85 #The sample string is taken from: http://en.wikipedia.org/w/index.php?title=Ascii85
assert_equal(ascii85decode(b'9jqo^BlbD-BleB1DJ+*+F(f,q'),b'Man is distinguished') self.assertEqual(ascii85decode(b'9jqo^BlbD-BleB1DJ+*+F(f,q'),b'Man is distinguished')
assert_equal(ascii85decode(b'E,9)oF*2M7/c~>'),b'pleasure.') self.assertEqual(ascii85decode(b'E,9)oF*2M7/c~>'),b'pleasure.')
def test_asciihexdecode(self): def test_asciihexdecode(self):
assert_equal(asciihexdecode(b'61 62 2e6364 65'),b'ab.cde') self.assertEqual(asciihexdecode(b'61 62 2e6364 65'),b'ab.cde')
assert_equal(asciihexdecode(b'61 62 2e6364 657>'),b'ab.cdep') self.assertEqual(asciihexdecode(b'61 62 2e6364 657>'),b'ab.cdep')
assert_equal(asciihexdecode(b'7>'),b'p') self.assertEqual(asciihexdecode(b'7>'),b'p')
class TestLzw(): class TestLzw(TestCase):
def test_lzwdecode(self): def test_lzwdecode(self):
assert_equal(lzwdecode(b'\x80\x0b\x60\x50\x22\x0c\x0c\x85\x01'),b'\x2d\x2d\x2d\x2d\x2d\x41\x2d\x2d\x2d\x42') self.assertEqual(lzwdecode(b'\x80\x0b\x60\x50\x22\x0c\x0c\x85\x01'),b'\x2d\x2d\x2d\x2d\x2d\x41\x2d\x2d\x2d\x42')
class TestRunlength(): class TestRunlength(TestCase):
def test_rldecode(self): def test_rldecode(self):
assert_equal(rldecode(b'\x05123456\xfa7\x04abcde\x80junk'),b'1234567777777abcde') self.assertEqual(rldecode(b'\x05123456\xfa7\x04abcde\x80junk'),b'1234567777777abcde')
if __name__ == '__main__': if __name__ == '__main__':
nose.runmodule() main()
+33 -44
View File
@@ -2,19 +2,14 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
from nose.tools import assert_equal, assert_true, assert_false
from nose import SkipTest
import nose
import logging import logging
from unittest import TestCase, main
from io import BytesIO
from pdfmajor.parser.PSStackParser import KWD, LIT, PSBaseParser, PSEOF, PSStackParser from pdfmajor.parser.PSStackParser import KWD, LIT, PSBaseParser, PSEOF, PSStackParser
## Simplistic Test cases ## Simplistic Test cases
## ##
class TestPSBaseParser: TESTDATA = br'''%!PS
TESTDATA = br'''%!PS
begin end begin end
" @ # " @ #
/a/BCD /Some_Name /foo#5f#xbaa /a/BCD /Some_Name /foo#5f#xbaa
@@ -33,36 +28,32 @@ func/a/b{(c)do*}def
[ 1 (z) ! ] [ 1 (z) ! ]
<< /foo (bar) >> << /foo (bar) >>
''' '''
TOKENS = [
TOKENS = [ (5, KWD(b'begin')), (11, KWD(b'end')), (16, KWD(b'"')), (19, KWD(b'@')),
(5, KWD(b'begin')), (11, KWD(b'end')), (16, KWD(b'"')), (19, KWD(b'@')), (21, KWD(b'#')), (23, LIT('a')), (25, LIT('BCD')), (30, LIT('Some_Name')),
(21, KWD(b'#')), (23, LIT('a')), (25, LIT('BCD')), (30, LIT('Some_Name')), (41, LIT('foo_xbaa')), (54, 0), (56, 1), (59, -2), (62, 0.5),
(41, LIT('foo_xbaa')), (54, 0), (56, 1), (59, -2), (62, 0.5), (65, 1.234), (71, b'abc'), (77, b''), (80, b'abc ( def ) ghi'),
(65, 1.234), (71, b'abc'), (77, b''), (80, b'abc ( def ) ghi'), (98, b'def \x00 4ghi'), (118, b'bach\\slask'), (132, b'foo\nbaa'),
(98, b'def \x00 4ghi'), (118, b'bach\\slask'), (132, b'foo\nbaa'), (143, b'this % is not a comment.'), (170, b'foo\nbaa'), (180, b'foobaa'),
(143, b'this % is not a comment.'), (170, b'foo\nbaa'), (180, b'foobaa'), (191, b''), (194, b' '), (199, b'@@ '), (211, b'\xab\xcd\x00\x124\x05'),
(191, b''), (194, b' '), (199, b'@@ '), (211, b'\xab\xcd\x00\x124\x05'), (226, KWD(b'func')), (230, LIT('a')), (232, LIT('b')),
(226, KWD(b'func')), (230, LIT('a')), (232, LIT('b')), (234, KWD(b'{')), (235, b'c'), (238, KWD(b'do*')), (241, KWD(b'}')),
(234, KWD(b'{')), (235, b'c'), (238, KWD(b'do*')), (241, KWD(b'}')), (242, KWD(b'def')), (246, KWD(b'[')), (248, 1), (250, b'z'), (254, KWD(b'!')),
(242, KWD(b'def')), (246, KWD(b'[')), (248, 1), (250, b'z'), (254, KWD(b'!')), (256, KWD(b']')), (258, KWD(b'<<')), (261, LIT('foo')), (266, b'bar'),
(256, KWD(b']')), (258, KWD(b'<<')), (261, LIT('foo')), (266, b'bar'), (272, KWD(b'>>'))
(272, KWD(b'>>')) ]
] OBJS = [
(23, LIT('a')), (25, LIT('BCD')), (30, LIT('Some_Name')),
OBJS = [ (41, LIT('foo_xbaa')), (54, 0), (56, 1), (59, -2), (62, 0.5),
(23, LIT('a')), (25, LIT('BCD')), (30, LIT('Some_Name')), (65, 1.234), (71, b'abc'), (77, b''), (80, b'abc ( def ) ghi'),
(41, LIT('foo_xbaa')), (54, 0), (56, 1), (59, -2), (62, 0.5), (98, b'def \x00 4ghi'), (118, b'bach\\slask'), (132, b'foo\nbaa'),
(65, 1.234), (71, b'abc'), (77, b''), (80, b'abc ( def ) ghi'), (143, b'this % is not a comment.'), (170, b'foo\nbaa'), (180, b'foobaa'),
(98, b'def \x00 4ghi'), (118, b'bach\\slask'), (132, b'foo\nbaa'), (191, b''), (194, b' '), (199, b'@@ '), (211, b'\xab\xcd\x00\x124\x05'),
(143, b'this % is not a comment.'), (170, b'foo\nbaa'), (180, b'foobaa'), (230, LIT('a')), (232, LIT('b')), (234, [b'c']), (246, [1, b'z']),
(191, b''), (194, b' '), (199, b'@@ '), (211, b'\xab\xcd\x00\x124\x05'), (258, {'foo': b'bar'}),
(230, LIT('a')), (232, LIT('b')), (234, [b'c']), (246, [1, b'z']), ]
(258, {'foo': b'bar'}), class TestPSBaseParser(TestCase):
]
def get_tokens(self, s): def get_tokens(self, s):
from io import BytesIO
class MyParser(PSStackParser): class MyParser(PSStackParser):
def flush(self): def flush(self):
self.add_results(*self.popall()) self.add_results(*self.popall())
@@ -76,8 +67,6 @@ func/a/b{(c)do*}def
return r return r
def get_objects(self, s): def get_objects(self, s):
from io import BytesIO
class MyParser(PSStackParser): class MyParser(PSStackParser):
def flush(self): def flush(self):
self.add_results(*self.popall()) self.add_results(*self.popall())
@@ -91,18 +80,18 @@ func/a/b{(c)do*}def
return r return r
def test_1(self): def test_1(self):
tokens = self.get_tokens(self.TESTDATA) tokens = self.get_tokens(TESTDATA)
logging.info(tokens) logging.info(tokens)
assert_equal(tokens, self.TOKENS) self.assertEqual(tokens, TOKENS)
return return
def test_2(self): def test_2(self):
objs = self.get_objects(self.TESTDATA) objs = self.get_objects(TESTDATA)
logging.info(objs) logging.info(objs)
assert_equal(objs, self.OBJS) self.assertEqual(objs, OBJS)
return return
if __name__ == '__main__': if __name__ == '__main__':
#import logging,sys,os,six #import logging,sys,os,six
#logging.basicConfig(level=logging.DEBUG, filename='%s_%d.%d.log'%(os.path.basename(__file__),sys.version_info[0],sys.version_info[1])) #logging.basicConfig(level=logging.DEBUG, filename='%s_%d.%d.log'%(os.path.basename(__file__),sys.version_info[0],sys.version_info[1]))
nose.runmodule() main()
+101
View File
@@ -0,0 +1,101 @@
from os import remove
from tempfile import mktemp
from pdfmajor.converters.writers.yaml import YAMLMaker
# Make temp file
FILE_NAME = mktemp()
with YAMLMaker(FILE_NAME) as yml:
yml.write("name", "bob")
yml.write("age", "20")
with yml.object("wife") as wife:
wife.write("name", "kory")
wife.write("age", "21")
with yml.array("children") as children:
with children.object() as child:
child.write("name", "Mike")
with child.array("toys") as toys:
toys.write("lion")
toys.write("bear")
with yml.object("computer") as computer:
computer.write("type", "awesome")
assert(open(FILE_NAME, 'r').read() == """name: bob
age: 20
wife:
name: kory
age: 21
children:
- name: Mike
toys:
- lion
- bear
computer:
type: awesome""")
with YAMLMaker(FILE_NAME) as yml:
yml.write("name", "bob")
yml.write("age", "20")
yml.place_object("wife", {
"name": "kory",
"age": 21
})
with yml.array("children") as children:
with children.object() as child:
child.write("name", "Mike")
child.place_array("toys", [
"lion", "bear"
])
yml.place_object("computer", {"type": "awesome"})
assert(open(FILE_NAME, 'r').read() == """name: bob
age: 20
wife:
name: kory
age: 21
children:
- name: Mike
toys:
- lion
- bear
computer:
type: awesome""")
with YAMLMaker(FILE_NAME) as yml:
with yml.array('pages') as pages:
with pages.object() as page:
page.write("id", 0)
page.write("width", 100)
page.write("height", 200)
with page.array("children") as children:
with children.object() as curve:
with curve.object("stroke") as fill:
fill.write("type", "DeviceRGB")
fill.write("value", ['1','2','3'])
curve.write("x0", 0)
curve.write("x1", 1)
with curve.object("fill") as fill:
fill.write("type", "DeviceRGB")
fill.write("value", ['1','2','3'])
curve.write("y0", 0)
curve.write("y1", 1)
assert(open(FILE_NAME, 'r').read() == """pages:
- id: 0
width: 100
height: 200
children:
- stroke:
type: DeviceRGB
value: ['1', '2', '3']
x0: 0
x1: 1
fill:
type: DeviceRGB
value: ['1', '2', '3']
y0: 0
y1: 1""")
# Clean up
remove(FILE_NAME)
+4 -5
View File
@@ -52,11 +52,10 @@ def main(args=None):
if len(parsed_args.files) == 1: if len(parsed_args.files) == 1:
if parsed_args.output_file != "-": if parsed_args.output_file != "-":
func_args['output_file'] = open(parsed_args.output_file, "wb") func_args['output_file'] = open(parsed_args.output_file, "wb")
with open(parsed_args.files[0], 'rb') as input_file: convert_file(
convert_file( parsed_args.files[0],
input_file, **func_args
**func_args )
)
else: else:
print("Multiple files not supported yet.") print("Multiple files not supported yet.")
return 1 return 1