diff --git a/README.md b/README.md index 6773938..f3bbb96 100644 --- a/README.md +++ b/README.md @@ -11,9 +11,7 @@ -PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability. - -The ultimate goal is to have a functional-programming with typing fork of the original library. +PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. The ultimate goal of this project is to provide a simpler, faster and more functional library that both casual and low-level developers could build upon. ## Documentation @@ -21,34 +19,32 @@ See documentation [here](https://asosnovsky.github.io/pdfmajor/). ## Basic Example ```py -from pdfmajor.extractor import extract_items_from_pdf +from pdfmajor.interpreter import PageInterpreter -for page in extract_items_from_pdf('path/to/your/file.pdf'): - print('page-start', page) +for page in PDFInterpreter("/path/to/pdf.pdf"): + print("page start", page.page_num) for item in page: - print(' ', item) - print('page-end', page) + print(" >", item) + print("page end", page.page_num) ``` ## Why Another Fork? -`PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with. - +`PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. However, after attempting to build and extend upon `PDFMiner.six` I found it rather difficult to extend and identify additional details regarding the items encoded in the pdf. This library will attempt to expose as much information as possible to end-users, without having them solely rely on to-xml or to-json exports. ## Features * Functional extraction method based on generators - * Written entirely in Python. * Parse, analyze, and convert PDF documents. * PDF-1.7 specification support. (well, almost) + * Font-Color extraction + * Shape fill and stroke color extraction * CJK languages and vertical writing scripts support. * Various font types (Type1, TrueType, Type3, and CID) support. * Basic encryption (RC4) support. - * Font-Color extraction - * Shape fill and stroke color extraction -**Note**: We took out the layout-analysis process in this version (there is no more LTTextHorizontal or LTTextVertical). While the mathematics behind the grouping process was sound, the coupling of the layout-analysis process with the parsing and interpretation process produced unfriendly-code. This feature could be brought back by extending the `pdfmajor.layouts.PDFLayoutAnalyzer` class, but at the current time is not supported. +**Note**: We took out the layout-analysis process in this version (there is no more LTTextHorizontal or LTTextVertical). While the mathematics behind the grouping process was sound, the coupling of the layout-analysis process with the parsing and interpretation process produced unfriendly-code. This feature could be brought back by running your own implementation of it on the `pdfmajor.interpreter.PageInterpreter` class, but at the current time is not supported. ## How to Install diff --git a/doc-src/api.md b/doc-src/api.md index 6c795bb..f7a4dd4 100644 --- a/doc-src/api.md +++ b/doc-src/api.md @@ -2,34 +2,37 @@ The library is constructed from 4 base modules: - * **extractor**: contains a high-level function for functional extraction of pdf-assets - * **interpreter**: contains low-level classes for extracting fundamental data-structures from the documents - * **converters**: contains high-level classes for conversion of the fundamental pdf structures to other formats + * **parser**: contains low-level classes for extracting fundamental data-structures from the documents + * **interpreter**: a interpreter of the pdf-standard commands + * **converters**: contains high-level functions for conversion of the fundamental pdf structures to other formats * **imagewriter**: contains a simple implementation for converting PDF Image Streams to png/bmp/img formats - * **layouts**: contains an abstraction layer for the data-structures found in the pdf -## extractors -Contains a high-level function for functional extraction of pdf-assets +## parser + +WIP + +## interpreter +a interpreter of the pdf-standard commands ### Example ```py -from pdfmajor.extractor import extract_items_from_pdf +from pdfmajor.interpreter import PageInterpreter -for page in extract_items_from_pdf('path/to/your/file.pdf'): - print('page-start', page) +for page in PDFInterpreter("/path/to/pdf.pdf"): + print("page start", page.page_num) for item in page: - print(' ', item) - print('page-end', page) - + print(" >", item) + print("page end", page.page_num) ``` -### extract_items_from_pdf(input_file_path,maxpages,password,caching,check_extractable,pagenos,debug_level) -> generator[LTPage] +### interpreter.PDFInterpreter This generator-function yields individual pages which contain their respected items. #### Arguments -- `input_file_path`: [str](str) +- `input_file_path`: [str](#) +- `preload`: [bool](#) defaults to False - `maxpages`: [int](#) defaults to 0 - `password`: [str](#) defaults to None - `caching`: [bool](#) defaults to True @@ -37,39 +40,57 @@ This generator-function yields individual pages which contain their respected it - `pagenos`: [List[int]](#) defaults to None - `debug_level`: [logging.levels](#https://docs.python.org/3/library/logging.html#levels) defaults logging.WARNING -#### Return Value -This function returns a generator that makes `LTPage` +#### Yield Value +This function returns a generator that yields [PDFInterpreter](#interpreterpageinterpreter). -## interpreter - WIP +### interpreter.PageInterpreter + +This generator-function-class yields individual [layout items](#layout-items). +### Layout Items + +All layout items extend the `LTItem` class. There are two kinds of layout items: + +- LTComponent: extends the base `LTItem` class, this class will have additional values such as boundary boxes, height and width +- LTContainer: extends the `LTComponent` class, this class is used to contain elements of the pdf that would have child elements. Iterating on this element will output its child elements. + + +#### Layout Containers +All of these classes extend the LTContainer class. + +- LTXObject: a layout item containing other additional layout items +- LTCharBlock: a layout item containing LTChars + +#### Layout Components +All of these classes extend the LTComponent class. + +- LTChar: an individual character +- LTCurves: represents a collection of svg-paths (available under `self.paths`) +- LTImage: a component containing information regarding an image ## converters -Contains high-level classes for conversion of the fundamental pdf structures to other formats. This library includes 4 high-level conversion classes: +Contains high-level functions for conversion of the fundamental pdf structures to other formats. This library includes 4 high-level conversion cases: -- HTMLConverter -- JSONConverter -- XMLConverter -- TextConverter +- HTML +- JSON +- XML +- Text -These classes all extend the [PDFConverter](#converters.PDFConverter). To use them simply call the static method [parse_file](#converterspdfconverterparse_fileinput_file-output_file-image_folder_path-codec-maxpages-password-caching-check_extractable-pagenos). +These formats are all generated using the [PageInterpreter](#PDFInterpreter). To use them simply call the static method [parse_file](#converterspdfconverter). ### Example ```py -from pdfmajor.converters import HTMLConverter +from pdfmajor.converters import convert_file -input_file = open('path/to/input/file.pdf', 'rb') -output_file = open('path/to/output/file.html', 'wb') -HTMLConverter.parse_file( - input_file, - output_file +convert_file( + "path/to/input/file.pdf", + "path/to/output/file.html", + out_type="html" ) -input_file.close() -output_file.close() ``` -### converters.convert_file(input_file, output_file, image_folder_path, codec, maxpages, password, caching, check_extractable, pagenos, out_type) +### converters.convert_file A high-level abstraction for the conversion classes. @@ -84,76 +105,6 @@ A high-level abstraction for the conversion classes. - `pagenos`: [List[int]](#) defaults to None - `out_type`: [str](#) defaults to 'html' -### Example (converters.convert_file) - -```py -from pdfmajor.converters import convert_file - -input_file = open('path/to/input/file.pdf', 'rb') -output_file = open('path/to/output/file.html', 'wb') -convert_file( - input_file, - output_file -) -input_file.close() -output_file.close() -``` - -### converters.PDFConverter - -Extends `layouts.PDFLayoutAnalyzer`. - -#### converters.PDFConverter.\_\_init\_\_(rsrcmgr, outfp, imagewriter, codec, pageno) - -- `rsrcmgr`: [PDFResourceManager](#interpreter.PDFResourceManager) -- `outfp`: [TextIOWrapper](https://docs.python.org/3/library/io.html#io.TextIOWrapper) -- `imagewriter`: [ImageWriter](#imagewriter.ImageWriter) -- `codec`: [str](#) defaults to 'utf-8' -- `pageno`: [int](#) defaults to 1 - -initialization function for the class - -#### converters.PDFConverter.parse_file(input_file, output_file, image_folder_path, codec, maxpages, password, caching, check_extractable, pagenos) - -returns the output file [TextIOWrapper](https://docs.python.org/3/library/io.html#io.TextIOWrapper). - -- `input_file`: [TextIOWrapper](https://docs.python.org/3/library/io.html#io.TextIOWrapper) -- `output_file`: [TextIOWrapper](https://docs.python.org/3/library/io.html#io.TextIOWrapper) -- `image_folder_path`: [str](#) defaults to None -- `codec`: [str](#) defaults to 'utf-8' -- `maxpages`: [int](#) defaults to 0 -- `password`: [str](#) defaults to None -- `caching`: [bool](#) defaults to True -- `check_extractable`: [bool](#) defaults to True -- `pagenos`: [List[int]](#) defaults to None - ## imagewriter WIP - -## layouts - -WIP: this documentation set is not yet complete. - -### Layout Items - -All layout items extend the `LTItem` class. There are two kinds of layout items: - -- LTComponent: extends the base `LTItem` class, this class will have additional values such as boundary boxes, height and width -- LTContainer: extends the `LTComponent` class, this class is used to contain elements of the pdf that would have child elements. Iterating on this element will output its child elements. - - -#### Layout Containers -All of these classes extend the LTContainer class. - -- LTPage: a layout item containing all elements of the given page. -- LTFigure: a layout item containing LTCurves and LTImages -- LTCharBlock: a layout item containing LTChars - -#### Layout Components -All of these classes extend the LTComponent class. - -- LTChar: an individual character -- LTCurves: represents a collection of svg-paths (available under `self.paths`) -- LTImage: a component containing information regarding an image - diff --git a/doc-src/index.md b/doc-src/index.md index 6773938..f3bbb96 100644 --- a/doc-src/index.md +++ b/doc-src/index.md @@ -11,9 +11,7 @@ -PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. Additionally, this version intends to be more up to date with recent versions of python and make liberal use of the typing capabilities of python for improved future maintainability. - -The ultimate goal is to have a functional-programming with typing fork of the original library. +PDF-Major is a fork of `PDFMiner.six`. It is meant to be a more light-weight implementation that makes fewer assumptions on the data. The ultimate goal of this project is to provide a simpler, faster and more functional library that both casual and low-level developers could build upon. ## Documentation @@ -21,34 +19,32 @@ See documentation [here](https://asosnovsky.github.io/pdfmajor/). ## Basic Example ```py -from pdfmajor.extractor import extract_items_from_pdf +from pdfmajor.interpreter import PageInterpreter -for page in extract_items_from_pdf('path/to/your/file.pdf'): - print('page-start', page) +for page in PDFInterpreter("/path/to/pdf.pdf"): + print("page start", page.page_num) for item in page: - print(' ', item) - print('page-end', page) + print(" >", item) + print("page end", page.page_num) ``` ## Why Another Fork? -`PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. This versions intends to make use of the latest improvement in the python language while remaining fairly simple to develop with. - +`PDFMiner` was designed to run in python 2.7>, `PDFMiner.six` was written to bring cross-version support for python 2 and 3. However, after attempting to build and extend upon `PDFMiner.six` I found it rather difficult to extend and identify additional details regarding the items encoded in the pdf. This library will attempt to expose as much information as possible to end-users, without having them solely rely on to-xml or to-json exports. ## Features * Functional extraction method based on generators - * Written entirely in Python. * Parse, analyze, and convert PDF documents. * PDF-1.7 specification support. (well, almost) + * Font-Color extraction + * Shape fill and stroke color extraction * CJK languages and vertical writing scripts support. * Various font types (Type1, TrueType, Type3, and CID) support. * Basic encryption (RC4) support. - * Font-Color extraction - * Shape fill and stroke color extraction -**Note**: We took out the layout-analysis process in this version (there is no more LTTextHorizontal or LTTextVertical). While the mathematics behind the grouping process was sound, the coupling of the layout-analysis process with the parsing and interpretation process produced unfriendly-code. This feature could be brought back by extending the `pdfmajor.layouts.PDFLayoutAnalyzer` class, but at the current time is not supported. +**Note**: We took out the layout-analysis process in this version (there is no more LTTextHorizontal or LTTextVertical). While the mathematics behind the grouping process was sound, the coupling of the layout-analysis process with the parsing and interpretation process produced unfriendly-code. This feature could be brought back by running your own implementation of it on the `pdfmajor.interpreter.PageInterpreter` class, but at the current time is not supported. ## How to Install diff --git a/pdfmajor/converters/JSONConverter.py b/pdfmajor/converters/JSONConverter.py deleted file mode 100644 index 8031de5..0000000 --- a/pdfmajor/converters/JSONConverter.py +++ /dev/null @@ -1,183 +0,0 @@ -import logging - -from typing import Dict, Union - -from io import BytesIO -from contextlib import contextmanager - -from ..layouts import LTPage -from ..layouts import LTItem -from ..layouts import LTCurve -from ..layouts import LTLine -from ..layouts import LTRect -from ..layouts import LTFigure -from ..layouts import LTImage -from ..layouts import LTChar, LTCharBlock -from ..utils import enc - -from .PDFConverter import PDFConverter -from ..interpreter.PDFGraphicState import PDFGraphicStateColor - -log = logging.getLogger(__name__) - -def get_color(col: PDFGraphicStateColor): - color_type, color_val = col.get_color() - if color_type is not None: - if color_type == 'rgb': - return f"rgb({','.join(map(str, color_val))})" - elif color_type == 'cmyk': - return f"cmyk({','.join(map(str, color_val))})" - elif color_type == 'gray': - return f'gray({color_val})' - return "" - -class JSONConverter(PDFConverter): - def __init__(self, - rsrcmgr, - outfp: BytesIO, - codec: str='utf-8', - pageno: int=1, - imagewriter=None, - ): - PDFConverter.__init__(self, rsrcmgr, outfp, imagewriter=imagewriter, codec=codec, pageno=pageno) - self.__last_levels_deep:int = None - self.__levels_deep = 0 - self.write_header() - - @property - def _levels_deep(self): return self.__levels_deep - - @_levels_deep.setter - def _levels_deep(self, val: int): - self.__last_levels_deep = self.__levels_deep - self.__levels_deep = val - - def write(self, text: str): - if self.__last_levels_deep == self._levels_deep: - text = ',' + text - self.write_raw(text) - - def write_raw(self, text): - if self.codec: - text = text.encode(self.codec) - self.outfp.write(text) - self._levels_deep = self._levels_deep - - def write_header(self): - self.write("{") - self._levels_deep += 1 - if self.codec: - self.write('"encoding": "%s"' % self.codec) - self.write('"pages": [') - self._levels_deep += 1 - - def write_footer(self): - self._levels_deep -= 1 - self.write(']') - self._levels_deep -= 1 - self.write('}') - - @contextmanager - def place_elm_with_child(self, tag_name: str, attr_dict: Dict[str, str] = {}): - attrs = ", ".join([ - f'"{name}": "{value}"' - for name, value in attr_dict.items() - ]) - self.write('{ "type": "%s", "attrs": { %s } , "children": [' % (tag_name, attrs) ) - self._levels_deep += 1 - yield - self._levels_deep -= 1 - self.write("]}") - - def place_elm_close(self, tag_name: str, attr_dict: Dict[str, str] = {}): - attrs = ", ".join([ - f'"{name}": "{value}"' - for name, value in attr_dict.items() - ]) - self.write('{ "type": "%s", "attrs": { %s } }' % (tag_name, attrs) ) - - def place_image(self, item: LTImage): - name = item.name - - if self.imagewriter is not None: - name = self.imagewriter.export_image(item) - - self.place_elm_close('img', { - 'src': enc(name, None), - "x0": item.x0, - "x1": item.x1, - "y0": item.y0, - "y1": item.y1, - }) - - def render_curve(self, item: LTCurve): - attr = { - "x0": item.x0, - "x1": item.x1, - "y0": item.y0, - "y1": item.y1, - "stroke-color": get_color(item.stroke), - "fill-color": get_color(item.fill), - } - if isinstance(item, LTRect): - return self.place_elm_close('rect', attr) - elif isinstance(item, LTLine): - return self.place_elm_close('line', attr) - - with self.place_elm_with_child('curve', attr): - for path in item.paths: - with self.place_elm_with_child('path', { "type": path.method.name }): - for point in path.points: - self.place_elm_close('point', { 'x': point.x, 'y': point.y }) - - def receive_layout(self, ltpage: LTPage): - def render(item: LTItem): - if isinstance(item, LTPage): - with self.place_elm_with_child('page', { - "width": item.width, - "height": item.height - }): - for child in item: - render(child) - elif isinstance(item, LTCurve): - self.render_curve(item) - elif isinstance(item, LTFigure): - with self.place_elm_with_child('figure', { - "x0": item.x0, - "x1": item.x1, - "y0": item.y0, - "y1": item.y1, - }): - for child in item: - render(child) - elif isinstance(item, LTImage): - self.place_image(item) - elif isinstance(item, LTChar): - self.place_elm_close('char', { - 'font-size': item.size, - 'font-family': item.fontname, - "x0": item.x0, - "x1": item.x1, - "y0": item.y0, - "y1": item.y1, - "stroke-color": get_color(item.graphicstate.scolor), - "fill-color": get_color(item.graphicstate.ncolor), - "text": item.get_text() - }) - elif isinstance(item, LTCharBlock): - self.place_elm_close('char-block', { - 'font-size': item.size, - 'font-family': item.fontname, - "x0": item.x0, - "x1": item.x1, - "y0": item.y0, - "y1": item.y1, - "stroke-color": get_color(item.graphicstate.scolor), - "fill-color": get_color(item.graphicstate.ncolor), - "text": item.get_text() - }) - render(ltpage) - - def close(self): - self.write_footer() - \ No newline at end of file diff --git a/pdfmajor/converters/PDFConverter.py b/pdfmajor/converters/PDFConverter.py deleted file mode 100644 index 01bbbd0..0000000 --- a/pdfmajor/converters/PDFConverter.py +++ /dev/null @@ -1,87 +0,0 @@ -import re -import os - -from typing import List -from io import BytesIO, StringIO, TextIOWrapper - -from tqdm import tqdm - -from ..layouts import PDFLayoutAnalyzer -from ..interpreter.PDFResourceManager import PDFResourceManager -from ..interpreter.PDFPage import PDFPage -from ..interpreter.PDFPageInterpreter import PDFPageInterpreter -from ..imagewriter import ImageWriter -from ..utils import set_log_level, get_logger, logging - -log = get_logger(__name__) - -## PDFConverter -## -class PDFConverter(PDFLayoutAnalyzer): - - @classmethod - def parse_file(cls, - input_file: TextIOWrapper, - output_file: TextIOWrapper, - image_folder_path: str = None, - codec: str = 'utf-8', - maxpages: int = 0, - password: str = None, - caching: bool = True, - check_extractable: bool = True, - pagenos: List[int] = None, - dont_export_images: bool = False, - debug_level: int = logging.WARNING, - ) -> TextIOWrapper: - - set_log_level(debug_level) - - if dont_export_images: - image_folder_path = None - - if not dont_export_images and image_folder_path is None and output_file.name is not None: - image_folder_path = os.path.dirname(output_file.name) - - log.debug("Creating resources....") - rsrcmgr = PDFResourceManager(caching=caching) - imagewriter=ImageWriter(image_folder_path) - device = cls( - rsrcmgr, - output_file, - codec=codec, - imagewriter=imagewriter - ) - interpreter = PDFPageInterpreter(rsrcmgr, device) - log.debug("Reading pages....") - pages = PDFPage.get_pages(input_file, pagenos=pagenos, maxpages=maxpages, password=password, caching=caching, check_extractable=check_extractable) - - page_waiter = tqdm(enumerate(pages), disable=debug_level > logging.INFO) - for _, page in page_waiter: - page_waiter.set_description("Processing pdf...") - interpreter.process_page(page) - page_waiter.close() - device.close() - return output_file - - def __init__(self, rsrcmgr: PDFResourceManager, outfp: TextIOWrapper, imagewriter: ImageWriter, codec: str='utf-8', pageno: int=1): - PDFLayoutAnalyzer.__init__(self, rsrcmgr, pageno=pageno) - self.imagewriter = imagewriter - self.outfp = outfp - self.codec = codec - if hasattr(self.outfp, 'mode'): - if 'b' in self.outfp.mode: - self.outfp_binary = True - else: - self.outfp_binary = False - else: - if isinstance(self.outfp, BytesIO): - self.outfp_binary = True - elif isinstance(self.outfp, StringIO): - self.outfp_binary = False - else: - try: - self.outfp.write(u"é") - self.outfp_binary = False - except TypeError: - self.outfp_binary = True - return diff --git a/pdfmajor/converters/TextConverter.py b/pdfmajor/converters/TextConverter.py deleted file mode 100644 index 15172c1..0000000 --- a/pdfmajor/converters/TextConverter.py +++ /dev/null @@ -1,52 +0,0 @@ -import logging - -from typing import Dict - -from contextlib import contextmanager - -from ..layouts import LTPage -from ..layouts import LTItem -from ..layouts import LTCurve -from ..layouts import LTLine -from ..layouts import LTRect -from ..layouts import LTFigure -from ..layouts import LTImage -from ..layouts import LTChar, LTCharBlock -from ..interpreter.PDFGraphicState import PDFGraphicStateColor -from ..utils import enc - -from .PDFConverter import PDFConverter - -log = logging.getLogger(__name__) - -class TextConverter(PDFConverter): - def __init__(self, - rsrcmgr, - outfp, - codec='utf-8', - pageno=1, - imagewriter=None, - ): - PDFConverter.__init__(self, rsrcmgr, outfp, imagewriter=imagewriter, codec=codec, pageno=pageno) - - def write(self, text: str): - if self.codec: - text = text.encode(self.codec) - self.outfp.write(text) - - def place_text(self, char: LTChar): - self.write(char.get_text()) - - def receive_layout(self, ltpage: LTPage): - def render(item: LTItem): - if isinstance(item, LTPage): - self.write(f'\n-----[Page {self.pageno-1}]-----\n') - for child in item: - render(child) - self.write(f'\n-----[Page {self.pageno-1}]-----\n') - elif isinstance(item, LTFigure): - for child in item: - render(child) - elif isinstance(item, LTChar) or isinstance(item, LTCharBlock): - self.place_text(item) - render(ltpage) diff --git a/pdfmajor/converters/__init__.py b/pdfmajor/converters/__init__.py index 7a702f3..517f838 100644 --- a/pdfmajor/converters/__init__.py +++ b/pdfmajor/converters/__init__.py @@ -7,10 +7,7 @@ from typing import List from .html import convert_to_html from .xml import convert_to_xml from .json import convert_to_json -# from .HTMLConverter import HTMLConverter -# from .JSONConverter import JSONConverter -# from .XMLConverter import XMLConverter -# from .TextConverter import TextConverter +from .text import convert_to_text from ..utils import logging def convert_file( @@ -78,18 +75,18 @@ def convert_file( dont_export_images=dont_export_images, debug_level=debug_level, ) - # elif out_type == 'text': - # return TextConverter.parse_file( - # input_file=input_file, - # output_file=output_file, - # image_folder_path=image_folder_path, - # codec=codec, - # maxpages=maxpages, - # password=password, - # caching=caching, - # check_extractable=check_extractable, - # pagenos=pagenos, - # dont_export_images=dont_export_images, - # debug_level=debug_level, - # ) + elif out_type == 'text': + return convert_to_text( + input_file_path=input_file, + output_file_path=output_file, + image_folder_path=image_folder_path, + codec=codec, + maxpages=maxpages, + password=password, + caching=caching, + check_extractable=check_extractable, + pagenos=pagenos, + dont_export_images=dont_export_images, + debug_level=debug_level, + ) else: raise Exception("Please specify out_type as 'html' or 'xml' or 'json' or 'text'") \ No newline at end of file diff --git a/pdfmajor/converters/text.py b/pdfmajor/converters/text.py new file mode 100644 index 0000000..39eae3a --- /dev/null +++ b/pdfmajor/converters/text.py @@ -0,0 +1,39 @@ +from typing import List + +from ..interpreter import PDFInterpreter, PageInterpreter, logging +from ..interpreter import LTCharBlock, LTXObject + + +def convert_to_text( + input_file_path: str, + output_file_path: str, + image_folder_path: str = None, + dont_export_images: bool = False, + codec: str = 'utf-8', + maxpages: int = 0, + password: str = None, + caching: bool = True, + check_extractable: bool = True, + pagenos: List[int] = None, + debug_level: int = logging.WARNING, +): + intepreter = PDFInterpreter(input_file_path, + maxpages=maxpages, + password=password, + caching=caching, + check_extractable=check_extractable, + pagenos=pagenos, + debug_level=debug_level + ) + with open(output_file_path, 'wb') as outfp: + def process_container(container: LTXObject): + for item in container: + if isinstance(item, LTCharBlock): + for char in item: + outfp.write(char.get_text().encode(codec)) + elif isinstance(item, LTXObject): + process_container(item) + for page in intepreter: + outfp.write(f"========== [ page {page.page_num} ] ==========\n".encode(codec)) + process_container(page) + outfp.write('\n'.encode(codec)) diff --git a/tests/test_extraction.py b/tests/test_conversions.py similarity index 97% rename from tests/test_extraction.py rename to tests/test_conversions.py index dedeae3..bfa1466 100755 --- a/tests/test_extraction.py +++ b/tests/test_conversions.py @@ -12,7 +12,7 @@ from tqdm import tqdm def rel_path(*arg: List[str]) -> str: return os.path.normpath(os.path.join(os.path.dirname(__file__), *arg)) -EXTS = ['json','xml','html'] +EXTS = ['text', 'json','xml','html'] INPUT_FOLDER = rel_path("./samples/pdf") OUTPUT_FOLDER = rel_path(f"./samples/output/")