• R/O
  • SSH

execsql: 提交

Default repository for execsql.py


Commit MetaInfo

修訂26c13ecbcedd69a4e8b686d7a015af353eabceef (tree)
時間2022-06-21 13:48:50
作者rdnielsen
Commiterrdnielsen

Log Message

Modified import of Feather and Parquet data to replace 'nan' values in a pandas DataFrame with 'None' values in a Python list.

Change Summary

差異

diff -r bc651354a0e4 -r 26c13ecbcedd doc/source/metacommands.rst
--- a/doc/source/metacommands.rst Sat Jun 18 16:27:48 2022 -0700
+++ b/doc/source/metacommands.rst Mon Jun 20 21:48:50 2022 -0700
@@ -1688,14 +1688,17 @@
16881688 single: Exporting; feather
16891689
16901690 FEATHER
1691- The feather binary file format established by the `Apache Arrow
1691+ The Feather binary file format established by the `Apache Arrow
16921692 <https://arrow.apache.org/>`_ project. The "APPEND" and "DESCRIPTION"
16931693 keywords are ignored when this format is used. Exporting data in
16941694 this format requires that the entire data set be first converted
16951695 to a `pandas <https://pandas.pydata.org/>`_ data frame in memory,
16961696 so there is a system-specific limit to the size of the data set
16971697 that can be exported in this format. The *feather* and *pandas*
1698- libraries must be installed to export data in the feather format.
1698+ libraries must be installed to export data in the Feather format.
1699+ Not all data types that may be present in a database can necessarily
1700+ be exported to a Feather data file, so some data types, like
1701+ timestamps, may have to be converted to character data before export.
16991702 Data exported in feather format cannot be written into a zipfile.
17001703
17011704 .. index::
diff -r bc651354a0e4 -r 26c13ecbcedd execsql/execsql.py
--- a/execsql/execsql.py Sat Jun 18 16:27:48 2022 -0700
+++ b/execsql/execsql.py Mon Jun 20 21:48:50 2022 -0700
@@ -27,12 +27,12 @@
2727 #
2828 # ===============================================================================
2929
30-__version__ = "1.101.1"
31-__vdate = "2022-06-18"
30+__version__ = "1.101.2"
31+__vdate = "2022-06-20"
3232
3333 primary_vno = 1
3434 secondary_vno = 101
35-tertiary_vno = 1
35+tertiary_vno = 2
3636
3737 import os
3838 import os.path
@@ -13854,12 +13854,14 @@
1385413854
1385513855 def import_feather(db, schemaname, tablename, filename, is_new):
1385613856 try:
13857+ import numpy as np
1385713858 import pandas as pd
1385813859 import pyarrow.feather
1385913860 except:
1386013861 raise ErrInfo("exception", exception_msg=exception_desc(), other_msg=u"The pandas and pyarrow Python libraries must be installed to import data from the Feather format.")
1386113862 #df = pyarrow.feather.read_feather(filename)
1386213863 df = pd.read_feather(filename)
13864+ df = df.replace({np.nan: None})
1386313865 hdrs = df.columns.values.tolist()
1386413866 data = df.values.tolist()
1386513867 import_data_table(db, schemaname, tablename, is_new, hdrs, data)
@@ -13867,10 +13869,12 @@
1386713869
1386813870 def import_parquet(db, schemaname, tablename, filename, is_new):
1386913871 try:
13872+ import numpy as np
1387013873 import pandas as pd
1387113874 except:
1387213875 raise ErrInfo("exception", exception_msg=exception_desc(), other_msg=u"The pandas and fastparquet or pyarrow Python libraries must be installed to import data from the Parquet format.")
1387313876 df = pd.read_parquet(filename)
13877+ df = df.replace({np.nan: None})
1387413878 hdrs = df.columns.values.tolist()
1387513879 data = df.values.tolist()
1387613880 import_data_table(db, schemaname, tablename, is_new, hdrs, data)
diff -r bc651354a0e4 -r 26c13ecbcedd setup.py
--- a/setup.py Sat Jun 18 16:27:48 2022 -0700
+++ b/setup.py Mon Jun 20 21:48:50 2022 -0700
@@ -5,7 +5,7 @@
55 long_description = f.read()
66
77 setuptools.setup(name='execsql',
8- version='1.101.1',
8+ version='1.101.2',
99 description="Runs a SQL script against a PostgreSQL, MS-Access, SQLite, MS-SQL-Server, MySQL, MariaDB, Firebird, or Oracle database, or an ODBC DSN. Provides metacommands to import and export data, copy data between databases, conditionally execute SQL and metacommands, and dynamically alter SQL and metacommands with substitution variables. Data can be exported in 18 different formats, including CSV, TSV, ODS, HTML, JSON, LaTeX, and Markdown tables, and using custom templates.",
1010 author='Dreas Nielsen',
1111 author_email='dreas.nielsen@gmail.com',
Show on old repository browser