I developed a python script which imports data to my Postgres DB and it works fine on my local machine but when I transferred it to the server it fails with:
Traceback (most recent call last): File "./importer.py", line 1, in <module> import psycopg2 ModuleNotFoundError: No module named 'psycopg2' here is the python script:
import psycopg2 import glob, os import zipfile import ntpath ## db connection string and objects t_host = "localhost" t_dbname = "*****" t_username = "******" t_password = "******" t_port = 5432 t_schema = "******" t_stageTable = "******" ## working directories inputDir = "./incoming" outputDir = "./processed" ## functions definition def clearDirectory(dir): for csvFile in glob.glob(dir+"/*.csv"): os.remove(csvFile) def stripQuotes(str): return str.strip('"') def uploadFile(file): db_conn = psycopg2.connect(host=t_host, port=t_port, dbname=t_dbname, user=t_username, password=t_password) db_cursor = db_conn.cursor() f_contents = open(file, 'r') for row in f_contents: record = row.rstrip().split(",") record.append('"'+ntpath.basename(file)+'"') insert_query = "INSERT INTO ...") #commit after the whole file is loaded db_conn.commit() #close connection db_cursor.close() db_conn.close() def processFiles(dirIn, dirOut): for file in glob.glob(dirIn+"/*.zip"): #extract files with zipfile.ZipFile(file, 'r') as zip_ref: zip_ref.extractall(dirOut) for file in glob.glob(dirOut+"/*.csv"): uploadFile(file) #move .zip files to processed folder for file in glob.glob(dirIn+"/*.zip"): os.replace(file, dirOut+"/"+ntpath.basename(file)) ################################### ## actual processing ################################### #clean old files first (just to be sure) clearDirectory(outputDir) #loop through zip files in the incoming folder processFiles(inputDir, outputDir) #clean again clearDirectory(outputDir) ################################### I tried to install the missing library psycopg2 with pip3
sudo pip3 install psycopg2
but I am getting this weird error:
Collecting psycopg2 Using cached psycopg2-2.8.6.tar.gz (383 kB) Building wheels for collected packages: psycopg2 Building wheel for psycopg2 (setup.py) ... error ERROR: Command errored out with exit status 1: command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-80qwhbn5/psycopg2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-80qwhbn5/psycopg2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-uwxpzq13 cwd: /tmp/pip-install-80qwhbn5/psycopg2/ Complete output (40 lines): running bdist_wheel running build running build_py creating build creating build/lib.linux-x86_64-3.8 creating build/lib.linux-x86_64-3.8/psycopg2 copying lib/_json.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/extensions.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/_ipaddress.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/extras.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/errorcodes.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/compat.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/sql.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/_range.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/tz.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/_lru_cache.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/errors.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/__init__.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/pool.py -> build/lib.linux-x86_64-3.8/psycopg2 running build_ext building 'psycopg2._psycopg' extension creating build/temp.linux-x86_64-3.8 creating build/temp.linux-x86_64-3.8/psycopg x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPSYCOPG_VERSION=2.8.6 (dt dec pq3 ext lo64) -DPG_VERSION_NUM=120006 -DHAVE_LO64=1 -I/usr/include/python3.8 -I. -I/usr/include/postgresql -I/usr/include/postgresql/12/server -c psycopg/psycopgmodule.c -o build/temp.linux-x86_64-3.8/psycopg/psycopgmodule.o -Wdeclaration-after-statement In file included from psycopg/psycopgmodule.c:28: ./psycopg/psycopg.h:36:10: fatal error: libpq-fe.h: No such file or directory 36 | #include <libpq-fe.h> | ^~~~~~~~~~~~ compilation terminated. It appears you are missing some prerequisite to build the package from source. You may install a binary package by installing 'psycopg2-binary' from PyPI. If you want to install psycopg2 from source, please install the packages required for the build and try again. For further information please check the 'doc/src/install.rst' file (also at <). error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 ---------------------------------------- ERROR: Failed building wheel for psycopg2 Running setup.py clean for psycopg2 Failed to build psycopg2 Installing collected packages: psycopg2 Running setup.py install for psycopg2 ... error ERROR: Command errored out with exit status 1: command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-80qwhbn5/psycopg2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-80qwhbn5/psycopg2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-4g11q7zi/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/psycopg2 cwd: /tmp/pip-install-80qwhbn5/psycopg2/ Complete output (40 lines): running install running build running build_py creating build creating build/lib.linux-x86_64-3.8 creating build/lib.linux-x86_64-3.8/psycopg2 copying lib/_json.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/extensions.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/_ipaddress.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/extras.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/errorcodes.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/compat.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/sql.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/_range.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/tz.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/_lru_cache.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/errors.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/__init__.py -> build/lib.linux-x86_64-3.8/psycopg2 copying lib/pool.py -> build/lib.linux-x86_64-3.8/psycopg2 running build_ext building 'psycopg2._psycopg' extension creating build/temp.linux-x86_64-3.8 creating build/temp.linux-x86_64-3.8/psycopg x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPSYCOPG_VERSION=2.8.6 (dt dec pq3 ext lo64) -DPG_VERSION_NUM=120006 -DHAVE_LO64=1 -I/usr/include/python3.8 -I. -I/usr/include/postgresql -I/usr/include/postgresql/12/server -c psycopg/psycopgmodule.c -o build/temp.linux-x86_64-3.8/psycopg/psycopgmodule.o -Wdeclaration-after-statement In file included from psycopg/psycopgmodule.c:28: ./psycopg/psycopg.h:36:10: fatal error: libpq-fe.h: No such file or directory 36 | #include <libpq-fe.h> | ^~~~~~~~~~~~ compilation terminated. It appears you are missing some prerequisite to build the package from source. You may install a binary package by installing 'psycopg2-binary' from PyPI. If you want to install psycopg2 from source, please install the packages required for the build and try again. For further information please check the 'doc/src/install.rst' file (also at <). error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 ---------------------------------------- ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-80qwhbn5/psycopg2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-80qwhbn5/psycopg2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-4g11q7zi/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/psycopg2 Check the logs for full command output. never saw this before.. any idea what's going on?
I even tried installing the psycopg2-binary .. which was successful .. but still the py script ends up with the same error -> no module ..
EDIT: @Simon Aldrich
sudo apt install libpg-dev ends up with:
Reading package lists... Done Building dependency tree Reading state information... Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: libpq-dev : Depends: libpq5 (= 12.6-0ubuntu0.20.04.1) but 13.2-1.pgdg20.04+1 is to be installed E: Unable to correct problems, you have held broken packages. when I tried to remove libpq5 sudo apt purge libpq5
it says:
Reading package lists... Done Building dependency tree Reading state information... Done The following packages were automatically installed and are no longer required: libapache2-mod-wsgi-py3 libboost-filesystem1.71.0 libboost-thread1.71.0 libllvm10 Use 'sudo apt autoremove' to remove them. The following packages will be REMOVED: libpq5* pgadmin4* pgadmin4-desktop* pgadmin4-server* pgadmin4-web* pgagent* postgresql* postgresql-12* postgresql-client-12* postgresql-contrib* 0 upgraded, 0 newly installed, 10 to remove and 0 not upgraded. I don't want the pgadmin to be removed :(
1 Answer
Sounds like you're missing the libpq-dev package which will have the headers you need.
