We're seeing many of the following emails. We are using postgresql, not oracle.
Any ideas to the cause. We've increased the memory to 72 gigs or ram and 16 cores.
I would think that would have to be enough resources. It patches about 160 hosts.
Please note that I have put in '###' etc, so, as a reference that is to our host that is running 2.10 spacewalk.
our repo server
Wed, Apr 13, 11:06 PM (9 hours ago)
to me
Exception reported from repo
Time: Wed Apr 13 23:06:22 2022
Exception type <class 'spacewalk.server.rhnSQL.sql_base.SQLConnectError'>
Request object information:
URI: /XMLRPC
Remote Host: ###protected host name###
Server Name: ###our repo server###:443
Headers passed in:
Accept-Encoding: identity
CONTENT_LENGTH: 2166
CONTENT_TYPE: text/xml
CONTEXT_DOCUMENT_ROOT: /var/www/html
CONTEXT_PREFIX:
DOCUMENT_ROOT: /var/www/html
GATEWAY_INTERFACE: CGI/1.1
HTTP_ACCEPT_ENCODING: identity
HTTP_HOST: ###our repo server###
HTTP_USER_AGENT: rhn.rpclib.py/2.10.72.10.7-1.el71.el7
HTTP_X_CLIENT_VERSION: 1
HTTP_X_INFO: RPC Processor (C) Red Hat, Inc (version 2.10.72.10.7-1.el71.el7)
HTTP_X_RHN_CLIENT_CAPABILITY: osad.ping(1)=1,scap.xccdf_eval(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,osad.rhn_check(1)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1,osad.ping(1)=1,scap.xccdf_eval(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,osad.rhn_check(1)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1
HTTP_X_RHN_TRANSPORT_CAPABILITY: follow-redirects=3
HTTP_X_TRANSPORT_INFO: Extended Capabilities Transport (C) Red Hat, Inc (version 2.10.72.10.7-1.el71.el7)
HTTP_X_UP2DATE_VERSION: 2.10.10-1.0.4.el7
Host: repo
PATH_INFO:
QUERY_STRING:
REMOTE_ADDR: Private IP
REMOTE_PORT: 48150
REQUEST_METHOD: POST
REQUEST_SCHEME: https
REQUEST_URI: /XMLRPC
SCRIPT_FILENAME: /usr/share/rhn/wsgi/xmlrpc.py
SCRIPT_NAME: /XMLRPC
SCRIPT_URI: https://repo/XMLRPC
SCRIPT_URL: /XMLRPC
SERVER_ADDR: private ip space
SERVER_ADMIN: root@localhost
SERVER_NAME:repo
SERVER_PORT: 443
SERVER_PROTOCOL: HTTP/1.1
SERVER_SIGNATURE:
SERVER_SOFTWARE: Apache
SSL_TLS_SNI: repo
UNIQUE_ID: YledvmFMN9cDkOUAT9S8dwAAA-8
User-Agent: rhn.rpclib.py/2.10.72.10.7-1.el71.el7
X-Client-Version: 1
X-Info: RPC Processor (C) Red Hat, Inc (version 2.10.72.10.7-1.el71.el7)
X-RHN-Client-Capability: osad.ping(1)=1,scap.xccdf_eval(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,osad.rhn_check(1)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1,osad.ping(1)=1,scap.xccdf_eval(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,osad.rhn_check(1)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1
X-RHN-Transport-Capability: follow-redirects=3
X-Transport-Info: Extended Capabilities Transport (C) Red Hat, Inc (version 2.10.72.10.7-1.el71.el7)
X-Up2Date-Version: 2.10.10-1.0.4.el7
mod_wsgi.application_group: ###our repo server|/xmlrpc
mod_wsgi.callable_object: application
mod_wsgi.enable_sendfile: 0
mod_wsgi.handler_script:
mod_wsgi.input_chunked: 0
mod_wsgi.listener_host:
mod_wsgi.listener_port: 443
mod_wsgi.process_group:
mod_wsgi.queue_start: 1649909182432708
mod_wsgi.request_handler: wsgi-script
mod_wsgi.script_reloading: 1
mod_wsgi.version: (3, 4)
wsgi.errors: <mod_wsgi.Log object at 0x7f963cbe46f0>
wsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f963a5ea918>
wsgi.input: <mod_wsgi.Input object at 0x7f963cbcaeb0>
wsgi.multiprocess: True
wsgi.multithread: False
wsgi.run_once: False
wsgi.url_scheme: https
wsgi.version: (1, 0)
Exception Handler Information
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/spacewalk/server/apacheHandler.py", line 87, in headerParserHandler
rhnSQL.initDB()
File "/usr/lib/python2.7/site-packages/spacewalk/server/rhnSQL/__init__.py", line 158, in initDB
raise_with_tb(e, sys.exc_info()[2])
File "/usr/lib/python2.7/site-packages/spacewalk/server/rhnSQL/__init__.py", line 145, in initDB
__init__DB(backend, host, port, username, password, database, sslmode, sslrootcert)
File "/usr/lib/python2.7/site-packages/spacewalk/server/rhnSQL/__init__.py", line 57, in __init__DB
__DB.connect()
File "/usr/lib/python2.7/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 194, in connect
return self.connect(reconnect=reconnect - 1)
File "/usr/lib/python2.7/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 199, in connect
"All attempts to connect to the database failed"), sys.exc_info()[2])
File "/usr/lib/python2.7/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 184, in connect
self.dbh = psycopg2.connect(" ".join("%s=%s" % (k, re.escape(str(v))) for k, v in dsndata.items()))
File "/usr/lib64/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect
conn = _connect(dsn, connection_factory=connection_factory, async=async)
SQLConnectError: (None, None, 'rhnschema', 'All attempts to connect to the database failed')
Local variables by frame
Frame connect in /usr/lib64/python2.7/site-packages/psycopg2/__init__.py at line 164
database = <type 'NoneType'> None
items = <type 'list'> []
connection_factory = <type 'NoneType'> None
dsn = <type 'str'> password=<CENSORED!> dbname=rhnschema user=rhnuser
host = <type 'NoneType'> None
user = <type 'NoneType'> None
kwargs = <type 'dict'> {}
async = <type 'bool'> False
password = <type 'NoneType'> None
port = <type 'NoneType'> None
cursor_factory = <type 'NoneType'> None
Frame connect in /usr/lib/python2.7/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py at line 199
dsndata = <type 'dict'> {'password': '<CENSORED!>', 'dbname': 'rhnschema', 'user': 'rhnuser'}
self = <type 'instance'> <spacewalk.server.rhnSQL.driver_postgresql.Database instance at 0x7f9635394710>
e = <class 'psycopg2.OperationalError'> FATAL: sorry, too many clients already
reconnect = \<type 'int'> 0
Frame connect in /usr/lib/python2.7/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py at line 194
dsndata = <type 'dict'> {'password': '<CENSORED!>', 'dbname': 'rhnschema', 'user': 'rhnuser'}
self = <type 'instance'> <spacewalk.server.rhnSQL.driver_postgresql.Database instance at 0x7f9635394710>
e = <class 'psycopg2.OperationalError'> FATAL: sorry, too many clients already
reconnect = \<type 'int'> 1
Frame __init__DB in /usr/lib/python2.7/site-packages/spacewalk/server/rhnSQL/__init__.py at line 57
username = <type 'str'> rhnuser
db_class = <type 'classobj'> spacewalk.server.rhnSQL.driver_postgresql.Database
database = <type 'str'> rhnschema
sslrootcert = <type 'NoneType'> None
host = <type 'NoneType'> None
sslmode = <type 'NoneType'> None
password = <type 'str'> <CENSORED!>
port = <type 'NoneType'> None
backend = <type 'str'> postgresql
Frame initDB in /usr/lib/python2.7/site-packages/spacewalk/server/rhnSQL/__init__.py at line 158
username = <type 'str'> rhnuser
e = <class 'spacewalk.server.rhnSQL.sql_base.SQLConnectError'> (None, None, 'rhnschema', 'All attempts to connect to the database failed')
database = <type 'str'> rhnschema
sslrootcert = <type 'NoneType'> None
initsecond = <type 'bool'> False
host = <type 'NoneType'> None
sslmode = <type 'NoneType'> None
password = <type 'str'> <CENSORED!>
port = <type 'NoneType'> None
backend = <type 'str'> postgresql
Frame headerParserHandler in /usr/lib/python2.7/site-packages/spacewalk/server/apacheHandler.py at line 89
self = <type 'instance'> <spacewalk.server.apacheHandler.apacheHandler instance at 0x7f9638788320>
req = <type 'instance'> <wsgi.wsgiRequest.WsgiRequest instance at 0x7f963cbad5f0>
options = <type 'dict'> {'RHNComponentType': 'server.xmlrpc', 'SERVER': 'xmlrpc'}
ret = <type 'int'> 0
Frame handle in /usr/share/rhn/wsgi/wsgiHandler.py at line 36
replacements = <type 'dict'> {'Xml': 'XML', 'Actualuri': 'ActualURI', 'Md5Sum': 'MD5sum', '_': '-', 'Rhn': 'RHN'}
component_type = <type 'str'> server.xmlrpc
start_response = <type 'builtin_function_or_method'> <built-in method start_response of mod_wsgi.Adapter object at 0x7f963a5ea918>
k = <type 'str'> Rhn
req = <type 'instance'> <wsgi.wsgiRequest.WsgiRequest instance at 0x7f963cbad5f0>
parseServ = <type 'instancemethod'> <bound method apacheHandler.headerParserHandler of <spacewalk.server.apacheHandler.apacheHandler instance at 0x7f9638788320>>
server = <type 'str'> xmlrpc
environ = <type 'dict'> {'mod_wsgi.listener_port': '443', 'X-RHN-Client-Capability': 'osad.ping(1)=1,scap.xccdf_eval(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,osad.rhn_check(1)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1,osad.ping(1)=1,scap.xccdf_eval(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,osad.rhn_check(1)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1', 'CONTEXT_DOCUMENT_ROOT': '/var/www/html', 'SERVER_SOFTWARE': 'Apache', 'SCRIPT_NAME': '/XMLRPC', 'HTTP_X_TRANSPORT_INFO': 'Extended Capabilities Transport (C) Red Hat, Inc
(version 2.10.72.10.7-1.el71.el7)', 'mod_wsgi.handler_script': '', 'SERVER_SIGNATURE': '', 'REQUEST_METHOD': 'POST', 'PATH_INFO': '', 'SERVER_PROTOCOL': 'HTTP/1.1', 'QUERY_STRING': '', 'X-Info': 'RPC Processor (C) Red Hat, Inc (version 2.10.72.10.7-1.el71.el7)', 'User-Agent': 'rhn.rpclib.py/2.10.72.10.7-1.el71.el7', 'SSL_TLS_SNI': '###our repo server', 'CONTENT_LENGTH': '2166', 'HTTP_USER_AGENT': 'rhn.rpclib.py/2.10.72.10.7-1.el71.el7', 'HTTP_X_UP2DATE_VERSION': '2.10.10-1.0.4.el7', 'SERVER_NAME': '###our repo server', 'REMOTE_ADDR': '172.16.3.41', 'mod_wsgi.queue_start': '1649909182432708', 'mod_wsgi.request_handler': 'wsgi-script', 'mod_wsgi.version': (3, 4), 'wsgi.url_scheme': 'https', 'Host': '###our repo server', 'X-RHN-Transport-Capability': 'follow-redirects=3', 'mod_wsgi.callable_object': 'application', 'SERVER_PORT': '443', 'wsgi.multiprocess': True, 'mod_wsgi.input_chunked': '0', 'SERVER_ADDR': '172.16.3.15', 'DOCUMENT_ROOT': '/var/www/html', 'mod_wsgi.process_group': '', 'Ac
cept-Encoding': 'identity', 'HTTP_X_RHN_CLIENT_CAPABILITY': 'osad.ping(1)=1,scap.xccdf_eval(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,osad.rhn_check(1)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1,osad.ping(1)=1,scap.xccdf_eval(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,osad.rhn_check(1)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1', 'SCRIPT_FILENAME': '/usr/share/rhn/wsgi/xmlrpc.py', 'SERVER_ADMIN': 'root@localhost', 'X-Up2Date-Version': '2.10.10-1.0.4.el7', 'X-Transport-Info': 'Extended Capabilities Transport (C) Red Hat, Inc (version 2.1
0.72.10.7-1.el71.el7)', 'SCRIPT_URI': 'https://###our repo server/XMLRPC', 'HTTP_X_CLIENT_VERSION': '1', 'wsgi.input': <mod_wsgi.Input object at 0x7f963cbcaeb0>, 'HTTP_HOST': '### our repo server ###', 'SCRIPT_URL': '/XMLRPC', 'wsgi.multithread': False, 'mod_wsgi.enable_sendfile': '0', 'REQUEST_URI': '/XMLRPC', 'X-Client-Version': '1', 'wsgi.version': (1, 0), 'GATEWAY_INTERFACE': 'CGI/1.1', 'wsgi.run_once': False, 'wsgi.errors': <mod_wsgi.Log object at 0x7f963cbe46f0>, 'REMOTE_PORT': '48150', 'mod_wsgi.listener_host': '', 'REQUEST_SCHEME': 'https', 'CONTEXT_PREFIX': '', 'HTTP_X_RHN_TRANSPORT_CAPABILITY': 'follow-redirects=3', 'CONTENT_TYPE': 'text/xml', 'mod_wsgi.application_group': '###our repo server### |/xmlrpc', 'mod_wsgi.script_reloading': '1', 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f963a5ea918>, 'HTTP_ACCEPT_ENCODING': 'identity', 'UNIQUE_ID': 'YledvmFMN9cDkOUAT9S8dwAAA-8', 'HTTP_X_INFO': 'RPC Processor (C) Red Hat, Inc (version 2.10.72.10.7-1.
el71.el7)'}
key = <type 'str'> HTTP_X_INFO
v = <type 'str'> RHN
servertype = <type 'str'> spacewalk.server.apacheServer
new_key = <type 'str'> X-Info
Frame application in /usr/share/rhn/wsgi/xmlrpc.py at line 22
start_response = <type 'builtin_function_or_method'> <built-in method start_response of mod_wsgi.Adapter object at 0x7f963a5ea918>
environ = <type 'dict'> {'mod_wsgi.listener_port': '443', 'X-RHN-Client-Capability': 'osad.ping(1)=1,scap.xccdf_eval(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,osad.rhn_check(1)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1,osad.ping(1)=1,scap.xccdf_eval(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,osad.rhn_check(1)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1', 'CONTEXT_DOCUMENT_ROOT': '/var/www/html', 'SERVER_SOFTWARE': 'Apache', 'SCRIPT_NAME': '/XMLRPC', 'HTTP_X_TRANSPORT_INFO': 'Extended Capabilities Transport (C) Red Hat, Inc
(version 2.10.72.10.7-1.el71.el7)', 'mod_wsgi.handler_script': '', 'SERVER_SIGNATURE': '', 'REQUEST_METHOD': 'POST', 'PATH_INFO': '', 'SERVER_PROTOCOL': 'HTTP/1.1', 'QUERY_STRING': '', 'X-Info': 'RPC Processor (C) Red Hat, Inc (version 2.10.72.10.7-1.el71.el7)', 'User-Agent': 'rhn.rpclib.py/2.10.72.10.7-1.el71.el7', 'SSL_TLS_SNI': '###our repo server ###', 'CONTENT_LENGTH': '2166', 'HTTP_USER_AGENT': 'rhn.rpclib.py/2.10.72.10.7-1.el71.el7', 'HTTP_X_UP2DATE_VERSION': '2.10.10-1.0.4.el7', 'SERVER_NAME': 'repo', 'REMOTE_ADDR': '172.16.3.41', 'mod_wsgi.queue_start': '1649909182432708', 'mod_wsgi.request_handler': 'wsgi-script', 'mod_wsgi.version': (3, 4), 'wsgi.url_scheme': 'https', 'Host': 'repo', 'X-RHN-Transport-Capability': 'follow-redirects=3', 'mod_wsgi.callable_object': 'application', 'SERVER_PORT': '443', 'wsgi.multiprocess': True, 'mod_wsgi.input_chunked': '0', 'SERVER_ADDR': '172.16.3.15', 'DOCUMENT_ROOT': '/var/www/html', 'mod_wsgi.process_group': '', 'Ac
cept-Encoding': 'identity', 'HTTP_X_RHN_CLIENT_CAPABILITY': 'osad.ping(1)=1,scap.xccdf_eval(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,osad.rhn_check(1)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1,osad.ping(1)=1,scap.xccdf_eval(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,osad.rhn_check(1)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1', 'SCRIPT_FILENAME': '/usr/share/rhn/wsgi/xmlrpc.py', 'SERVER_ADMIN': 'root@localhost', 'X-Up2Date-Version': '2.10.10-1.0.4.el7', 'X-Transport-Info': 'Extended Capabilities Transport (C) Red Hat, Inc (version 2.1
0.72.10.7-1.el71.el7)', 'SCRIPT_URI': 'https://repo/XMLRPC', 'HTTP_X_CLIENT_VERSION': '1', 'wsgi.input': <mod_wsgi.Input object at 0x7f963cbcaeb0>, 'HTTP_HOST': 'repo', 'SCRIPT_URL': '/XMLRPC', 'wsgi.multithread': False, 'mod_wsgi.enable_sendfile': '0', 'REQUEST_URI': '/XMLRPC', 'X-Client-Version': '1', 'wsgi.version': (1, 0), 'GATEWAY_INTERFACE': 'CGI/1.1', 'wsgi.run_once': False, 'wsgi.errors': <mod_wsgi.Log object at 0x7f963cbe46f0>, 'REMOTE_PORT': '48150', 'mod_wsgi.listener_host': '', 'REQUEST_SCHEME': 'https', 'CONTEXT_PREFIX': '', 'HTTP_X_RHN_TRANSPORT_CAPABILITY': 'follow-redirects=3', 'CONTENT_TYPE': 'text/xml', 'mod_wsgi.application_group': 'repo|/xmlrpc', 'mod_wsgi.script_reloading': '1', 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f963a5ea918>, 'HTTP_ACCEPT_ENCODING': 'identity', 'UNIQUE_ID': 'YledvmFMN9cDkOUAT9S8dwAAA-8', 'HTTP_X_INFO': 'RPC Processor (C) Red Hat, Inc (version 2.10.72.10.7-1.
el71.el7)'}
Environment for PID=1444 on exception:
LANG = C
NOTIFY_SOCKET = /run/systemd/notify
PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin