pyspark log4j2：如何记录完整的异常堆栈跟踪？-解网

问：

我试过了

logger.error('err', e)
logger.error('err', exc_info=e) # syntax for python's logging

>>> 
>>> logger = spark.sparkContext._jvm.org.apache.log4j.LogManager.getLogger('my-logger')
>>> 
>>> try: 1/0
... except Exception as e: logger.error('err', e)
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/home/kash/project1/.venv/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1313, in __call__
  File "/home/kash/project1/.venv/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1283, in _build_args
  File "/home/kash/project1/.venv/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1283, in <listcomp>
  File "/home/kash/project1/.venv/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 298, in get_command_part
AttributeError: 'ZeroDivisionError' object has no attribute '_get_object_id'
>>> 
>>> 
>>> 
>>> try: 1/0
... except Exception as e: logger.error('err', exc_info=e)
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
TypeError: __call__() got an unexpected keyword argument 'exc_info'
>>>

当然，我可以自己转换堆栈跟踪并将其作为字符串传递给 log4j 而不是异常对象。但如果我能避免的话，我不想做这一切。

>>> try: 1/0
... except Exception as e: l.error(f'err {"".join(traceback.TracebackException.from_exception(e).format())}')
... 
23/03/09 11:38:47 ERROR my-logger: err Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero

>>>

Python 异常日志记录 pyspark log4j

>>> 
>>> import traceback
>>> def e2s(ex: Exception):
...     return {''.join(traceback.TracebackException.from_exception(ex).format())}
... 
>>> 
>>> l = spark.sparkContext._jvm.org.apache.log4j.LogManager.getLogger('my_logger')
>>> 
>>> try: 1/0
... except Exception as e: l.error(f'Some exception: {e2s(e)}')
... 
ERROR Some exception: {'Traceback (most recent call last):\n  File "<stdin>", line 1, in <module>\nZeroDivisionError: division by zero\n'}
>>>

pyspark log4j2：如何记录完整的异常堆栈跟踪？

pyspark log4j2: How to log full exception stack trace?

评论

评论