提问人:neverMind 提问时间:11/17/2023 更新时间:11/17/2023 访问量:58
Python CSV 编写器:解析器无法将行正确写入 CSV
Python CSV Writer: parser does not write rows correctly into CSV
问:
我的用例如下:从 oracle 的表中读取并将其写入 10 个文件。 我在每个文件中附加了几个 fecthes,以避免超出我的限制。 到目前为止,天哪。 今天,它开始在包含 \n、\n\n、\r\n 和特殊字符 suri points、big hiphens、Idk 等特定字符组合的 1/10 文件上失败。似乎 CSV Writer 在写入文件时没有正确解析或引用,因为我总是得到相同的结果:应该是值的空列等,就像之前在某个地方捕获的换行符一样。 我也尝试过打印行,它们很好,所以这不是获取的问题。
def write_to_gzip(rows, header, is_header, local_file):
start_gzip_ts = datetime.now()
with gzip.open(local_file, mode='a', newline='', encoding='utf-8', compresslevel=c.GZIP_COMPRESS_LEVEL) as gz:
# line terminator will be handled automatically based on the platform
writer = csv.writer(gz, delimiter=',', quoting=csv.QUOTE_MINIMAL, escapechar='\\', quotechar='"' )
if is_header:
writer.writerow(header)
else:
# Write the rows using csv.writer
writer.writerows(rows)
elapsed_ts = datetime.now() - start_gzip_ts
return elapsed_ts
def fetch_and_write(cursor:cx_Oracle.Cursor, array_sizes, header, filename:str):
rows_size = 0
nr_fetches = len(array_sizes)
write_ts = timedelta()
fetch_ts = timedelta()
try:
# Reset cursor fetch size to a maximum of 5000
cursor.arraysize = min(cursor.arraysize, c.ARRAY_SIZE)
# Write header to the file
write_ts += write_to_gzip([], header, True, filename)
for size in array_sizes:
init_fetch_ts = datetime.now()
# Fetch rows
rows = cursor.fetchmany(numRows=size)
if not rows:
break # No more rows to fetch
rows_size += len(rows)
end_fetch_ts = datetime.now()
fetch_ts += end_fetch_ts - init_fetch_ts
# Write rows to the file
#@WIP
write_ts += write_to_gzip(rows, header, False, filename)
# Log total number of records fetched
logger.info(f"[{filename}] Total records fetched: {rows_size}")
except Exception as e:
print(f"Error during fetch_and_write: {e}")
# Handle the exception as needed
return rows_size, nr_fetches, fetch_ts, write_ts
此代码允许我写入 csv 文件并同时压缩,避免首先有 2 个文件,因为我在这个 unix 服务器中的磁盘空间有一些限制。
我已经写了十几次write_to_gzip,但我无法使用 CSV 编写器或其他模块,因为转义的过程令人难以置信地烦人并且......慢。
我在这里错过了什么?
答: 暂无答案
评论