Problem
Recently I got a problem with export large amount of data, and send it to the client side.
The detailed problem description shows in the linked page below:
How can I adapt my code to make it compatible to Microsoft Excel?
What's Different
Although, the first answer in the linked page help me to solve the problem of messy code when open the .csv file by excel. But as I commented, it would be a little inconvenient for the user. So I tried to export an .xls file directly.
My Question Is
Because the dataset is quite large, I cannot generate the whole .xls file all at once, maybe it's a good idea to send one line or several lines to the client side per time as I did with the .csv file.
So how can I send the .xls data piece by piece to the client side? or any better recommendations?
I would be really appreciated for your answer!
This is a possible solution using dependencies flask + sql-alchemy + pandas
def export_query(query, file_name = None):
results = db.session.execute(query)
fetched = results.fetchall()
dataframe = pd.DataFrame(fetched)
dataframe.columns = get_query_coloumn_names(query)
base_path = current_app.config['UPLOAD_FOLDER']
workingdocs = base_path + datetime.now().strftime("%Y%m%d%H%M%S") + '/'
if not os.path.exists(workingdocs):
os.makedirs(workingdocs)
if file_name is None:
file_name = workingdocs + str(uuid.uuid4()) + '-' + 'export.xlsx'
else:
file_name = workingdocs + file_name
dataframe.to_excel(file_name)
return file_name
def export_all(q, page_limit, page):
query = db.session.query(...).\
outerjoin(..).\
filter(..).\
order_by(...)
paging_query = query.paginate(page, page_limit, True)
# TODO need to return total to help user know to keep trying paging_query.total
return export_query(paging_query)
@api.route('/export_excel/', methods=['POST'])
@permission_required(Permission.VIEW_REPORT)
def export_excel():
json = request.get_json(silent=False, force=True)
q = ''.join(('%',json['q'],'%'))
page_limit = try_parse_int(json['page_limit'])
page = try_parse_int(json['page'])
file_name = export_all(q, page_limit, page)
response = send_file(file_name)
response.headers["Content-Disposition"] = "attachment; filename=export.xlsx"
return response