Split large CSV into multiple smaller CSV files with Python script

Author - Deepika

Problem: If you are working with millions of record in a CSV itt is difficult to handle large sized file.

Solution: You can split the file into multiple smaller files according to the number of records you want in one file. Python helps to make it easy and faster way to split the file in microseconds.

For Example: Save this code in testsplit.py file:

import sys
csvfilename = open(fil, 'r').readlines()
file = 1
for j in range(len(csvfilename)):
if j % 10000 == 0:
open(str(fil)+ str(file) + '.csv', 'w+').writelines(csvfilename[j:j+10000])
file += 1

you can run this file on the command line using the following command:

python testsplit.py test.csv

If test.csv is the name of the file you want to split then split files are named as test.csv1.csv, test.csv2.csv and so on.

Webner Solutions is a Software Development company focused on developing Insurance Agency Management Systems, Learning Management Systems and Salesforce apps. Contact us at dev@webners.com for your Insurance, eLearning and Salesforce applications.

Leave a Reply

Your email address will not be published. Required fields are marked *