Split large CSV into multiple smaller CSV files with Python script

Author - Deepika
26.05.2017
|
6 Comments
|

Problem: If you are working with millions of record in a CSV it is difficult to handle large sized file.

Solution: You can split the file into multiple smaller files according to the number of records you want in one file. Python helps to make it easy and faster way to split the file in microseconds.

For Example: Save this code in testsplit.py file:

import sys

if(len(sys.argv) < 2): #if no file location is provided then show error message and exit
    print('Provide File Location')
    sys.exit()

fil = sys.argv[1]
csvfilename = open(fil, 'r').readlines()
#store header values
header = csvfilename[0] 
#remove header from list
csvfilename.pop(0) 
file = 1
#Number of lines to be written in new file
record_per_file = 50

for j in range(len(csvfilename)):
	if j % record_per_file == 0:
		write_file = csvfilename[j:j+record_per_file]
                #adding header at the start of the write_file
                write_file.insert(0, header)
 	 	#write in file
 	        open(str(fil)+ str(file) + '.csv', 'w+').writelines(write_file)
                file += 1

You can run this file on the command line using the following command:

python testsplit.py test.csv

If test.csv is the name of the file you want to split then split files are named as test.csv1.csv, test.csv2.csv and so on.

Webner Solutions is a Software Development company focused on developing Insurance Agency Management Systems, Learning Management Systems and Salesforce apps. Contact us at dev@webners.com for your Insurance, eLearning and Salesforce applications.

6 responses on “Split large CSV into multiple smaller CSV files with Python script

  1. Sylvio says:

    Where do I put the .py file in order to use “python testsplit…” in terminal??????

  2. Webner says:

    We can put .py file in any directory that has write permissions and you also need to keep .csv file in the same directory.

  3. revathi says:

    headers are missing in the files after the split. can you please help me on this

  4. Daniel says:

    What I was looking for! A simple solution much better than what I found until now googling! Working and thanks!

  5. deb says:

    how to access the csv file from a different directory?

Leave a Reply

Your email address will not be published. Required fields are marked *