Problem: If you are working with millions of record in a CSV it is difficult to handle large sized file.
Solution: You can split the file into multiple smaller files according to the number of records you want in one file. Python helps to make it easy and faster way to split the file in microseconds.
For Example: Save this code in testsplit.py file:
import sys if(len(sys.argv) < 2): #if no file location is provided then show error message and exit print('Provide File Location') sys.exit() fil = sys.argv[1] csvfilename = open(fil, 'r').readlines() #store header values header = csvfilename[0] #remove header from list csvfilename.pop(0) file = 1 #Number of lines to be written in new file record_per_file = 50 for j in range(len(csvfilename)): if j % record_per_file == 0: write_file = csvfilename[j:j+record_per_file] #adding header at the start of the write_file write_file.insert(0, header) #write in file open(str(fil)+ str(file) + '.csv', 'w+').writelines(write_file) file += 1
You can run this file on the command line using the following command:
python testsplit.py test.csv
If test.csv is the name of the file you want to split then split files are named as test.csv1.csv, test.csv2.csv and so on.
Where do I put the .py file in order to use “python testsplit…” in terminal??????
We can put .py file in any directory that has write permissions and you also need to keep .csv file in the same directory.
headers are missing in the files after the split. can you please help me on this
We have updated the code in which the header has been included now in all the files which are being created after splitting the large CSV file.
What I was looking for! A simple solution much better than what I found until now googling! Working and thanks!
how to access the csv file from a different directory?