Problem: If you are working with millions of record in a CSV itt is difficult to handle large sized file.
Solution: You can split the file into multiple smaller files according to the number of records you want in one file. Python helps to make it easy and faster way to split the file in microseconds.
For Example: Save this code in testsplit.py file:
import sys fil=sys.argv csvfilename = open(fil, 'r').readlines() file = 1 for j in range(len(csvfilename)): if j % 10000 == 0: open(str(fil)+ str(file) + '.csv', 'w+').writelines(csvfilename[j:j+10000]) file += 1
you can run this file on the command line using the following command:
python testsplit.py test.csv
If test.csv is the name of the file you want to split then split files are named as test.csv1.csv, test.csv2.csv and so on.