How to split one big text file into multiple files in powershell – update

by john651Jan 28, 2015

Hello this is an updated question to this one powershell.com/…/38721.aspx

A couple months ago BOB has helped me to create this super fast code for splitting one big file into multiple small files (see link above). Unfortunately format of my data has changed so I was forced to make some changes to his code. What has changed:

1. There is no header line in the big text file anymore.

2.Names for small files are now stored in first column.

Below is updated code:

# Split-File.ps1
#
$src = "C:Ephemeralbigfile.csv"
$dstDir = ""C:Ephemeralfiles""

$inData = New-Object -TypeName System.IO.StreamReader -ArgumentList $src

$currentFile = ""
while ($line = $inData.ReadLine())
{
    $newFile = "$(($line -split ",")[0]).csv"
    if ($newFile -ne $currentFile)
    {
        # We're starting on a new file
        if ($currentFile -ne "")
        {
            # Write out contents of current file
            $outData.ToString() | Out-File -FilePath $dstDir$currentFile -Encoding ascii
        }
        # Get ready for a new current file
        $currentFile = $newFile
        $outData = New-Object -TypeName System.Text.StringBuilder

    }
    Write-Verbose "$currentFile,$line"
   [void]$outData.Append("$($line)`r`n")

}
# Write out contents of last file
$outData.ToString() | Out-File -FilePath $dstDir$currentFile -Encoding ascii

Problem with this code is that there is always an extra line at the end of the each split file. I've played a little bit with those "Backtick" parameters (`r`n) but I'm unable to get rid of those extra empty rows at the end of each file.

Also I would like to skip first column(column with the file names) from being copied into small files.

Example:

BIGFILE.txt

BIGFILE.TXT

                                                     #NO HEADER
FILE1,01/20/2015,38.700,39.250,38.580,38.750,170
FILE1,01/21/2015,15.720,16.050,15.720,15.920,156
FILE1,01/22/2015,53.990,54.520,52.540,52.695,197
FILE2,01/15/2015,32.220,32.800,32.060,32.510,566
FILE2,01/16/2015,19.120,19.490,19.050,19.410,194
FILE3,01/20/2015,14.090,14.345,14.060,14.280,132
FILE4,01/22/2015,15.110,15.400,15.060,15.290,223
FILE4,01/23/2015,46.560,47.160,45.640,46.720,144
FILE5,01/16/2015,63.530,63.720,63.330,63.550,127
FILE5,01/17/2015,25.670,25.960,25.620,25.800,211
FILE5,01/18/2015,19.620,19.755,19.580,19.680,148
FILE5,01/19/2015,62.330,63.500,62.200,63.100,653
FILE5,01/20/2015,47.600,48.020,46.930,47.100,302

FILE1.txt

01/20/2015,38.700,39.250,38.580,38.750,170
01/21/2015,15.720,16.050,15.720,15.920,156
01/22/2015,53.990,54.520,52.540,52.695,197

FILE2.txt

01/15/2015,32.220,32.800,32.060,32.510,566
01/16/2015,19.120,19.490,19.050,19.410,194

FILE3.txt

01/20/2015,14.090,14.345,14.060,14.280,132

FILE4.txt

01/22/2015,15.110,15.400,15.060,15.290,223
01/23/2015,46.560,47.160,45.640,46.720,144

FILE5.txt

01/16/2015,63.530,63.720,63.330,63.550,127
01/17/2015,25.670,25.960,25.620,25.800,211
01/18/2015,19.620,19.755,19.580,19.680,148
01/19/2015,62.330,63.500,62.200,63.100,653
01/20/2015,47.600,48.020,46.930,47.100,302





Thank you for your help.

How to split one big text file into multiple files in powershell – update

Tags