Speed Difference: Reading Large Log Files

When it comes to reading large log files and, for example, extracting error messages, PowerShell can either use the low memory pipeline, or the high memory classical loops. The difference is not just the memory consumption, though, but also the speed.

With the pipeline, you do not need much memory but the script may also be very slow. By using the classic loop, the script produces the same results 10x to 100x faster:

# make sure this file exists, or else
# pick a different text file that is very large
$path = 'C:\Windows\Logs\DISM\dism.log'
# SLOW
# filtering text file via pipeline (low memory usage)
Measure-Command {
  $result = Get-Content -Path $Path | Where-Object { $_ -like '*Error*' }
}

# FAST
# filtering text by first reading in all
# content (high memory usage!) and then
# using a classic loop

Measure-Command {
  $lines = Get-Content -Path $Path -ReadCount 0
  $result = foreach ($line in $lines)
  {
    if ($line -like '*Error*') { $line }
  }
}

ReTweet this Tip!