Split Logs into Files with a Certain Number of Entries

Dealing with large log files is often cumbersome. With PowerGREP you can easily split logs into separate files with a specific number of entries per log. E.g. splitlog0.txt would have the entries 0 to 999, splitlog1.txt has entries 1,000 to 1,999, and so on. You can put all the entries from the original log or logs into the target files, or you can extract only those entries that you’re interested in.

  1. Select the log files you want to split in the File Selector.
  2. Start with a fresh action.
  3. Set the action type to “split files”.
  4. In the “file sectioning” list, select “line by line, including line breaks”. This works for logs that use one line for each entry.
  5. Turn on the option “split whole sections”. This makes sure lines will be extracted as a whole into the target files.
  6. If you want all log entries to be in the split files, enter the regular expression . into the search box, and make sure “dot matches newlines” is off. This puts any line that is not blank into the target files. If you want only certain lines, enter a search term or regex that (partially) matches the log entries you want to extract. E.g. search for ^Error to extract only those entries starting with the word “Error”.
  7. In the Target File box, enter something along the lines of c:\logs\splitlog%MATCHNZ:/1000%.txt to specify the target path. The match placeholder %MATCHNZ:/1000% takes the number of the match counting from zero, and divides it by 1000. This results in 0 for the first 1,000 matches, 1 for the second 1,000 matches, and so on. If you want the first log file to be number one, use %MATCHNZ:/1000+1%. Match placeholders use integer arithmetic that is calculated strictly from left to right. If you expect to have more than 10 but less than, say, 100 log files, you can pad the number in the target file name to have two digits by using %MATCHNZ:/1000:2Z% or %MATCHNZ:/1000+1:2Z% as the placeholder. 2Z means to pad with zeros to make the placeholder have at least two digits.
  8. Set “between collected text” set to “nothing”. Since we’re collecting whole lines including line breaks, there’s no need to add more delimiters.
  9. Set the backup file options as you like them.
  10. Click the Quick Split button to split the file. Use this button instead of Split Files. Otherwise PowerGREP will waste a lot of time and memory to display your entire log files on the Results panel.

Splitting files does not delete the original files. It may overwrite original files if the Target File for one or more search matches is a file that is searched through. When splitting files PowerGREP does not write the final target files until the action has completed. Overwriting source files won’t alter the search matches.

This action is available in the PowerGREP5.pgl library as “Logs: Split Logs into Files with a Certain Number of Entries”.

Recombining Log Files

The above example can also be used to recombine log files. Say you previously split your logs into files with 1,000 entries and deleted the original logs. Now want the logs to be split into files with 2,500 entries each.

Follow the exact same steps as above. In the first step, select all the previously split log files. In step 7, use %MATCHNZ:/2500% as the placeholder.

In PowerGREP, a “split files” action can put search matches from one file into different target files. It can also put search matches from different files into the same target file. In our example, all matches from old logs 1 and 2 (with 1,000 entries each) are saved into new log number 1. The first 500 matches from old log number 3 are saved into new log number 1, and the remaining 500 are saved into old log number 2.

The “order of collected matches” drop-down list determines the order in which matches (log entries in this case) from different files are written when a “split files” action calculates the same target file path from matches from multiple files. This is important if you want your log entries in the combined file to have the same order as in the original files. If your original log files put the log entries in order if you sort the files alphabetically by name, then choose “sort files Alphabetically A..Z”. If the time stamp on the log files puts the files in the correct order (e.g. the time stamp on each log file is the time the last entry was written) then you can choose “oldest file to newest file”.