What does it mean to "load into memory"?
I watched some videos online where they were talking about reducing processing reports times, and they constantly say "you don't want to load a huge file into memory, instead read line by line."
Can I have an example of what loading a file (text file for simplicity) into memory, and an example of handling the file NOT loading it into memory.
What exactly are these benefits they talk about besides processing times being reduced?
Best How To :
When they refer to reading something "into memory" it is simply a way of saying that you are reading it and storing it in a variable (which stores it in memory).
Use ReadAllLines to read the entire file into memory:
Dim readText() As String = File.ReadAllLines(path)
See File.ReadAllLines Method (String)
Use a StreamReader to read a file line by line:
Dim sr As StreamReader = New StreamReader(path)
Do While sr.Peek() >= 0
See StreamReader.ReadLine Method
As the comments above mention, the worst performance implications is that it will simply crash if you try to read too large of a file. Whether you can save a millisecond or two here or there by reading line-by-line, it depends on a number of factors (size of file, disk latency, etc.) In general you don't want to be using 100 MB of memory if you don't need to be, so much of the time it's not really about performance, but about not hogging system resources unnecessarily.
If you are processing each line, it's also usually easier to code if you read line-by-line.