So I have a C++ code that finds the inverse of a square matrix. The inverse is full and it would be bad to keep it all in memory because I am working with hundreds of thousands of columns. My code generates the columns one by one. After finding the inverse, the rest of my code requires the rows one by one. Ideally, I would like to proceed as follows:
Repeat: 1) find a column 2) store it on the disk 3) delete the column from memory.
Repeat: 1) read a row from the disk 2) process it (it is basically the objective function of a linear program) 3) delete the row from memory.
One way of doing that is writing the matrix in the file as it is.. Meaning, I write the first column as an "actual" column in the file, then I write the second next to it (which would be a bit less efficient), and so on. The last column will take O(n^2) to write if I'm not mistaken. Doing this makes reading the rows very easy though. The other way is to write the columns as rows then reading the columns later on.. But again, the last column will take O(n^2) to read. A third approach is to use n files, but opening and closing n files is inefficient.
Any ideas on how to approach such a problem? Maybe I have to use databases (possibly SQL) to make my reading and writing O(1) per entry?