Optimize the memory consumption in XYCurve
For the setup in #185 (import 2.9GB of data and visualize it) the memory consumption is around 8GB. The biggest contributors are:
-
4GB allocated in QVector<QPointF> in XYCurve::recalcLogicalPoints() for m_logicalPoints, etc. - this seems to be very expensive in terms of memory and we need to reduce here the number of points we hold in memory:
-
2.1GB allocated in QVector<double> in Spreadsheet - this is where the allocations for the actual data in Columns are triggered. For 250Mil doubles (50Mil rows, 5 columns) we would expect to have 50e6*5*8B/1024/1024/1024 \approx 1.86GB memory allocated. We need to understand why we're consuming more here:
-
further 536.9MB allocated in QVector<double> in Spreadsheet caused by setRowCount(actualRows) in Spreadsheet::prepareImport(). This seems to be duplicating, at least partially, the allocations done later in this function in the vector->resize() calls:
-
536.9MB allocated in QVector<int> - this is to hold the integers in the first column that has the type "Integer" and to hold integers (same number) for the vertical header in the spreadsheet model. So, we have here 268.4MB twice:
-
268.4MB allocated in QVector<bool> in XYCurvePrivate::recalcLogicalPoints() for m_pointVisible - here we should switch to std::bitset or to std::vector<bool>:
-
1.5GB allocated in std::vector in XYCurve::recalcLogicalPoints() for validPointsIndicesLogical:
Other contributions can be ignored for now.