Bug 100244 - CRASH: Pivot table seems to cause massive memory leak
Summary: CRASH: Pivot table seems to cause massive memory leak
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
Inherited From OOo
Hardware: All All
: high major
Assignee: Not Assigned
URL:
Whiteboard:
Keywords: haveBacktrace, perf
Depends on:
Blocks: Pivot-Table Memory Crash
  Show dependency treegraph
 
Reported: 2016-06-06 21:06 UTC by Bob
Modified: 2024-03-30 20:29 UTC (History)
11 users (show)

See Also:
Crash report or crash signature:


Attachments
ods with pivot table that causes memory leak (475.26 KB, application/vnd.oasis.opendocument.spreadsheet)
2016-06-06 21:06 UTC, Bob
Details
Simplified smaller version of the original document (209.92 KB, application/vnd.oasis.opendocument.spreadsheet)
2016-10-09 16:15 UTC, Dennis Francis
Details
bt at random with debug symbols (7.35 KB, text/plain)
2017-08-12 22:55 UTC, Julien Nabet
Details
perf flamegraph with simplified example (66.97 KB, application/x-bzip)
2019-09-24 19:13 UTC, Julien Nabet
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Bob 2016-06-06 21:06:46 UTC
Created attachment 125520 [details]
ods with pivot table that causes memory leak

Refreshing the pivot table in the attached file causes LO memory consumption to spike.  Similar results between Windows 10 and Ubuntu 14.04 on the 5.1.3.2 release.

Warning:  Be prepared to kill LO when you refresh this pivot table!  If you have a system with LOTS of memory, it will eventually refresh - for me, resident memory reaches 0.05t with virtual over 52g (on linux "top").

A strange/temporary "fix" is to "Edit Layout.." and remove column "item_num" *prior* to Refreshing pivot table.  OR remove the "Data Fields" where a "Difference From" display option is set.

Note:  The source range is only 8300 rows by 12 columns, so not "huge" by any means.

My guess is that its some combination of "Row fields" and the "Difference From" that I'm using in the "Data Fields" that is causing the issue.

-Bob
Comment 1 raal 2016-06-07 09:52:33 UTC
Version: 5.2.0.0.alpha1+; win7  crash with "bad allocation" error
Comment 2 Katarina Behrens (Inactive) 2016-06-10 10:39:34 UTC
Is this a regression? i.e. did it ever work reasonably and only got worse in recent releases?
Comment 3 Bob 2016-06-10 20:24:09 UTC
This is the first time I've experienced this problem - and I've never tried using the "Difference From" display option before now.

I did test it using OpenOffice 3.2 on Ubuntu 14.04, and it behaved badly as well.
Comment 4 Julien Nabet 2016-06-19 08:32:28 UTC
On pc Debian x86-64 with master sources updated yesterday, I got an hang.

I noticed a lot of these before the hang:
warn:sc:4374:1:sc/source/core/data/dptabsrc.cxx:2696: ScDPMember::GetItemData: what data? nDim 15, mnDataId 2830
Comment 5 Dennis Francis 2016-10-09 16:13:43 UTC
I am working on this one and have some updates.

I have created a reduced row and simpler version of the original document(attached) such that it cause large memory usage on refresh of pivot table but fit in my system's RAM (16GB) even when calc run with valgrind-memcheck. It uses about lots of memory, but valgrind's memcheck tool shows no big leaks (all are "definitely lost ones are in KB's").

valgrid's massif tool reports that large number of allocations happen at 
ScDPResultDimension::AddMember(ScDPParentDimData const&) (dptabres.cxx:3961)

However I observed that the high memory usage after pivot table refresh does not go away even after I close the document (with calc still running with no other document). After some code hunting I suspect this behavior is due to holding of ScDPSource object inside ScDPObject object via a uno Reference which may be shared with other objects which do not get de-allocated while closing the document having pivot tables. It might only be released after calc is closed. This would explain the valgrind-memcheck's result.

I will next try to trace the reference count changes of ScDPSource object via gdb.

Calc version - latest master built 7th Oct 2016.
OS : Fedora 24 64 bit.
Comment 6 Dennis Francis 2016-10-09 16:15:31 UTC
Created attachment 127895 [details]
Simplified smaller version of the original document
Comment 7 Julien Nabet 2017-06-05 14:30:56 UTC
Dennis: any update here?
Comment 8 Dennis Francis 2017-06-05 14:57:30 UTC
Sorry, I did not proceed further. I will take a look again soon. Till then I am changing the status to NEW and have reset the assignee, so others have a chance to work on this.
Comment 9 Xisco Faulí 2017-06-19 10:02:48 UTC
it crashes for me when I click on cust_name pivot table

Version: 6.0.0.0.alpha0+
Build ID: 9e4502f0e393d2bc2810488b3ebb0a5c23038436
CPU threads: 1; OS: Windows 6.1; UI render: default; 
TinderBox: Win-x86@42, Branch:master, Time: 2017-06-16_08:52:00
Locale: es-ES (es_ES); Calc: group

In

Version: 6.0.0.0.alpha0+
Build ID: 08f6f9dded1b142b858c455da03319abac691655
CPU Threads: 4; OS Version: Linux 4.8; UI Render: default; VCL: gtk2; 
Locale: ca-ES (ca_ES.UTF-8); Calc: group

it hangs
Comment 10 Julien Nabet 2017-08-12 22:55:39 UTC
Created attachment 135505 [details]
bt at random with debug symbols

On pc Debian x86-64 with master sources updated yesterday, it hangs when clicking on cust_name with the second attachment.

I attached a bt at random
Comment 11 QA Administrators 2018-09-20 02:51:07 UTC Comment hidden (obsolete)
Comment 12 Roman Kuznetsov 2018-09-20 12:02:31 UTC
still repro CRASH without any crashreport in 

Version: 6.2.0.0.alpha0+
Build ID: ace6bbf3da9ae27aca87865b6be887a3aed341fc
CPU threads: 4; OS: Windows 6.1; UI render: default; 
TinderBox: Win-x86@42, Branch:master, Time: 2018-09-20_05:45:56
Locale: ru-RU (ru_RU); Calc: threaded

@Xisco: why isn't it critical?
Comment 13 Xisco Faulí 2018-09-20 18:36:31 UTC
> @Xisco: why isn't it critical?

It's an old one, inherit from OOo times...
Comment 14 Julien Nabet 2019-04-21 08:42:00 UTC
On pc Debian x86-64 with master sources updated today, I gave it a new try with initial Bob's file.

I noticed these logs:
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 0
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 1
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 2
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 3
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 4
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 5
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 6
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 7
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 8
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 9
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 10
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 11
warn:sc.core:19920:19920:sc/source/core/data/dptabsrc.cxx:2615: ScDPMember::GetItemData: what data? nDim 13, mnDataId 12
etc.

+ this bt at random:
#0  0x00007fffddc4f410 in ScDPResultMember::InitFrom(std::__debug::vector<ScDPDimension*, std::allocator<ScDPDimension*> > const&, std::__debug::vector<ScDPLevel*, std::allocator<ScDPLevel*> > const&, unsigned long, ScDPInitState&, bool) (this=0x5556094292b0, ppDim=std::__debug::vector of length 5, capacity 8 = {...}, ppLev=std::__debug::vector of length 5, capacity 8 = {...}, nPos=4, rInitState=..., bInitChild=true)
    at /home/julien/lo/libreoffice/sc/source/core/data/dptabres.cxx:1077
#1  0x00007fffddc556f9 in ScDPResultDimension::InitFrom(std::__debug::vector<ScDPDimension*, std::allocator<ScDPDimension*> > const&, std::__debug::vector<ScDPLevel*, std::allocator<ScDPLevel*> > const&, unsigned long, ScDPInitState&, bool)
    (this=0x5556093d5770, ppDim=std::__debug::vector of length 5, capacity 8 = {...}, ppLev=std::__debug::vector of length 5, capacity 8 = {...}, nPos=3, rInitState=..., bInitChild=true)
    at /home/julien/lo/libreoffice/sc/source/core/data/dptabres.cxx:2846
#2  0x00007fffddc4f444 in ScDPResultMember::InitFrom(std::__debug::vector<ScDPDimension*, std::allocator<ScDPDimension*> > const&, std::__debug::vector<ScDPLevel*, std::allocator<ScDPLevel*> > const&, unsigned long, ScDPInitState&, bool) (this=0x5556093d5660, ppDim=std::__debug::vector of length 5, capacity 8 = {...}, ppLev=std::__debug::vector of length 5, capacity 8 = {...}, nPos=3, rInitState=..., bInitChild=true)
    at /home/julien/lo/libreoffice/sc/source/core/data/dptabres.cxx:1077
#3  0x00007fffddc556f9 in ScDPResultDimension::InitFrom(std::__debug::vector<ScDPDimension*, std::allocator<ScDPDimension*> > const&, std::__debug::vector<ScDPLevel*, std::allocator<ScDPLevel*> > const&, unsigned long, ScDPInitState&, bool)
    (this=0x555605ae9c00, ppDim=std::__debug::vector of length 5, capacity 8 = {...}, ppLev=std::__debug::vector of length 5, capacity 8 = {...}, nPos=2, rInitState=..., bInitChild=true)
    at /home/julien/lo/libreoffice/sc/source/core/data/dptabres.cxx:2846
#4  0x00007fffddc4f444 in ScDPResultMember::InitFrom(std::__debug::vector<ScDPDimension*, std::allocator<ScDPDimension*> > const&, std::__debug::vector<ScDPLevel*, std::allocator<ScDPLevel*> > const&, unsigned long, ScDPInitState&, bool) (this=0x555605ae9af0, ppDim=std::__debug::vector of length 5, capacity 8 = {...}, ppLev=std::__debug::vector of length 5, capacity 8 = {...}, nPos=2, rInitState=..., bInitChild=true)
    at /home/julien/lo/libreoffice/sc/source/core/data/dptabres.cxx:1077
#5  0x00007fffddc556f9 in ScDPResultDimension::InitFrom(std::__debug::vector<ScDPDimension*, std::allocator<ScDPDimension*> > const&, std::__debug::vector<ScDPLevel*, std::allocator<ScDPLevel*> > const&, unsigned long, ScDPInitState&, bool)
    (this=0x5555ec0d1390, ppDim=std::__debug::vector of length 5, capacity 8 = {...}, ppLev=std::__debug::vector of length 5, capacity 8 = {...}, nPos=1, rInitState=..., bInitChild=true)
    at /home/julien/lo/libreoffice/sc/source/core/data/dptabres.cxx:2846
...
etc.
Comment 15 Oliver Brinzing 2019-09-23 17:01:15 UTC
still reproducible with:

Version: 6.4.0.0.alpha0+ (x64)
Build ID: 71ef762f21ada8c25aad2183065478171e985e8c
CPU threads: 4; OS: Windows 10.0; UI render: default; VCL: win; 
Locale: de-DE (de_DE); UI-Language: en-US
Calc: threaded
Comment 16 Xisco Faulí 2019-09-24 07:43:43 UTC
@Julien, is it possible to have a perf chart here ;-) ?
Comment 17 Julien Nabet 2019-09-24 08:29:26 UTC
(In reply to Xisco Faulí from comment #16)
> @Julien, is it possible to have a perf chart here ;-) ?
No pb.
I can also retrieve a Valgrind trace, perhaps as for Flamegraph with enable-symbols build instead of enable-dbgutil build.
Comment 18 Julien Nabet 2019-09-24 19:13:25 UTC
Created attachment 154453 [details]
perf flamegraph with simplified example

On pc Debian x86-64 with master sources updated today + enable-symbols, I retrieved a Flamegraph
Comment 19 Xisco Faulí 2019-09-25 11:19:02 UTC
Hi Noel,
I thought you might be interested in this issue...
Comment 20 Roman Kuznetsov 2020-12-25 16:38:41 UTC
the memory leak is still here (over 5Gb) in

Version: 7.2.0.0.alpha0+ (x64)
Build ID: ad9e04321df25824d2288a2ef1f4275f070f1cf7
CPU threads: 4; OS: Windows 6.1 Service Pack 1 Build 7601; UI render: Skia/Raster; VCL: win
Locale: ru-RU (ru_RU); UI: ru-RU
Calc: CL

but I didn't get any crashes, possibly because I use 64 bit OS and LO
Comment 21 Roman Kuznetsov 2022-06-19 21:47:44 UTC
(In reply to Roman Kuznetsov from comment #20)
> the memory leak is still here (over 5Gb) in
> 
> Version: 7.2.0.0.alpha0+ (x64)
> Build ID: ad9e04321df25824d2288a2ef1f4275f070f1cf7
> CPU threads: 4; OS: Windows 6.1 Service Pack 1 Build 7601; UI render:
> Skia/Raster; VCL: win
> Locale: ru-RU (ru_RU); UI: ru-RU
> Calc: CL
> 
> but I didn't get any crashes, possibly because I use 64 bit OS and LO

Memory leak still here in

Version: 7.5.0.0.alpha0+ / LibreOffice Community
Build ID: e4d23c27288b99c3ed3cfa332ff308b31c01f97d
CPU threads: 4; OS: Linux 5.14; UI render: default; VCL: gtk3
Locale: ru-RU (ru_RU.UTF-8); UI: en-US
Calc: threaded Jumbo
Comment 22 Pavel Kysilka 2024-03-30 10:10:59 UTC
I tested this bug today and the app is consuming about 260MB of RAM.

Version: 24.8.0.0.alpha0+ (X86_64) / LibreOffice Community
Build ID: 13250e1aa589453534d113442da1ac8a2cbb71b9
CPU threads: 128; OS: Linux 6.1; UI render: default; VCL: gtk3
Locale: cs-CZ (cs_CZ.UTF-8); UI: en-US
Calc: threaded
Comment 23 Roman Kuznetsov 2024-03-30 18:26:14 UTC
(In reply to Pavel Kysilka from comment #22)
> I tested this bug today and the app is consuming about 260MB of RAM.
> 
> Version: 24.8.0.0.alpha0+ (X86_64) / LibreOffice Community
> Build ID: 13250e1aa589453534d113442da1ac8a2cbb71b9
> CPU threads: 128; OS: Linux 6.1; UI render: default; VCL: gtk3
> Locale: cs-CZ (cs_CZ.UTF-8); UI: en-US
> Calc: threaded

Pavel did you try just update the pivot table in the file?

the memory leak is still here (over 11 Gb and my Windows 10 just died instead just kill LO process!)

Version: 24.8.0.0.alpha0+ (X86_64) / LibreOffice Community
Build ID: 2146e66d8df2b7b6a2dd868e886cae76aaf7f48b
CPU threads: 16; OS: Windows 10.0 Build 19045; UI render: Skia/Vulkan; VCL: win
Locale: ru-RU (ru_RU); UI: ru-RU
Calc: CL threaded
Comment 24 Pavel Kysilka 2024-03-30 20:29:37 UTC
Roman, you're right. After the refreshing pivot table, Calc consumes 61GB of RAM.

There is a similar error with pivot tables.

https://bugs.documentfoundation.org/show_bug.cgi?id=126710