Bug 108673

Summary: FILESAVE XLSX: Copy-pasting cell validation may result in a sheet reference error saved as invalid xlsx content
Product: LibreOffice Reporter: Gabor Kelemen (allotropia) <kelemeng>
Component: CalcAssignee: Serge Krot (CIB) <serge.krot>
Status: VERIFIED FIXED    
Severity: normal CC: aron.budea, erack, himajin100000, ilmari.lauhakangas, libreoffice, samuel.mehrbrodt, thb
Priority: medium Keywords: filter:xlsx
Version: 5.4.0.0.beta2   
Hardware: All   
OS: All   
Whiteboard: target:7.1.0 target:7.0.2 target:6.4.7
Crash report or crash signature: Regression By:
Bug Depends on:    
Bug Blocks: 104839, 108897, 108917    
Attachments: Example source file
Cells with validation pasted to the first sheet
Cells with validation pasted to the first sheet - in XLSX format
Original cell and its pasted version in a new file
Excel 2013 reacts with an error to opening the attached XLSX
Copied cell validation saved by Excel 2013
Copied cell validation saved by Excel 2016

Description Gabor Kelemen (allotropia) 2017-06-21 12:24:23 UTC
Created attachment 134188 [details]
Example source file

Attached file has two sheets, with cell validation in the B1 cells.

When copy-pasting the B1 cell from both sheets to the firsd sheet of a new spreadsheet document, the sheet reference part of the validation source becomes a #REF! error: $Sheet2.$A$1:$A$3 becomes $#REF!.$A$1:$A$3

This may make some sense in itself, as there is no second sheet in the new document. Adding a new sheet and pasting there gives correct results.

The main problem is that the validation source with #REF! error can be saved to both ODS and XLSX format, and in the case of XLSX, Excel 2013 considers it an invalid file.
Comment 1 Gabor Kelemen (allotropia) 2017-06-21 12:27:12 UTC
Created attachment 134189 [details]
Cells with validation pasted to the first sheet

In A2 we can see the #REF! error.
Comment 2 Gabor Kelemen (allotropia) 2017-06-21 12:27:40 UTC
Created attachment 134190 [details]
Cells with validation pasted to the first sheet - in XLSX format
Comment 3 Gabor Kelemen (allotropia) 2017-06-21 12:28:31 UTC
Created attachment 134191 [details]
Original cell and its pasted version in a new file
Comment 4 Gabor Kelemen (allotropia) 2017-06-21 12:29:05 UTC
Created attachment 134192 [details]
Excel 2013 reacts with an error to opening the attached XLSX
Comment 5 Buovjaga 2017-07-01 18:33:04 UTC
Repro.

Win 8.1 32-bit
MSO 2013
LibreOffice Version: 6.0.0.0.alpha0+
Build ID: cac5c9f6081590b0548d3116bc4cd4a2509ec576
CPU threads: 4; OS: Windows 6.29; UI render: default; 
TinderBox: Win-x86@42, Branch:master, Time: 2017-07-01_00:41:48
Locale: fi-FI (fi_FI); Calc: group
Comment 6 QA Administrators 2018-07-04 02:49:26 UTC Comment hidden (obsolete)
Comment 7 Eike Rathke 2020-07-10 10:53:09 UTC
(In reply to Gabor Kelemen from comment #0)
> The main problem is that the validation source with #REF! error can be saved
> to both ODS and XLSX format, and in the case of XLSX, Excel 2013 considers
> it an invalid file.

For ODFF the #REF! is perfectly fine, see
https://docs.oasis-open.org/office/OpenDocument/v1.3/cs01/part4-formula/OpenDocument-v1.3-cs01-part4-formula.html#References

What does Excel save instead for an invalid reference?
Comment 8 NISZ LibreOffice Team 2020-07-13 06:31:38 UTC
(In reply to Eike Rathke from comment #7)
> (In reply to Gabor Kelemen from comment #0)
> > The main problem is that the validation source with #REF! error can be saved
> > to both ODS and XLSX format, and in the case of XLSX, Excel 2013 considers
> > it an invalid file.
> 
> For ODFF the #REF! is perfectly fine, see
> https://docs.oasis-open.org/office/OpenDocument/v1.3/cs01/part4-formula/
> OpenDocument-v1.3-cs01-part4-formula.html#References
> 
> What does Excel save instead for an invalid reference?

My Excel 2013 writes this to xl/worksheets/sheet1.xml when a cell with validation is copied to a new file:

<x14:dataValidation type="list" allowBlank="1" showInputMessage="1" showErrorMessage="1"><x14:formula1><xm:f>[1]Munka2!#REF!</xm:f></x14:formula1><xm:sqref>A1</xm:sqref></x14:dataValidation>

This also happens to be invalid according to Excel 2013 :).

Before saving, the GUI shows the validation condition as:
='[broken-reference-orig.xlsx]Munka2'!#HIV!

While Calc 7.0RC1 writes this:
<dataValidation allowBlank="true" operator="between" showDropDown="false" showErrorMessage="true" showInputMessage="true" sqref="A1" type="list"><formula1>#HIV!!$C$2:$C$4</formula1><formula2>0</formula2></dataValidation>

(#HIV! being Hungarian for #REF!)
Comment 9 NISZ LibreOffice Team 2020-07-13 06:32:32 UTC
Created attachment 162951 [details]
Copied cell validation saved by Excel 2013
Comment 10 NISZ LibreOffice Team 2020-07-13 12:42:47 UTC
Excel 2016 xl/worksheets/sheet1.xml looks like this when a cell with validation is copied to a new file

<x14:dataValidation type="list" operator="equal" allowBlank="1" showErrorMessage="1" xr:uid="{A8E4395C-A788-4CEA-9E70-BD38B4629442}"><x14:formula1><xm:f>'[validation-copypaste.xlsx]Sheet1'!#REF!</xm:f></x14:formula1><xm:sqref>A2</xm:sqref></x14:dataValidation></x14:dataValidations>
Comment 11 NISZ LibreOffice Team 2020-07-13 12:43:36 UTC
Created attachment 162969 [details]
Copied cell validation saved by Excel 2016
Comment 12 Eike Rathke 2020-07-13 13:17:23 UTC
Seeing that in the ODF case
table:condition="of:cell-content-is-in-list([$#REF!.$A$1:.$A$3])"
is written that's also wrong, it should be [#REF!] instead, i.e. not split into particles anymore.

I think we could write a plain #REF! also in the OOXML case, which is what is written also for other reference errors.

(In reply to NISZ LibreOffice Team from comment #8)
> (#HIV! being Hungarian for #REF!)
Really? The translated error keyword is written? Geez.. that's even worse.
Comment 13 Eike Rathke 2020-07-15 19:48:54 UTC
It's getting even worse.. if the original validity (on Sheet2 of a *yet unsaved* document) was Sheet1.A1 or a formula like OFFSET(Sheet1.A2;-1;0) (note relative sheet reference) and copied to new document's Sheet1 it correctly displays #REF!.A1 or OFFSET(#REF!.A2;-1;0) but saved to .xlsx is Sheet1!A1 or OFFSET(Sheet1!A2,-1,0) resulting in a valid but wrong reference when reloading.

Saved to .ods is the correct [#REF!] or OFFSET([#REF!];-1;0)

Funny enough, if in the original bug doc on Sheet2 column A is deleted the validity displays as $Sheet2.$#REF!$1:$#REF!$3 and saving to .xlsx writes an expected <formula1>#REF!</formula1>, so something is already producing the correct string but ignores an invalid sheet reference.
Comment 14 Eike Rathke 2020-07-15 22:22:36 UTC
A hint:
The missing handling and erroneous use of native UI error string
(Hungarian #HIV!) instead of #REF! are in
sc/source/core/tool/compiler.cxx struct ConventionXL_OOX function
makeRefStr() and the ConventionXL_A1::makeRefStr() it is calling and
thereunder. Caveat, the same methods are used for the Excel A1 native UI
formula syntax notation (hence XL_A1).
Comment 15 Commit Notification 2020-09-07 23:20:38 UTC
Serge Krot committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/a3b4831208da615789bd1e2d5660dd130807f504

tdf#108673 XLSX: Don't export invalid sheet references in cell validation

It will be available in 7.1.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 16 Commit Notification 2020-09-09 07:17:33 UTC
Serge Krot committed a patch related to this issue.
It has been pushed to "libreoffice-7-0":

https://git.libreoffice.org/core/commit/03f601d10c8fe8772dcb8542e1d44f2d2e87fbb4

tdf#108673 XLSX: Don't export invalid sheet references in cell validation

It will be available in 7.0.2.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 17 Commit Notification 2020-09-10 10:10:46 UTC
Serge Krot committed a patch related to this issue.
It has been pushed to "libreoffice-6-4":

https://git.libreoffice.org/core/commit/bd416f4a44acfea54afc377b9880863fd969dbe1

tdf#108673 XLSX: Don't export invalid sheet references in cell validation

It will be available in 6.4.7.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.