Parse more CSL vars embedded in CSL JSON note field

[njbart]

It emerges that citeproc-js parses (some) CSL date vars embedded in CSL JSON note fields into their components:

• If they are in EDTF (Level 1) format, citeproc-js parses dates such as YYYY, YYYY-MM, YYYY-MM-DD, YYYY-MM-DD/YYYY-MM-DD in embedded accessed, container, event-date, issued, original-date, submitted vars into their components.

citeproc-js also parses CSL creator vars embedded in CSL JSON note fields:

• It parses creators with single- and two-part fields (using || as a field separator), e.g., {:editor: Doe||John} or {:editor: United Nations}. Multiple creators can be entered by repeating this, e.g., {:editor: Doe||John}{:editor: Roe||Ron}.

It would be great if pandoc-citeproc could support this, too.

[njbart]

Just pinging: Since in Zotero there has been no progress on date ranges at all for many years, it would be great if pandoc-citeproc could parse ISO date ranges of the following forms out of the “note” variable:

{:issued:2015-07-27/2016-03-30}, or {:issued:2015-07/2016-03}, or {:issued:2015/2016}

The other date variables (accessed, container, event-date, original-date, submitted) would of course best be supported, too.

For consistency, pandoc-citeproc should probably parse any date, not just date ranges here. For a simple calendar date, the Zotero Date field is of course sufficient, but for circa dates, seasons etc. in EDTF format it would again be helpful if pandoc-citeproc could parse these.

pandoc-citeproc should continue to overwrite existing variables: For date ranges in particular this would allow having one date (range) in Zotero’s date field for display and sorting purposes (e.g., 2015-07-27/2016-03-30 can be entered in a Zotero Date field, but it is always the start date only that is parsed and exported) and the same date range in the note variable that can be parsed correctly by pandoc-citeproc.

Any content in one of the date variables that cannot be identified as an ISO or (better) EDTF date or date range should of course be treated as a “literal” date.

[njbart]

To clarify further, here’s an example showing the actual and the expected output from pandoc-citeproc -y csl-json-cheater-syntax-1.json.

The CSL JSON format for dates – including the so-called “cheater syntax” that embeds variables inside the note variable – has recently been updated and is documented in https://citeproc-js.readthedocs.io/en/latest/csl-json/markup.html#cheater-syntax-for-odd-fields.

csl-json-cheater-syntax-1.json contains:

[
    {
        "id": "cheater-syntax-braced-entry-1",
        "note": "Foo {:issued: 2015-06-30} bar"
    },
    {
        "id": "cheater-syntax-braced-entry-2",
        "note": "Foo {:issued:2015-06-30/2016-07-31} bar"
    },
    {
        "id": "cheater-syntax-line-entry-1",
        "note": "Foo\nissued: 2015-06-30\nbar"
    },
    {
        "id": "cheater-syntax-line-entry-2",
        "note": "Foo\nissued:2015-06-30/2016-07-31\nbar"
    }
]

actual output:

---
references:
- id: cheater-syntax-braced-entry-1
  type: no-type
  note: 'Foo {:issued: 2015-06-30} bar'

- id: cheater-syntax-braced-entry-2
  type: no-type
  note: Foo {:issued:2015-06-30/2016-07-31} bar

- id: cheater-syntax-line-entry-1
  type: no-type
  note: 'Foo issued: 2015-06-30 bar'

- id: cheater-syntax-line-entry-2
  type: no-type
  note: Foo issued:2015-06-30/2016-07-31 bar
...

expected output:

---
references:
- id: cheater-syntax-braced-entry-1
  type: no-type
  note: 'Foo bar'
  issued:
  - year: '2015'
    month: '6'
    day: '30'

- id: cheater-syntax-braced-entry-2
  type: no-type
  note: Foo bar
  issued:
  - year: '2015'
    month: '6'
    day: '30'
  - year: '2016'
    month: '7'
    day: '31'

- id: cheater-syntax-line-entry-1
  type: no-type
  note: 'Foo bar'
  issued:
  - year: '2015'
    month: '6'
    day: '30'

- id: cheater-syntax-line-entry-2
  type: no-type
  note: Foo bar
  issued:
  - year: '2015'
    month: '6'
    day: '30'
  - year: '2016'
    month: '7'
    day: '31'
...

Again, it’s support for cheater-syntax dates that’s particularly high on my wish list, since there’s currently no other way to get Zotero to export date ranges to CSL JSON via zotxt, for onward processing by pandoc.

Hence:

• pandoc-citeproc already supports the braced-entry syntax in general, but the ability to parse ISO dates from it should be added. Most likely this could just reuse the existing mechanisms used for parsing biblatex dates.
• In addition, support for the line-entry syntax in general should be added. Zotero devs have declared they vastly prefer this syntax, and Zotero’s import mechanisms now increasingly use this format when adding extra variables for which no GUI fields are available, e.g., “DOI” for books, book chapters, or reports.

[njbart]

It seems most of the issues described in this thread have been fixed in the meantime – much appreciated. EDIT: Wrong – it seems pandoc-citeproc wasn’t updated after all; the effect seen is zotxt now using BBT’s cheater syntax parser if BBT is installed.

The sole exception is pandoc-citeproc -y / -j, whose output is still problematic, in particular examples such as note: 'Foo issued: 2015-06-30 bar', which, after the removal of the \n, cannot even be parsed correctly if fed to pandoc-citeproc again. So, at least the newlines should be kept, but, even better, the “cheater” variables should be removed from the note variable and written out as regular variables.