dump_model giving "JSONDecodeError: Expecting ',' delimiter" for large (250mb+) models

[nquinn1]

Description

I have created a few large lgbm models (250mb+ when saved as txt) and when i run dump_model on them in Python it returns
"JSONDecodeError: Expecting ',' delimiter"

Reproducible example

import lightgbm as lgb

lgb_model = lgb.Booster(model_file = "VeryLargeModel.txt")
lgb.dump_model()

Environment info

python == 3.9.7
lightgbm == 3.3.2

Additional Comments

I have experienced the issue for several months and my other colleagues also get it. I'm not certain if it's a lightgbm issue or a json issue more generally. I've tried a few fixes from stack overflow on similar but not identical issues but nothing works. The models work completely fine, I use them daily and no issue when training etc.

[jameslamb]

Thanks very much for using LightGBM.

Can you please either provide the model file where you've observed this issue, or a sample code using publicly-available data (e.g. the datasets from sklearn.datasets) that reproduces this issue?

Without those specifics, it'll be difficult for maintainers here to work on this problem.

[nquinn1]

Hi,

Thanks for the quick response. I have uploaded a 1GB ish model here https://github.com/nquinn1/LModel/blob/main/model_5.txt
made with random numbers as data. I have tested dump_model on it an get the same error.

[jameslamb]

Excellent, thanks for that! I or another maintainer will try to investigate the issue at some point, but I can't make any guarantees about how soon that will be.

Linking this related (although not identical) issue: #3858