[R] Custom objective for multi-class

[albedan]

I'm trying to use a custom objective and metric for a multiclass problem, following @Laurae2 example here: https://github.com/Microsoft/LightGBM/blob/master/R-package/demo/early_stopping.R

Using R 3.5.1 and lightgbm 2.2.2.
So far I managed to implement the eval metric, but objective seems tricky.

I think a trivial example should be useful: maybe just implementing the default multi class logloss with a custom objective in R (in order to be able to compare the results with the default objective).

[BruceZhaoR]

@albedan #1682 maybe helpful 😁

[albedan]

@albedan #1682 maybe helpful grin

Thanks @BruceZhaoR, but that one is about the evaluation metric (and I have no issues with that).
I'm referring to the objective function, i.e. the one that defines the optimization through gradient and hessian

[BruceZhaoR]

multi class logloss with a custom objective :
https://github.com/Microsoft/LightGBM/blob/425503d7e254130eaea4a5ff287f324f5a5cc1ea/R-package/demo/multiclass_custom_objective.R#L37-L55

It's that what you want?

[albedan]

Yes! Somehow I couldn't find it, I'm going to try it soon. Thank you @BruceZhaoR

[albedan]

@BruceZhaoR ,
Actually I think there could be a couple of issues:

1. The example has balanced classes. Let's change it at the beginning of the script:

train <- as.matrix(iris[c(1:30, 51:95, 101:140), ])
test <- as.matrix(iris[c(31:50, 96:100, 141:150), ])

The builtin model starts with a "boost from average"

[LightGBM] [Info] Start training from score -1.343735
[LightGBM] [Info] Start training from score -0.938270
[LightGBM] [Info] Start training from score -1.056053

The custom one does not (and setting the flag does not seem to work).

2. Running the example as it is, the final "identical" function returns False, as the two predictions are different. Actually, reshaping and applying manually a softmax function gets to identical predictions (but I expected the raw ones to be identical, too).

[vnijs]

I'm seeing the same as @albedan. See output below.

@albedan Could you perhaps share/post the code you used to get identical probability predictions? Thanks

> # compare predictions
> identical(preds_builtin, preds_custom)
[1] FALSE
> head(data.frame(preds_builtin, preds_custom))
  preds_builtin preds_custom
1      1.540058     2.638670
2     -2.912387    -1.813775
3     -2.912387    -1.813775
4      1.540058     2.638670
5     -2.912387    -1.813775
6     -2.912387    -1.813775

EDIT: The following produces equivalent results after a bit of rounding

preds_custom <- predict(model_custom, test[, 1:4], rawscore = TRUE, reshape = TRUE)
preds_custom <- exp(preds_custom) / rowSums(exp(preds_custom))

preds_builtin <- predict(model_builtin, test[, 1:4], rawscore = TRUE, reshape = TRUE)
preds_builtin <- exp(preds_builtin) / rowSums(exp(preds_builtin))

# compare predictions
identical(round(preds_builtin, 14), round(preds_custom, 14))

[vnijs]

I don't mean to thread-jack but ... I'm trying to setup a customer objective function for a binary classifier. I tried adapting the cost @ * https://github.com/Microsoft/LightGBM/blob/master/R-package/demo/multiclass_custom_objective.R but not having much luck so far.

Is the result from https://github.com/Microsoft/LightGBM/blob/master/R-package/demo/early_stopping.R supposed to produce the same result as the below?

bst_builtin <- lgb.train(param,
                 dtrain,
                 num_round,
                 valids,
                 objective = "binary",
                 metric = "binary_logloss",
                 early_stopping_round = 3)

FYI An example of a custom objective function for a regression model with lightgbm:

https://towardsdatascience.com/custom-loss-functions-for-gradient-boosting-f79c1b40466d

[grand-home-projects]

@vnijs

Were you ever able to get a working version of the binary classifier? I am currently trying to do the same - yet I'm running into weird behavior when submitting to run.

This is what I'm trying to use to penalize false positives (and is the custom function I use to do the same within xgboost), however it doesn't appear to actually work within LightGBM.

    weighted_logloss <- function(preds, dtrain) {
        y <- getinfo(dtrain, "label")
        p <- preds
        beta <- 2
        grad <- p * (beta + y - beta*y) - y
        hess <- p * (1 - p) * (beta + y - beta*y)
        return(list(grad = grad, hess = hess))
        }

Maybe I'm misunderstanding what I need to do to tailor the function to work within LightGBM.

[Laurae2]

@grand-home-projects For binary classification (logloss in your case here), the returned values are the raw values (not the transformed values).

You need to transform the predictions as follow: new_preds <- 1 / (1 + exp(-preds))

Using your code, it should look like the following:

    weighted_logloss <- function(preds, dtrain) {
        y <- getinfo(dtrain, "label")
        p <- 1 / (1 + exp(-preds))
        beta <- 2
        grad <- p * (beta + y - beta*y) - y
        hess <- p * (1 - p) * (beta + y - beta*y)
        return(list(grad = grad, hess = hess))
        }

[grand-home-projects]

@Laurae2 Ah, that makes complete sense. Just dropped it in and it appears to be working now.

Thank you much!

[StrikerRUS]

@Laurae2 Seems this issue can be closed (#1940 didn't have autoclose keyword).