Skip to content

Restore best score in PassthroughScoring when continuing training - #1175

Open
syahra712 wants to merge 1 commit into
skorch-dev:masterfrom
syahra712:fix/passthrough-scoring-best-score
Open

syahra712 wants to merge 1 commit into
skorch-dev:masterfrom
syahra712:fix/passthrough-scoring-best-score

Conversation

@syahra712

Copy link
Copy Markdown

ScoringBase.on_train_begin (#268) picks up the best score from the history, so EpochScoring/BatchScoring keep track of it when training continues from a loaded history. PassthroughScoring has no such hook. It is what records the default train_loss and valid_loss, so after

net.initialize()
net.load_params(f_history='history.json')
net.partial_fit(X, y)

its best_score_ starts again from inf, and the first new epoch is flagged valid_loss_best=True whatever its value. A Checkpoint with the default monitor='valid_loss_best' then overwrites the better checkpoint.

Repro (5 epochs, then one more epoch with a learning rate that is far too high):

import numpy as np, torch
from sklearn.datasets import make_classification
from skorch import NeuralNetClassifier
from skorch.callbacks import EpochScoring

X, y = make_classification(1000, 20, random_state=0)
X = X.astype(np.float32)

class M(torch.nn.Module):
    def __init__(self):
        super().__init__(); self.l = torch.nn.Linear(20, 2)
    def forward(self, X): return torch.softmax(self.l(X), -1)

def make():
    torch.manual_seed(0)
    return NeuralNetClassifier(M, max_epochs=5, lr=0.1, verbose=0, callbacks=[
        ('nll', EpochScoring('neg_log_loss', lower_is_better=False))])

net = make().fit(X, y)
net.save_params(f_history='hist.json')

net2 = make().initialize()
net2.load_params(f_history='hist.json')
net2.set_params(lr=50.0)
net2.partial_fit(X, y, epochs=1)
print(net2.history[-1, 'valid_loss'], net2.history[-1, 'valid_loss_best'])
print(net2.history[-1, 'neg_log_loss_best'])

On master, the resumed epoch has valid_loss 1.27 against a previous best of 0.22. Even so, valid_loss_best is True, while neg_log_loss_best (an EpochScoring) is correctly False. With this change both are False.

Change: I moved the lookup from ScoringBase.on_train_begin into a small helper, _best_score_from_history, and call it from a new PassthroughScoring.on_train_begin. ScoringBase behaves as before.

Tests: I added TestPassthrougScoring::test_uses_best_score_when_continuing_training, which mirrors the existing EpochScoring/BatchScoring test of the same name with the same score sequence and expected flags. On master, 5 of its 8 cases fail. skorch/tests/callbacks/, test_net.py and test_classifier.py pass locally, except the two TestTorchCompile tests, which also fail on master with my local torch 2.2.2.

I also added a CHANGES.md entry under Fixed.

ScoringBase.on_train_begin picks up the best score from a loaded
history, but PassthroughScoring (which records the default train_loss
and valid_loss) did not. After load_params(f_history=...) and
partial_fit, the first new epoch was therefore always flagged as
valid_loss_best, even when it was worse, so a Checkpoint monitoring
valid_loss_best would overwrite the better checkpoint.

Move the lookup into a shared helper and call it from both callbacks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants