In commit 62191804822c5f42456a7075f3db8368b98c4381, the progress looks promising. There are actually no changes. Sometimes the model encounters "NaN" (not a number) errors with unknown reasons. I rolled back some steps to bypass that issue. There was an accident so the older records in the chart is lost.
In commit 8cb54369787640881b4f32ca7556289f58bd987a, got NaN after 18900 batches:
In commit 7bbc486576a27f1106ced18f5f86bed800ae73c1, got NaN after 5400 batches:
- tensorflow
- numpy
- scipy
- lmdb