Skip to content
/ ACT Public

Alternative approach for Adaptive Computation Time in TensorFlow

License

Notifications You must be signed in to change notification settings

for-ai/ACT

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Adaptive Computation Time

Alternative approach on Adaptive Computation Time for Recurrent Neural Networks by Alex Graves in TensorFlow.

Contributor: Berkay Antmen, Divyam Madaan, Bryan M. Li, Aidan N. Gomez

Idea

We use a small trick to help with the training of ACT modules. The idea is to penalize pondering less when the model is incorrect and more when it is correct.

In effect, relaxing the objective constraints when the model is struggling and then asking for computation efficiency only once the model has solved the problem. We implement this simply by scaling the ponder cost term by the inverse of the task loss. See function calculate_ponder_cost()

To use new ponder cost, set flag --use_new_ponder_cost=True

Running this Code

Call pip install -r requirements.txt to install all dependencies.

Training Data

Sample Call

The settings for the reader used is passed as flag. For example, to run the parity task, call:

python -m ACT.train --model=act --hparam_sets=act_parity  --output_dir=runs/act_parity --data=parity  --train_steps=100000

Results

Task Accuracy Sequence error rate Pondering
Parity parity_acc parity_ser parity_ponder
Sort sort_acc sort_ser sort_ponder
Addition add_acc add_ser add_ponder
  • #0d6306 Our ponder cost
  • #00e5ff LSTM cell
  • #001289 Original ponder cost

Results for parity, sort and addition tasks are displayed. Light blue color represents LSTM cell, dark blue represents ACT cell with old ponder cost and green represents ACT cell with our new ponder cost.

The model with the new ponder cost ponders comparatively more than the model with the original ponder cost. The ACT cell with new ponder cost performed better than the old ponder cost for parity task, though performed similarly for sort and addition tasks.

Contributing

We'd love to accept your contributions to this project. Please feel free to open an issue, or submit a pull request as necessary. If you have implementations of this repository in other ML frameworks, please reach out so we may highlight them here.

Acknowledgment

The ACT code is build upon DeNeutoy/act-tensorflow