Admin S3 Object Deletion - Build out - add Asset Image File field....

[jkueloc]

Admin S3 Object Deletion - Build out - add Asset Image File field
Research/discussions ticket #1573
Solution Decision ticket #1574 (should be more than one ticket/issue to be granular about the path of individual choices and results that lead to the 'whole' solution...?)

[jkueloc]

Update: Image Field

• Added Image Field to Asset model
• How to handle populating new field?
  • Makemigrations and migrate don't actually populate and associate field to existing s3 objects.
  • I wrote a "data" migration to populate the field. - testing it now. It keeps stopping ~ 1 hr and 18 min into it
    zsh: killed pipenv run ./manage.py migrate. Suspect not enough memory. ~700k records
  • switching to direct sql - for now ...
  • Need to create a management command and/or "one off" celery task based on "data" migration logic @ queryset level...? See Admin S3 Object Deletion - Build a process to populate the Storage image field value for existing assets  #1593
• How does this impact the Import process?
  - The import process will need modification to populated the Asset storage_image ImageField or another process added...
• Is more logging needed? See Logging messages specific to the Storage image field. #1594
• Is more messaging needed on "Are you sure?" panel/screen?
  - Not at this time - image file field is part of the asset model.

Tests run in local dev environment attached to development content:

• Successfully tested delete of asset with related S3 object deleted.
• Successfully tested adding a local file thru admin asset change form and uploading to S3 with correct object key in db
• Successfully tested importing items and populating storage_image value.
• Successfully deleted imported items at both the item level and the asset level - both successfully deleted the associated S3 asset image file.

[lalgee]

@jkueloc I tested the small import using test Lincoln items and the storage image field successfully populated. Image replacement also worked successfully. But deletion failed for 1 or more items -- recieved a 500 error.

[lalgee]

Testing was also successful with an import of 9,000+ images. I did not see a noticeable delay in import time. I also tested asset replacement and deletion with this group. Replacement worked well and I was able to delete many assets at a time, successfully removing the Concordia record and the AWS files/directories.

In fact, I was able to delete up to 7 items including 657 assets, much higher than we were previously able to do.

So I also tried a much larger deletion of 133 items including 8,189 assets, the remainder of the imported campaign. As I expected, the task appeared to fail showing me a 500 error, but when I checked Concordia and AWS it did successfully delete all of the Concordia records and all of the AWS files and directories!

[jkueloc]

@lalgee That's great! Thank you for testing!

The 500 is that response time thing between the UI and the backend. Which was a topic of a possible solution in the chuck or celery task conversation on the research ticket #1573, as well as prefetching of an older delete ticket #1090.

Building that out will be part of the batch|bulk delete | campaign/project retire ticket(s) that I haven't created / broken out into new tickets yet. I'd like to talk about that in the meeting tomorrow.

I still need to build out the production worthy solution to populate the existing asset storage image fields - but that should inform (for me at least) the coming new tickets.

[elainekamlley]

This ticket will close once set permissions in AWS are applied to test, stage, and prod

[jkueloc]

The permissions in AWS have been applied to test, stage, prod.