Add download option to skip accession numbers #142
Conversation
This improves efficiency when some filings have been already downloaded to another location.
|
@jadchaar This feature has been manually tested -- would love a quick sanity check that this feature is reasonable to add and that I didn't miss it somewhere. Then will add automated tests. |
sec_edgar_downloader/_Downloader.py
Outdated
| @@ -67,6 +67,7 @@ def get( | |||
| before: Optional[Date] = None, | |||
| include_amends: bool = False, | |||
| download_details: bool = False, | |||
| skip_accession_numbers: Optional[set[str]] = None, | |||
There was a problem hiding this comment.
Please use Set from typing library here :)
There was a problem hiding this comment.
IMHO we should not expose this as a parameter, but instead automatically use the existing file system to determine which items to skip.
Thoughts?
There was a problem hiding this comment.
That solution works well for when everything is done on a single server, and indeed it worked well for me until I started decoupling storage and computation. Once that happened, I needed an efficient way to let the ephemeral downloading jobs -- which have only small, temporary file systems -- know what had already been downloaded, which led me here.
It was previously not fully initialized
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #142 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 8 8
Lines 220 223 +3
Branches 30 31 +1
=========================================
+ Hits 220 223 +3 ☔ View full report in Codecov by Sentry. |
|
I have applied the suggestions and added a test. Let me know if there are any outstanding issues! |
This improves download efficiency when some filings have been already downloaded to another location, such as when doing incremental updates.
The existing logic to skip already downloaded files when they are in the same
download_folderis useful for local-only use, but is less useful when files are only local for a brief period before being sent to long-term storage.