EcsTaskRunner

[yaronlevi]

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar request and didn't find it.
• I searched the Prefect documentation for this feature.

Prefect Version

3.x

Describe the current behavior

Creating this idea after a chat I had with Will Raphaelson.

So currently we have:

• RayTaskRunner()
• DaskTaskRunner()

What about an EcsTaskRunner()?
Ray and Dask TaskRunners allows me to .submit() a task to some “compute pool”, there the task will be attached to some one-off instance with dedicated memory and cpu.
Isn’t this the exact same story when looking at AWS’s ECS?
Ray and Dask are quite “exotic” technologies and often come up when you start talking about highly parallel machine learning/AI workloads.
But what about a more common use case: A weekly job that needs to process many files in parallel where each file is significantly different in size:

@flow(task_runner=EcsTaskRunner())
def my_weekly_huge_files_job():
    files_on_s3 = get_list_of_huge_files()
    for path,file_size in files_on_s3:
        if file_size > xxx:
            process_file.submit(path, memory_size='4gb')
        else
            process_file.submit(path, memory_size='1gb')

The code above would be super robust since a certain huge file might crash, but it won’t affect the other instances.
I believe such EcsTaskRunner() would be adopted by the community very quickly as many already use ECS for their ECS push work pools.
This opens up many possibilities for distributed compute on remote, separated, machines, without intruding big guns like Ray or Dask.

As a side note, maybe Dask and Ray are already much more accessible these days? I could swipe a credit card and use Coiled.io (managed Dask cluster) to get a very similar experience to what I’ve described above in the code. But still ECS would be much more common and affordable.

Describe the proposed behavior

Add support for EcsTaskRunner() as many already use ECS, instead of using "big guns" like Dask and Ray.

Example Use

No response

Additional context

No response

[WillRaphaelson]

thanks @yaronlevi - the need is clear to me - to be able to execute prefect tasks as ecs tasks with an elegant UX. We'll be taking up design for this problem in the next weeks to months and I'll keep the community apprised on this and solicit feedback via this issue.