Native Concourse Provider

[brumhard]

Hi there,

as already mentioned in pulumiverse/awesome-pulumi#17 I'm also trying to support concourse with Pulumi since I love the idea of reusability of tasks through normal code and not some random YAML linting tools that are used right now.

You can find my repo at https://github.com/brumhard/pulumi-concourse.
As you can see it does not have any real docs, and it's using just the fresh Pulumi template currently (just like this repo).
The only difference is that I tried to implement a real native Pulumi provider which differs from this generated terraform bridge.

The advantage is that the concourse pipeline does not need to be some arbitrary JSON string (that is used here if I understand correctly) but instead all resources are implemented in the provider's JSON schema which gives you nice intellisense and IDE support.
(Sadly not all features are supported currently since this needs to be implemented manually ofc).

What do you think? Would you like to maybe merge our efforts or maybe there's someone who would like to further improve the support? Or is plain JSON just the right thing to do since it does not require any additional maintenance?

[blarghmatey]

As far as the job definition, another thing that may be worth looking at is https://github.com/SAP/pipeline-dsl (recognizing that it's Python specific). There is also https://github.com/cappyzawa/concourse-pipeline-jsonschema which is a (admittedly incomplete) JSON schema of the pipeline definition structure which could simplify cross-language support.

[brumhard]

Yes I also opened a discussion in the concourse repo on available json schema's that's worth looking at, but there nothing officially supported right now: concourse/concourse#8084

[ringods]

@brumhard I gave this some thought myself a few months back.

For tools like Terraform or Pulumi that work with providers, each resource usually maps to an entity which can be managed using the API. In the case of a REST API, this maps usually to an endpoint that contains the resource type in it, e.g. /api/<resourcetype>/:id. In a PUT or POST, you typically encode the properties as a JSON or XML document which is sent as the payload for the request. If one of the resource properties takes a document as a property value, that content is passed as a value, whether the content is correct or not. The provider in this case doesn't have any further intrinsic knowledge on this.

To make an analogy, the AWS Identity & Access Management APIs work with endpoints which accept a JSON policy as input. The different pieces of such a policy are not top level entities available as separate AWS API endpoints. Therefor, the AWS providers do not provide code to construct such a policy document. However, there were people who wanted to generate correct IAM policies and not have to continue creating JSON documents by way of trial and error. Out of that need came a pure NPM package: @thinkinglabs/aws-iam-policy. You construct the IAM policy in code, call .json property at the end and pass the generated JSON to the Pulumi AWS provider.

Mapping this back to this Concourse provider. Looking at the routes which the REST API offers, the ones related to pipeline configuration are these:

https://github.com/concourse/concourse/blob/6601af4d15e953311c5baddb6647de661db21edb/atc/routes.go#L126-L127

As you can see, you get or set a full pipeline YAML file only with a single call. There are no Concourse API endpoints for separate jobs, rescource types or resources. With the analogy of the AWS IAM JSON policy documents in mind, I propose to not mix both aspects in a single project. My suggestion is this:

• this project keeps in sync with bridging the upstream Terraform provider, only providing resources and get functions which map onto entities provided by the REST API.
• you create a library (in analogy to @thinkinglabs/aws-iam-policy) that can build the pipeline description and has an accessor to generate the YAML version of it. Your library will be free of any Pulumi dependency.

If you agree on this approach, do let us know so we can link to it!

[brumhard]

@ringods first of all thanks for the detailed feedback.

I'm not sure whether I understand correctly tho.
Why would you not create a single type for pipeline configs then? I wouldn't create sub resources for jobs or resources either, but I still think it would be nice to have a fully typed pipeline resource since all the properties are well defined (not only the path parameters like team ID but also the whole body).
What's the difference for example to a Kubernetes Pod or sth that is also only created and managed as one big entity but is still fully typed?

Also, is there any option to still use Pulumi's great code generation features if I would maintain sth like a separate pipeline constructor library? Like maybe a multi-language component provider? I would really appreciate it if we could make it available for all Pulumi languages.

[ringods]

@brumhard there are two paths:

1. your path, with a complete native provider, where you have the full freedom to define the type of the args yourself.
2. my path, using the Terraform bridge, where the provider is generated with the same typings as in the TF provider. I don't think you can change the type of a property with something you add on and becomes integrated into this provider.

For now, I'm not fond of going the full native route because that means we can't leverage from all the efforts already done on the TF provider.

In my previous comment, I mentioned how you can still write a pure language library (no Pulumi dependencies) which works well in combination with the TF bridged provider. As you mentioned, you can't benefit from the multi-language code generation for this. I don't know if there are any Pulumi specific or general solutions to generating multi-language code libraries.

[brumhard]

Thanks @ringods. I'll close this issue for now and maybe come back when I find sth.

[ringods]

@brumhard I'm reopening this myself. Given Pulumi CrossCode, I'm investigating this provider to be a native provider rather than a wrapped TF provider. I added a new detached branch to this repository with the initial work:

https://github.com/pulumiverse/pulumi-concourse/tree/main-native

I made it a detached branch given there is no common history with the files in the current main branch. The build setup is working, meaning from an OpenAPI file to Pulumi Schema, then from the schema to generated SDKs, although there is nothing in it yet.

I will investigate more, but I could use some help by reverse engineering the routes.go file into the swagger.json file:

https://github.com/concourse/concourse/blob/master/atc/routes.go
https://github.com/concourse/concourse/blob/d81c8901b62b84ee79362ae4a225460e7bbcb5fe/atc/api/handler.go

[ringods]

@brumhard I made some progress. I was able to created separate types for every section of a pipeline. Not every property has been converted yet, but you will get the idea. Starting from a Pulumi schema, I created the Pipeline resource:

https://github.com/pulumiverse/pulumi-concourse/blob/main-native/provider/cmd/pulumi-resource-concourse/schema.json#L169-L185

One of its properties is the config, which points to a PipelineConfig object. Such a PipelineConfig object is built up from other object types:

https://github.com/pulumiverse/pulumi-concourse/blob/main-native/provider/cmd/pulumi-resource-concourse/schema.json#L87-L167

When generating the different language SDKs, you get proper generated types in each of the supported languages. Let's take Typescript as an example:

• Pipeline resource with PipelineArgs: https://github.com/pulumiverse/pulumi-concourse/blob/main-native/sdk/nodejs/pipeline.ts
• PipelineConfigArgs and the types it uses: https://github.com/pulumiverse/pulumi-concourse/blob/main-native/sdk/nodejs/types/input.ts

You can see the result for all the different languages here:

https://github.com/pulumiverse/pulumi-concourse/tree/main-native/sdk

Is this what you are after? If so, I hope you can join me in continuing the development of the native provider, rather than the Terraform bridged provider.

Is this what you are after?

[brumhard]

Hi @ringods, I really appreciate your work towards a native provider 🚀 Sadly I'll be absent from work for two months and afterwards I'm not sure whether I'll work with concourse again and if I'll be able to take the time to support you with that.
For the schema I translated some of the properties to a schema myself here: https://github.com/brumhard/pulumi-concourse/blob/main/provider/cmd/pulumi-resource-concourse/schema.json.
Maybe that helps a bit for the tedious part of creating the schema.json since sadly there is no official json schema for concourse.