Add support for COS public endpoint to Kubeflow Pipelines runtime config

[portellaa]

What changes were proposed in this pull request?

This aims to introduce a new property to configure the public endpoint for the Cloud Objecte Storage if necessary.

In some cases (ours for example) we run separate services with separate policies for access, so we use a private endpoint for the API (which normally writes or it's meant to be used by automated components) and a public endpoint only for read that is meant to be provided to the user.

This is the result of this change

How was this pull request tested?

This change is very easy, no "new logic", well apart of creating a variable that tries to get the configured value from the runtime configuration an if doesn't exists uses the API endpoint that is mandatory.

Make sure that the pre change works well:

• create a runtime configuration without public endpoint configured
• run pipeline
• click on the object storage link and open the private one

Make sure that the new public endpoint works

• create a runtime configuration with public endpoint
• • run pipeline
• click on the object storage link and open the public one

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

[elyra-bot]

Thanks for making a pull request to Elyra!

To try out this branch on binder, follow this link:

[ptitzler]

Conceptually this is looking good! I haven't tried the code yet but wanted to bring up a few things we need to add to close the loop:

• Update the user documentation to list the new property (source which surfaces here)
• Update the Runtimes metadata display to render the public object storage endpoint URL as a link instead of the non-public endpoint - see changes to the RuntimesWidget in this PR, which delivered similar functionality for the Kubeflow Pipelines endpoint.

I tagged this as a Kubeflow Pipelines feature for now. We can always extend this to Apache Airflow in a follow-up PR (or have somebody with access to an Airflow deployment augment your PR since the new functionality is not runtime specific).

[ptitzler]

Code changes look good and work as expected:

• KFP RTC configured without public COS endpoint
  • tested editor and RTC list view
  • tested run pipeline and export pipeline
• KFP RTC configured with public COS endpoint
  • tested editor and RTC list view
  • tested run pipeline and export pipeline
• no impact on Airflow RTC

Last remaining item is to fix the broken integration test Pipeline Editor tests -> should save runtime configuration. This call fails because there are now two elements in the DOM with titles containing the string object storage endpoint.

[portellaa]

Code changes look good and work as expected:

• KFP RTC configured without public COS endpoint

  • tested editor and RTC list view
  • tested run pipeline and export pipeline

• KFP RTC configured with public COS endpoint

  • tested editor and RTC list view
  • tested run pipeline and export pipeline

• no impact on Airflow RTC

Last remaining item is to fix the broken integration test Pipeline Editor tests -> should save runtime configuration. This call fails because there are now two elements in the DOM with titles containing the string object storage endpoint.

Oh thanks. I was looking for that. fixing

[ptitzler]

If you can make the final change by tomorrow we'll include the PR in the upcoming release, otherwise we might have to defer.

[ptitzler]

LGTM
Thank you for your contribution. Much appreciated!

[portellaa]

@ptitzler thank you us for the project.

sorry busy morning, i tried to run the tests, with what you said, but they failed with the error:

[akchinSTC]

sorry busy morning, i tried to run the tests, with what you said, but they failed with the error:

@portellaa - Were there additional non committed changes you were testing against locally? The CI tests here seems to have passed without issue.

[ptitzler]

When I run the cypress tests in my environment I occasionally see 'random' errors that are totally unrelated to the code changes. The same behavior can be observed in some of the official CI runs that fail only to succeed the next time without any changes in between. We've also confirmed through manual testing that the code works as expected. Unless there is a clear pattern I wouldn't worry about it.

[portellaa]

sorry busy morning, i tried to run the tests, with what you said, but they failed with the error:

@portellaa - Were there additional non committed changes you were testing against locally? The CI tests here seems to have passed without issue.

no no i didn't.

they fail at different steps, but probably some configuration or bad package on my conda env 🤷