You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
This line of code is causing failures in Databricks for the column anomalies test when a column_timestamp is missing.
Caused by: org.apache.spark.sql.catalyst.ExtendedAnalysisException: [UNRESOLVED_COLUMN.WITHOUT_SUGGESTION] A column, variable, or function parameter with name `last_session_start_ts` cannot be resolved. SQLSTATE: 42703; line 30 pos 19
at org.apache.spark.sql.catalyst.ExtendedAnalysisException.copyPlan(ExtendedAnalysisException.scala:91)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.$anonfun$execute$1(SparkExecuteStatementOperation.scala:688)
I've resolved this locally by adding a timestamp_column but others might not have that option.
To Reproduce
Create a column_anomalies test for a model that doesn't have a timestamp_column
Run test on Databricks
Observe that the extra , at the end of start_bucket_in_data causes an issue with Databricks
Expected behavior
This test should not produce an error.
Screenshots
Environment (please complete the following information):
Elementary CLI (edr) version: 0.15.1, can be found by running pip show elementary-data
Elementary dbt package version: 0.15.2, can be found in packages.yml file
Hi @angeml !
Thanks for opening this issue and sorry for the delayed response.
Yes you are absolutely right, it seems this flow was broken and we actually have a PR that fixes it which should be merged in the near future.
Describe the bug
This line of code is causing failures in Databricks for the column anomalies test when a column_timestamp is missing.
I've resolved this locally by adding a timestamp_column but others might not have that option.
To Reproduce
,
at the end ofstart_bucket_in_data
causes an issue with DatabricksExpected behavior
This test should not produce an error.
Screenshots
![Screenshot 2024-06-13 at 12 46 17 PM](https://private-user-images.githubusercontent.com/54677646/339475846-eed5c2c9-cfb4-4245-b902-d3f60d52ee1f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk5Nzg1NTYsIm5iZiI6MTcxOTk3ODI1NiwicGF0aCI6Ii81NDY3NzY0Ni8zMzk0NzU4NDYtZWVkNWMyYzktY2ZiNC00MjQ1LWI5MDItZDNmNjBkNTJlZTFmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzAzVDAzNDQxNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTIzZmIwNmYxNzFmNmRlYWMyZDhmODE1ZGNiOTA2OGM1ZjEyOTAwZmFmNDU4ZjllYjM5YjU3MGQ4MjA3OWU2NzEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.JJ7BXLzfvREFvpVKNNwvRVRpjY7xhy2bgD5xIYLt5V4)
Environment (please complete the following information):
pip show elementary-data
packages.yml
fileAdditional context
Slack - https://elementary-community.slack.com/archives/C02CTC89LAX/p1716306300184349
Would you be willing to contribute a fix for this issue?
For sure 👍 But I think it just needs a comma removal 😄
The text was updated successfully, but these errors were encountered: