-
Notifications
You must be signed in to change notification settings - Fork 929
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: The Foreign Keys are getting broken down to their component pieces for MongoDB to BigQuery #911
Comments
@theshanbhag Can you please take a look / triage this? |
Here's an example of what I have been doing and how I worked around it for the moment if this helps:
|
This issue has been marked as stale due to 180 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the issue at any time. Thank you for your contributions. |
Bump |
Although it is not a definitive solution, I've created a UDF that recursively transforms mongo objectid into strings. Here the code for my UDF. Hope it helps... function parseOid(obj) {
if (obj instanceof Array) {
for(var i = 0; i < obj.length; i++) {
obj[i] = parseOid(obj[i]);
}
} else {
for(var prop in obj) {
if (prop == "$oid") {
obj = obj.$oid
return obj
}
if (obj[prop] instanceof Object || obj[prop] instanceof Array) {
obj[prop] = parseOid(obj[prop])
}
}
}
return obj
}
function process(inJson) {
var obj = JSON.parse(inJson)
obj = parseOid(obj)
return JSON.stringify(obj)
} |
Related Template(s)
MongoDB to BigQuery
What happened?
The Reference Ids (aka Foreign Keys), which are the same format as the primary MongoId / ObjectId / _id, are getting broken down into their individual component pieces.
Input
accountId: 63e131fb63ccf7e561f96920
Output to source_data in BigQuery:
"accountId":{"timestamp":1675702779,"counter":16345376,"randomValue1":6540535,"randomValue2":-6815},"
I figured out a temporary workaround with a GoogleSQL javascript UDF, but handling this in the template library will be much better.
Thank you!
Beam Version
Newer than 2.46.0
Relevant log output
No response
The text was updated successfully, but these errors were encountered: