add(scripts): combine dumping duplicated record methods and search ep report number duplicates - #578
Conversation
0a3eef6 to
cb1be7d
Compare
zubeydecivelek
changed the title
|
|
||
| EP_RE = re.compile(r"^(?:CERN-EP|CERN-PH-EP|CERN-PPE|CERN-TH-EP)-\d{2}(?:\d{2})?-\d+$") | ||
| RN_TAGS = ("037__a", "037__9", "088__a", "088__9", "9031_b", "909C0r") | ||
| SEARCH_FIELDS = ("reportnumber", "037", "088", "9031_", "909C0r") |
There was a problem hiding this comment.
it's added report number in legacy i was trying to find all the fields can have report number. I dont know if we have any example, i added just in case, if you think it's not needed i can remove
| """Keep a live record; prefer SSO among live ones, else the first live. | ||
|
|
||
| Deleted recids become redirect keys. If every recid is deleted, fall | ||
| back to SSO then the first in the list. |
There was a problem hiding this comment.
what does it mean that record is live? why do we prefer SSO?
There was a problem hiding this comment.
it was already prefering SSO, i didnt changed that https://github.com/CERNDocumentServer/cds-migrator-kit/blob/master/scripts/dump_legacy_recids_to_redirect.py#L83-L85
but i had duplicates of the same recids for example:
2767378: 2766207,
2766207: 2767378,
and we should select the one is not deleted. I'll change the explanation
|
|
||
|
|
||
| # Configure the collection before running. | ||
| collection_q = '980__a:ARTICLE or 980__a:PREPRINT or 980__a:CONFERENCENOTE or 980__a:SLIDE or 980:CONFERENCEPAPER or 980__a:NOTE and 693:"FASER" -980__a:SCICOMMPUBLLHCC' |
There was a problem hiding this comment.
what if the duplicate is outside this collection, like it happened with one record of FASER?
There was a problem hiding this comment.
it searches the whole cds for any ep report number not the collection.
There was a problem hiding this comment.
how do I define the collection query then? If it is whole CDS, do we need the collection query?
There was a problem hiding this comment.
collection query is your selected collection query for migration. script is collecting all the ep report numbers using your query and searches whole cds to find if there is any duplicate
There was a problem hiding this comment.
Understood! Can we call this code using the query from the model I added and integrate it as part of the migration step? Like an analysis step, after extract and before transform? The point is to reduce the amount of extra manual steps we need to run apart from the migrate collection run...
There was a problem hiding this comment.
I understand and I'm also trying to reduce the manual steps, but we need to run this in cds legacy so i think this has to be a manual step :( Or we can combine dump_legacy_recids_to_redirect.py and search_ep_report_number_dups.py` scripts. what do you think?
There was a problem hiding this comment.
I see, I forgot that these are running to legacy.... I think it would be nice to combine into one new script that does an analysis and outputs content that is then used as input to the migration. I would still make this running as e.g ./analyze_collection_migration.py --collection-query='....' --param a etc. and have a clear path of steps inside that produce a consistent output that is then copied over in the new under migraiton collection.
There was a problem hiding this comment.
I added a new script, can we add the script to the legacy main node? so we'll just need to run the command?
sudo -u apache python2 scripts/analyze_collection_migration.py \\
--collection-name faser \\
--collection-q '980__a:ARTICLE or 980__a:PREPRINT and 693:"FASER"'I kept the previous scripts so you can compare, i just combined 2 scripts. if it looks ok i'll remove the other 2 scripts.
e27a78c to
6b52ca7
Compare
6b52ca7 to
8dc82cc
Compare
|
|
||
|
|
||
| # Configure the collection before running. | ||
| collection_q = '980__a:ARTICLE or 980__a:PREPRINT or 980__a:CONFERENCENOTE or 980__a:SLIDE or 980:CONFERENCEPAPER or 980__a:NOTE and 693:"FASER" -980__a:SCICOMMPUBLLHCC' |
There was a problem hiding this comment.
how do I define the collection query then? If it is whole CDS, do we need the collection query?
6066b5b to
45143ed
Compare
45143ed to
f6fcc06
Compare
f6fcc06 to
8dc9c4d
Compare
| "CDS search query for the collection. Do not use -980:DELETED " | ||
| "-980:HIDDEN -980__c:MIGRATED -980__a:DUMMY. Include any 710__g " | ||
| 'clauses, e.g. 710__g:"IT" or 710__g:"CN".' |
There was a problem hiding this comment.
| "CDS search query for the collection. Do not use -980:DELETED " | |
| "-980:HIDDEN -980__c:MIGRATED -980__a:DUMMY. Include any 710__g " | |
| 'clauses, e.g. 710__g:"IT" or 710__g:"CN".' | |
| "CDS search query for the collection. Do not use -980__c:MIGRATED' |
because we do want to exclude from duplicate searched hidden, deleted and dummy records
There was a problem hiding this comment.
i remember we want to mint the legacy recid of the deleted redirected records (for example duplicated record deleted and keeps the redirect record id in 970__d: example record) in the rdm record. if we want to do this we should not exclude the deleted records.
8dc9c4d to
9265371
Compare
4 participants
combined duplicated record search so we have one step in https://gitlab.cern.ch/cds-team/cds-rdm-openshift/-/blob/master/.gitlab/issue_templates/migration.md
added a new script to check if there is any ep report number in the collection has duplicates
UPDATE: combined script added, redirect and ep report number script will be removed
45143ed
update here after this pr merged: https://gitlab.cern.ch/digital-repositories/internal-docs/-/merge_requests/251
and update migration issue template: https://gitlab.cern.ch/cds-team/cds-rdm-openshift/-/blob/master/.gitlab/issue_templates/migration.md?ref_type=heads