I’m using university journal portal OAI endpoints for the third time in a row to fix a typo and handle DOI lookups.
Does anyone know how disrespectful or intrusive this is considered to be?
A few years ago, I scraped arXiv and they blocked my IP almost immediately; perhaps if I had used arXiv’s OAI interface, they wouldn’t have blocked me.
But I really have no idea just how disrespectful or intrusive this is!
Basic etiquette, you’d probably want to ask the university portal maintainer.
Scraping is generally considered bad manners, and yes you would be immediately blocked if you ignore robots.txt. I assume by OAI you mean OpenAI? I’m unsure what kind of integration your university has, but like with many AI integrations, there is little oversight, which leads to… oversights.
OAI are API endpoints for metadata access of articles. I imagine API access is more respectful in terms of bandwidth since it’s organized specifically for access by the universities?
