Academic peer review has been broken forever and AI appears to be the final 1 ton tungston rod that will break the camels back. The wave of AI generated papers is crushing this years conference cycle and the traditional academic preference for opaque, ponderous papers over clear, straightforward ones means we are going to get heaps of even worse rubbish than in the past.
"These are important questions because scientists are increasingly overwhelmed with the volume of new work posted on preprint servers and published in journals. As a result, traditional quality signals used for triaging papers, such as journal, conference venue, and institution, are becoming less reliable."
The diagnosis is right but the treat is dead wrong. Instead of silly scoring systems, please improve recommender systems for papers. In this space, also opt-in and query-based personalization would be okay.
Recommender systems for papers tend to be pretty bad, so there's a lot of room for improvement. I'll use Semantic Scholar as an example. I have a bunch of folders in what they call a "Library" with recommendations turned on. Semantic Scholar tends to recommend things that are in the same general area but not specific enough. So I guess that Semantic Scholar seems to interpret adding a paper to a folder as expanding the scope of the folder, but it could be narrowing. There's no way to distinguish between the two. Their recommender system is supposed to magically figure it out. Some way to add additional context like relevant keywords or a way to select which parts of the papers are relevant would be helpful. As it stands, I have to repeatedly thumbs down recommendations, and Semantic Scholar doesn't figure out what I mean from that vague signal and instead stops recommending much anything. It's not that there are no additional papers to go into these folders either as I've added more over time that I've found through other means.
What’s concerning about the single QED score aside from the arguments in the paper is that it’s likely a ploy for QED to sell their paper validation solutions.
In a future where QED scores gain traction, every researcher will need a QED validator subscription to score their paper before publishing. Allowing researchers to tweak their papers for a higher score, potentially without substantial changes to their discovery.
Interesting but immoral customer acquisition tactic.
Academic peer review has been broken forever and AI appears to be the final 1 ton tungston rod that will break the camels back. The wave of AI generated papers is crushing this years conference cycle and the traditional academic preference for opaque, ponderous papers over clear, straightforward ones means we are going to get heaps of even worse rubbish than in the past.
Bad time for me to decide to get a PhD!
"These are important questions because scientists are increasingly overwhelmed with the volume of new work posted on preprint servers and published in journals. As a result, traditional quality signals used for triaging papers, such as journal, conference venue, and institution, are becoming less reliable."
The diagnosis is right but the treat is dead wrong. Instead of silly scoring systems, please improve recommender systems for papers. In this space, also opt-in and query-based personalization would be okay.
Recommender systems for papers tend to be pretty bad, so there's a lot of room for improvement. I'll use Semantic Scholar as an example. I have a bunch of folders in what they call a "Library" with recommendations turned on. Semantic Scholar tends to recommend things that are in the same general area but not specific enough. So I guess that Semantic Scholar seems to interpret adding a paper to a folder as expanding the scope of the folder, but it could be narrowing. There's no way to distinguish between the two. Their recommender system is supposed to magically figure it out. Some way to add additional context like relevant keywords or a way to select which parts of the papers are relevant would be helpful. As it stands, I have to repeatedly thumbs down recommendations, and Semantic Scholar doesn't figure out what I mean from that vague signal and instead stops recommending much anything. It's not that there are no additional papers to go into these folders either as I've added more over time that I've found through other means.
Indeed, but according to my testing, Semantic Scholar is actually still the best in the game. But even that is extremely slow. What gives?
And please give me something like:
./query --topic "whatever I am interested at the moment" --OAonly --allPapersAsPDFs --download --maxRelated=10
Or:
./query --topic "whatever I am interested at the moment" --RSSfeed --maxEntriesPerDay=25 --withPersonalization
etc. In other words, most scientists are not particularly interested on navigating obscure and slow web platforms.
What’s concerning about the single QED score aside from the arguments in the paper is that it’s likely a ploy for QED to sell their paper validation solutions.
In a future where QED scores gain traction, every researcher will need a QED validator subscription to score their paper before publishing. Allowing researchers to tweak their papers for a higher score, potentially without substantial changes to their discovery.
Interesting but immoral customer acquisition tactic.
Yes, and when the system is not auditable it makes it easy for biases to propagate!
To save people the bother:
An LLM read a paper and concluded it was a top 1% paper.
Everyone involved were happy.
It actually sounds like not everyone involved is happy. Maybe read the article, because it's a criticism, not an endorsement.