• 1 Post
  • 1.03K Comments
Joined 1 year ago
cake
Cake day: June 30th, 2023

help-circle






  • The AI summaries were judged significantly weaker across all five metrics used by the evaluators, including coherency/consistency, length, and focus on ASIC references. Across the five documents, the AI summaries scored an average total of seven points (on ASIC’s five-category, 15-point scale), compared to 12.2 points for the human summaries.

    The focus on the (now-outdated) Llama2-70B also means that “the results do not necessarily reflect how other models may perform” the authors warn.

    to assess the capability of Generative AI (Gen AI) to summarise a sample of public submissions made to an external Parliamentary Joint Committee inquiry, looking into audit and consultancy firms

    In the final assessment ASIC assessors generally agreed that AI outputs could potentially create more work if used (in current state), due to the need to fact check outputs, or because the original source material actually presented information better. The assessments showed that one of the most significant issues with the model was its limited ability to pick-up the nuance or context required to analyse submissions.

    The duration of the PoC was relatively short and allowed limited time for optimisation of the LLM.

    So basically this study concludes that Llama2-70B with basic prompting is not as good as humans at summarizing documents submitted to the Australian government by businesses, and its summaries are not good enough to be useful for that purpose. But there are some pretty significant caveats here, most notably the relative weakness of the model they used (I like Llama2-70B because I can run it locally on my computer but it’s definitely a lot dumber than ChatGPT), and how summarization of government/business documents is likely a harder and less forgiving task than some other things you might want a generated summary of.




  • IMO the most valid argument is that there are way more people making a middling income than people making a high income, so any reduction in taxes for those people would need a proportionally much larger increase in the upper brackets to maintain the same level of tax revenue, if it’s possible to make the numbers work at all depending on how much of a tax break you want to give. The minimum amount to be taxed is set based on where the tail end of the bell curve is, the number of people who are poor enough not to be taxed is small.

    Of course there’s also the fact that the richest people don’t get their money from having a job at all, it’s all in investments, so messing with income tax rates doesn’t even affect them.


  • But I think the point is, the OP meme is wrong to try painting this as some kind of society-wide psychological pathology, when it’s rather business people coming up with simple reliable formulas to make money. The space of possible products people could want is large, and this choice isn’t only about what people want, but what will get attention. People will readily pay attention to and discuss with others something they already have a connection to in a way they wouldn’t with some new thing, even if they would rather have something new.



  • The biggest reason that is often overlooked is wealth inequality. The rich keep accumulating wealth, and real estate is a scarce form of wealth that holds value, produces a return, and can be accumulated. It probably accelerated recently because of the large amount of money that was dumped into the system around covid; that was yet another opportunity for the wealthy to grab a bigger share of the pie.

    If things keep going this way, we’re going to get into a situation where regular people don’t own houses anymore, and rent is a much larger percentage of your income.




  • that is not the … available outcome.

    It demonstrably is already though. Paste a document in, then ask questions about its contents; the answer will typically take what’s written there into account. Ask about something you know is in a Wikipedia article that would have been part of its training data, same deal. If you think it can’t do this sort of thing, you can just try it yourself.

    Obviously it can handle simple sums, this is an illustrative example

    I am well aware that LLMs can struggle especially with reasoning tasks, and have a bad habit of making up answers in some situations. That’s not the same as being unable to correlate and recall information, which is the relevant task here. Search engines also use machine learning technology and have been able to do that to some extent for years. But with a search engine, even if it’s smart enough to figure out what you wanted and give you the correct link, that’s useless if the content behind the link is only available to institutions that pay thousands a year for the privilege.

    Think about these three things in terms of what information they contain and their capacity to convey it:

    • A search engine

    • Dataset of pirated contents from behind academic paywalls

    • A LLM model file that has been trained on said pirated data

    The latter two each have their pros and cons and would likely work better in combination with each other, but they both have an advantage over the search engine: they can tell you about the locked up data, and they can be used to combine the locked up data in novel ways.


  • Ok, but I would say that these concerns are all small potatoes compared to the potential for the general public gaining the ability to query a system with synthesized expert knowledge obtained from scraping all academically relevant documents. If you’re wondering about something and don’t know what you don’t know, or have any idea where to start looking to learn what you want to know, a LLM is an incredible resource even with caveats and limitations.

    Of course, it would be better if it could also directly reference and provide the copyrighted/paywalled sources it draws its information from at runtime, in the interest of verifiably accurate information. Fortunately, local models are becoming increasingly powerful and lower barrier of entry to work with, so the legal barriers to such a thing existing might not be able to stop it for long in practice.