AllyshipContent Radar

Hands-on 278

Công cụ, thử nghiệm và hướng dẫn thực hành

Crawl live-1795090500000 · 19:15 19 thg 11, 2026 · Xuất bản 19:35 19 thg 11, 2026

13

ParseBench-2 hit 30k monthly downloads in 3 days, and is #3 on @huggingface trending benchmarks! 📈

52.4

Thử nghiệm thực tế jerryjliu0

ParseBench-2 hit 30k monthly downloads in 3 days, and is #3 on @huggingface trending benchmarks! 📈 In ParseBench-2, we tried hard to create a comprehensive taxonomy of metrics to evaluate document processing, while creating a severity-based framework to align these metrics with human judgment. Whether you’re deeply familiar with document OCR and document OCR benchmarks, or just getting started, we encourage you to check out the ParseBench-2 methodology. We’re happy to help you pick whatever subset of metrics you care about if you’re looking to process documents in production. We’ve merged dozens of PRs since the release date, including fixes/metric updates from some of our peers. We’re aggressively fixing any lingering annotation and metric issues. Report: https://www.llamaindex.ai/files/parsebench-v2-severity-aware-evaluation-of-document-parsing.pdf Hugging Face: https://huggingface.co/datasets/llamaindex/ParseBench-2 Github: https://github.com/run-llama/ParseBench/

X · 16 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
14

I think this rule-based vs. model-based eval debate is missing the point we are trying to make (maybe this is valid feedback that we need to write our paper better😅)

49.0

Thử nghiệm thực tế disiok

I think this rule-based vs. model-based eval debate is missing the point we are trying to make (maybe this is valid feedback that we need to write our paper better😅) ultimately, we are trying to say that severity is human preference, and human preference can be quite granular and nuanced. we need evals that reflect that reality. we love rule-based, deterministic evals AND LLM-as-judge evals. with LLM-as-judge, what's in the rubric (i.e. prompt) becomes really important, to properly encode your preference. in fact, we tried to do exactly that, writing down our preference in https://www.parseprotocol.com/ that can be used as rubric for LLM-as-judge. There's no free lunch, you gotta state your preference somewhere. In ParseBench-2. We chose to state the preference with what we call severity policies and optimize / generate deterministic rules to implement those. There's definitely mistakes, as optimization is imperfect, and we have finite resources. But we are making a genuine effort here, taking in feedback, and improving in the public quickly.

X · 14 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
15

Gemini Decisions API

44.8

Thử nghiệm thực tế Simon Willison

<p><strong><a href="https://blog.google/innovation-and-ai/technology/developers-tools/gemini-decisions-api/">Gemini Decisions API</a></strong></p> Google join <a href="https://simonwillison.net/2026/Sep/21/jev/">Jev</a> and <a href="https://simonwillison.net/2026/Oct/6/llm-openai-decisions/">OpenAI</a> in the decision model category with "Gemini Decision 1". This new model API is almost identical to Jev: it supports the same three types of questions (yes/no, choice, and rubric-scoring) and even reuses Jev's term "Noul" for a yes/no question.</p> <p>It's priced the same as the GPT-6 Luna decision model: 10 cents per million input tokens, and output is free. Jev remains the cheapest of the three, at 4.2 cents per million.</p> <p>Jev is text (and JSON) only. OpenAI can handle text and images. Gemini Decision 1 is fully multi-model: it covers text, images, audio, video, and PDFs.</p> <p>I tested it against video by passing these questions to it along with my <a href="https://simonwillison.net/2026/Nov/10/yonkers-rises/">Yonkers stomping robot</a> video file:</p> <pre><code>dragon: Does this video have a dragon in it? robot: Does this video have a robot in it? france: Is this video set in France? usa: Is this video set in the USA? </code></pre> <p>Jev and OpenAI both return 1.0 or 0.0 for yes/no questions, but Gemini insists on being a little less confident than that:</p> <div class="highlight highlight-source-json"><pre>{ <span class="pl-ent">"dragon"</span>: { <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>yes_no<span class="pl-pds">"</span></span>, <span class="pl-ent">"probability"</span>: <span class="pl-c1">0.04976021841520312</span> }, <span class="pl-ent">"robot"</span>: { <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>yes_no<span class="pl-pds">"</span></span>, <span class="pl-ent">"probability"</span>: <span class="pl-c1">0.9234055387792485</span> }, <span class="pl-ent">"france"</span>: { <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>yes_no<span class="pl-pds">"</span></span>, <span class="pl-ent">"probability"</span>: <span class="pl-c1">0.08329171353166241</span> }, <span class="pl-ent">"usa"</span>: { <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>yes_no<span class="pl-pds">"</span></span>, <span class="pl-ent">"probability"</span>: <span class="pl-c1">0.9230459995157706</span> } }</pre></div> <p><small></small>Via <a href="https://news.ycombinator.com/item?id=50456791">Hacker News</a></small></p> <p>Tags: <a href="https://simonwillison.net/tags/ai">ai</a>, <a href="https://simonwillison.net/tags/generative-ai">generative-ai</a>, <a href="https://simonwillison.net/tags/gemini">gemini</a>, <a href="https://simonwillison.net/tags/llm-release">llm-release</a>, <a href="https://simonwillison.net/tags/jev">jev</a>, <a href="https://simonwillison.net/tags/decision-models">decision-models</a></p>

Simon Willison · 12 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
16

How we tested, what we found and the controls security teams ask about: https://www.llamaindex.ai/blog/how-we-harden-application-testing-for-the-most-sensitive-documents?utm_source=socials&utm_medium=

44.5

Thử nghiệm thực tế llama_index

How we tested, what we found and the controls security teams ask about: https://www.llamaindex.ai/blog/how-we-harden-application-testing-for-the-most-sensitive-documents?utm_source=socials&utm_medium=tw_channel&utm_campaign=blog_harden_application

X · 12 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
17

Show HN: Hob - Open-source, on device alternative to Claude Cowork and ChatGPT

43.7

Thử nghiệm thực tế pancomplex

Hey HN, I'm Christian and this is a new OSS project I've been noodling on for the last couple of weeks. Hob is a free, MIT-licensed desktop app for Mac, Windows and Linux that works on your files the way the Claude and ChatGPT desktop apps do, except the model runs on your own computer. It uses open-weight models (currently Gemma 4 and gpt-oss). All your chats and files never leave your system. The only things that go online are web search (can be turned off in settings) and any MCP connectors you add yourself. Main features: - Automatically finds the best open-weight model for your hardware on first install (Mac/Linux/Windows) - Writes sandboxed Python code for ad hoc tools (data processing, document conversion, graphs etc.) - Built-in web search (powered by SearXNG) - MCP support - Image recognition - CRUD to your local filesystem - Works fully offline with no internet Before building this I tried every local AI app I could find, but it seemed each one had different problems. The main thing I noticed is that none of them could do what the ChatGPT and Claude apps do with your files, e.g. read and change files in a folder, look at images, run code in a sandbox, draw a chart. The most advanced thing any of them did was paste a plain text file into a small context window, which basically failed as soon as you had a file larger than a few KB. I couldn't find anything that could easily parse images and bigger PDFs. The OSS solutions that were more advanced all seemed to default to hosted models (either OSS or closed), which defeats the entire point of the privacy that on-device OSS offers. They also seemed to push towards an account/open-core style business. I build it using Electron bridged with llama.cpp in its own process. On initial load, the system will try and figure out if you have a compatible graphics card and if not default to CPU-only (I ran into a lot of issues of graphics card beeing seemingly compatible but then crashing on first query, there's a whole saga to this lol). Code execution happens in a sandbox that can only write to the chat's folder and has no network unless asked (sandbox-exec on macOS, Landlock on Linux and a low-integrity process on Windows). Images go to a small Qwen3-VL model. Dictation is a local Whisper model. The hardest part was definitely to get the smaller models to use tools reliably. Ran into a ton of issues of 4B models happily saying "here's your chart" without ever calling the chart tool or get stuck in an infinite call loop. I spent a fair amount of the work building guards vs tweaking prompts (that didn't really work for small models). I understand the pushback that local models are weaker than frontier cloud models. This is true, but for most tasks for most normal people, I'd argue you rarely need the newest frontier model. From my own evals, systems with 16+ GB of RAM do really well, and if you have a GPU or more RAM, you get very close to what you'll get out of the box with a frontier hosted model. Excited to hear what you think and see how it runs on other systems!

HN: open weights · 26 thg 10, 2026

Nguyên văn nguồn
Chi tiết nguồn
18

NaiveAI Promises 2,100 Tokens per Second. How Well Does It Code?

42.6

Thử nghiệm thực tế Gabriele Beltramo

Editor’s note: With this post, we are launching a new series focused on trying new models and products with unbiased reviews. Reach out with suggestions to tips@latent.space. Naive-N0.5-Flash is a new open-weights model built for AI coding tasks and tuned to generate as many tokens per second as possible . It was developed by a Chinese company called NaiveAI , with the aim of speeding up engineering and research workflows so that teams can experiment, iterate and solve complex problems faster. The model is an evolution of Xiaomi’s MiMo-V2.5 , which NaiveAI has trained further for long-context reasoning and paired with its own NaiveRT inference engine. Naive-N0.5-Flash arrives with a bold pitch: a model built largely by AI and trained to do AI research itself, with recursive self-improvement as the long-term goal. To see what changed, it helps to know where it started. MiMo-V2.5 is a 310B-parameter sparse Mixture-of-Experts model with 15B active parameters, trained on 48T tokens. Its language backbone inherits the hybrid design of MiMo-V2-Flash, which mixes sliding-window attention (SWA) with global self-attention. NaiveAI's recipe starts from those weights. Through continued, multi-stage training, the team adapted the checkpoint to DeepSeek Sparse Attention layers. According to the NaiveAI technical blog , this took roughly 3.25T additional tokens on top of MiMo's original 48T. About 3T of those additional tokens were trained at a fixed 1M context length and with a constant learning rate. Benchmarking NaiveAI vs. DeepSeek and Z.ai We benchmarked the ultrafast mode of Naive-N0.5-Flash against DeepSeek V4.1 Flash 1017 and GLM 5.3 Flash , and then gave all three the same app-building prompts. The trade-off, we discovered, is that Naive-N0.5-Flash takes longer to reason through harder problems, yet it still delivers answers faster overall. However, that comes at a slightly higher cost per question — although that may be an acceptable price for applications where speed comes first. Most popular models on OpenRouter as of the end of October 2026. DeepSeek V4.1 Flash 1017 and GLM 5.3 Flash are two of the most popular models in OpenRouter . Both belong to the Flash model class and demonstrate excellent capabilities in AI-assisted coding . As of this writing, performance statistics for NaiveAI models are not yet available on Artificial Analysis . Therefore, we report the Intelligence Index scores of the models used in our comparison: Model Intelligence Index DeepSeek V4.1 Flash 1017 (Max): 45 GLM-5.3-Flash: 43 Naive-N0.5-Flash: not yet available Published benchmark results offer some insight into the new model's capabilities. They place its coding ability between the two comparison models, and ahead of both on NL2Repo-Bench . The table below shows benchmark scores reported in the official model cards and technical blog posts of the respective models. Sources: GLM-5.3-Flash – Technical Blog , DeepSeek V4.1 Flash 1017 – Model Card , Naive AI – Research . Model NL2Repo-Bench DeepSWE v1.1 Agent's Last Exam (ALE) GLM-5.3-Flash 56.3 63.4 26.3 DeepSeek V4.1 Flash 1017 68.2 77.0 35.4 Naive-N0.5-Flash 71.9 67.8 32.4 Our own evaluations below look at the differences between these models in more detail. Speed and verbosity experiments Speed is the headline feature. 1 NaiveAI promises faster research iteration than comparable models, which matters to developers and AI researchers who need to test ideas quickly. 2 We evaluated each model on 10 randomly sampled, text-only questions at each of three difficulty levels: 10 easy questions from MATH-500 , 10 medium questions from GPQA-Diamond and 10 hard questions from HLE-Diamond . That makes 30 questions per model. All experiments used the same maximum completion-token limit of 256K. For each request, we recorded response speed (tokens per second, TPS), the total number of completion tokens, and end-to-end response time. The code for the experiments is at https://github.com/gbeltramo/llm-verbosity . The tables below report the mean and median across all 30 successful requests for each configuration. Throughput figures come from different sources: OpenRouter for DeepSeek V4.1 Flash 1017 and GLM 5.3 Flash, and server-side measurements for Naive-N0.5-Flash. They should be compared with that in mind. Graphic by Gabriele Beltramo Key observations Naive-N0.5-Flash reports exceptionally high throughput , approaching or passing 1,000 tokens per second in both thinking configurations. Because its figures are server-side and the other models' figures come from OpenRouter, we do not claim an exact speed multiple over DeepSeek V4.1 Flash 1017 or GLM 5.3 Flash. Thinking has a large effect on verbosity. On the hard HLE questions, Naive-N0.5-Flash with thinking on averaged about 108,500 completion tokens, roughly nine times the GLM 5.3 Flash (high) average and 11 times the DeepSeek V4.1 Flash 1017 (high) average. The medians tell a different story from the means. Across all 30 questions, the median completion for Naive-N0.5-Flash with thinking on was about 3,200 tokens, far below its average of more than 51,000. A small number of requests produced exceptionally long reasoning traces and pulled the average up. The same pattern appears, less sharply, in the other models. The tables also include end-to-end response time, which is the fairer basis for judging speed. The median question took 3.1 seconds with Naive-N0.5-Flash (thinking), against 5.3 seconds for GLM 5.3 Flash (high) and 10.1 seconds for DeepSeek V4.1 Flash 1017 (high). The means are closer: 45.3 seconds for Naive-N0.5-Flash, 45.0 for DeepSeek V4.1 Flash 1017, and 86.9 for GLM 5.3 Flash. On the hard questions, Naive-N0.5-Flash (thinking) was slower on average than DeepSeek V4.1 Flash 1017 (high) , at 89.8 seconds against 79.7. Two caveats apply. The sample is small, at 10 questions per difficulty level, and we did not grade answer accuracy. Short, fast replies are only useful if they are correct. App experiments We als

Latent Space · 11 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
13–18 / 33
Trang 3 / 6