Vision-Language Models increasingly rely on textual reasoning to solve advanced multimodal tasks. Despite its effectiveness, this paradigm faces two fundamental limitations. First, relying on textual reasoning alone can
Firehose
Filtered to Papers · clear filters
Browse: People · Companies · Papers · Podcasts · Hacker News · Deep dives
Browse by tag
Organizations increasingly deploy downloaded open-weight GUI agents. A backdoored agent passes acceptance tests on clean data and misbehaves only when a visual trigger appears on screen, turning an unvetted checkpoint in
Pretrained Time-Series Foundation Models (TSFMs) provide strong representations for downstream classification tasks. Yet adapting a TSFM under changing operating conditions remains challenging. A model must improve on ea
Game2Player: Scaling Gameplay Experience for Visual Agents through Environment-Player Co-Development
This paper develops a framework called Game2Player that allows visual agents to learn from game environments and players at scale, by co-developing the environment and player through a loop of adaptation, task definition, and verification. Practitioners might care because it could help train more effective visual agents in a wider range of games.
This paper introduces Jeveration, a system that uses a fast and parallel prediction paradigm to generate high-quality images in a single forward pass, allowing for efficient and unified image generation, prediction, and other tasks. Practitioners might care because it could lead to faster and more efficient AI models for applications like image editing and object detection.
This paper develops a method to efficiently design neural processing units (NPUs) for large language models by using a mechanistic critic to evaluate and improve NPU configurations, allowing for faster and more informed design decisions. Practitioners may care about this research as it can lead to more efficient and effective NPU designs for real-world applications.
Large language models (LLMs) are increasingly used to infer psychological attributes, predict individual behaviour, and simulate human populations. However, despite generating plausible responses, they often misrepresent
Reinforcement learning from terminal answer rewards enables search-augmented language models to interleave reasoning and retrieval without step-level annotations, yet outcome supervision assigns identical credit to every
Automatic CAD generation is commonly evaluated with volumetric IoU, Chamfer Distance, and surface F-score, which barely penalize local errors that occupy only a small fraction of a part's volume or surface area, such as
This paper introduces a benchmark for automatic video rough-cut using large vision-language models, which can help improve the efficiency and quality of media production workflows. Practitioners can use this benchmark to evaluate and improve the performance of their own models on rough-cut tasks.
This paper proposes a new decoding strategy for looped language models, called late entry, which allows the model to skip initial passes and directly use a deep context cache, resulting in faster decoding times. Practitioners might care about this approach because it could lead to significant speedups in applications where decoding speed is crucial.
This paper develops a new AI system called PReMind that helps personalized vision-language assistants respond to visual hazards by considering a user's personal memories and context. Practitioners in the field of AI and robotics might care about this research because it aims to improve the safety and effectiveness of personalized assistants.
This paper proposes a system called WorldRuler that helps create interactive worlds with customizable rules, allowing for varied gameplay experiences. Practitioners in game development or AI research might care about this system because it can help them create more realistic and dynamic interactive worlds.
This paper proposes a method to improve the robustness of large language model agents against indirect prompt injection attacks by using a unified framework that combines red-teaming and adversarial training. Practitioners might care because it could help prevent agents from being hijacked by attackers and ensure they can complete their intended tasks.
This paper develops a new method for generating realistic 3D textures that accurately capture material properties, and how this might be useful for improving the realism of virtual objects in computer graphics.
Adapting a film into a game requires selecting source elements and translating them into player actions, rules, and feedback. We introduce Movie2Game-Bench (M2G-bench), a benchmark for evaluating film-to-game adaptation
This paper introduces Piccolo, a family of decision models that improve themselves by generating new data and adapting to that data, allowing for more accurate predictions on various tasks. Practitioners might care because Piccolo's approach could lead to more robust and adaptable AI systems.
This paper proposes a new method for training world action models, called HINT-WAM, which uses hindsight information to improve their performance on tasks that require reliable execution. A practitioner might care about this method if they want to improve the generalization and data efficiency of world action models in robot manipulation tasks.
This paper develops a framework to adapt large language models while preserving their general capabilities, allowing for selective unlearning and learning of new capabilities. Practitioners caring about model adaptation and generalization might care about this work.
This paper introduces AnyS1, a system that allows AI agents to make decisions more efficiently by leveraging human-authored procedural skills, reducing the need for complex model invocations. Practitioners might care about this research because it could lead to faster and more reliable AI decision-making in real-world applications.
This paper introduces StrideWorld, a benchmark that evaluates how well video world models maintain a stable scene structure and consistent actions when a user gives commands. Practitioners should care because it can help identify models that fail to maintain a realistic experience.
This paper introduces WaveView, a novel view synthesis method that improves upon existing techniques by injecting the warped source into the noise phase of a pre-trained generator, allowing for more accurate and efficient rendering of scenes. Practitioners in the field of computer vision and graphics may care about this approach as it has the potential to significantly improve the quality and realism of generated images.
Attention scores candidate supports of an answer, which we call witnesses, and reads the scores at a temperature. We build a higher-order attention layer whose state lives on pairs of items: for a pair (a, b), each third
This paper presents a system called QDA that uses an agent to design the best quantization recipe for large language models, allowing for faster and more accurate quantization. Practitioners might care about QDA because it can save time and improve model accuracy, making it easier to deploy large language models in real-world applications.
This paper introduces a new method called Decoding Without a Decoder (DWD) that allows diffusion transformers to generate final pixels directly, reducing complexity and inference cost. Practitioners might care because DWD can improve the quality of generated images and videos while reducing the computational cost of inference.
This paper proposes a new hypothesis about why agent evolution often fails to improve, and how to fix it. Practitioners might care because it provides a solution to a common problem in AI research, allowing for more reliable and efficient development of self-improving agents.
Deep models often rely on spurious correlations that fail on underrepresented groups. Existing methods either correct human-specified bias attributes or infer unknown biases from the learned representation space. However
This paper proposes a new method for generating controllable videos, allowing for fine control over the scene and characters while still being able to generate realistic motion. Practitioners might care about this research because it could enable more realistic and interactive video content, such as video games or virtual reality experiences.
Large language models (LLMs) are increasingly deployed as autonomous shopping agents that process multimodal product listings (consisting of visual and text data) and recommend purchases on behalf of consumers. Naturally
The goal of this paper is to reduce the input token cost of Omni-modal large language models (Omni-LLMs) at inference time. Omni-LLMs reason jointly over audio, video and text, but the cost of the three streams is highly
This paper develops a new approach to training agentic models to understand long videos, allowing them to find specific moments within hours of footage. Practitioners might care about this work because it could lead to more accurate and reliable video analysis in applications such as surveillance, healthcare, or autonomous vehicles.
We introduce Faithfulness Video Score (FVSCORE), a neuro-symbolic framework for specification-driven video generation evaluation. Existing evaluators rely on holistic MLLM judgments or vision-language metrics, yet aligni
This paper proposes a new way to use n-gram embeddings in language models, allowing multiple layers to share the same embedding table but adapt it to each layer's needs. Practitioners might care because this approach can significantly reduce the memory required for n-gram tables, enabling larger and more complex models.
Autoregressive vision-language-action (VLA) models predict discrete action tokens with the next-token objective of their vision-language backbone, so the action tokenizer defines the interface between the policy and the
Large language models (LLMs) have renewed interest in Page Stream Segmentation (PSS), splitting a multi-page file into its constituent documents. Yet published LLM methods still frame the task as boundary classification
This paper explores whether the Transformer architecture is still suitable for weight-space programming, a technique where a large language model writes target weights by hand without gradient-based training. Practitioners might care because it challenges the conventional choice of Transformers and could lead to more efficient and accurate models.
This paper introduces a unified AI model that can perceive 4D scene geometry and motion from diverse camera observations, such as images or videos. A practitioner might care about this model because it can help improve tasks like scene understanding, object tracking, and motion analysis in various applications.
This paper develops a system for robots to work together seamlessly, enabling them to accomplish more than individual robots can. Practitioners in robotics and AI might care about this work because it could lead to more efficient and effective collaboration in complex tasks.
This paper develops a unified model for robot manipulation that can track and predict robot actions and gripper states from RGB videos, and can adapt to new robot embodiments. Practitioners may care about this model for building more accurate and robust robot manipulation systems.
This paper investigates whether large language model (LLM) agents still need human guidance to self-improve, and whether they can develop their own learning strategies. Practitioners might care because understanding how LLMs can improve themselves could lead to more efficient and autonomous AI systems.
This paper creates a new type of video generation model that can produce frames quickly enough for interactive use, making it suitable for applications like real-time video generation in games or video editing. Practitioners might care about this because it could improve the user experience in interactive visual applications.
This paper develops a new series of large language models (Xing 4) by optimizing architecture and training infrastructure for efficient use on specialized chips (Ascend). Practitioners may care about the techniques described here because they could improve the performance and scalability of their own AI models.
This paper proposes a new lossless image compression format called FMT that can decode images in a fixed number of parallel steps, making it faster than classical codecs and learned codecs. Practitioners might care about FMT because it can potentially be used in applications where speed is critical, such as video streaming or image archiving.
This paper creates a system to train coding agents for long-term tasks by generating environments that provide timely and reliable feedback, and shows that this approach can significantly improve training results. Practitioners working on coding agent development might care about how to design more effective training environments.
Hyper-Connections (HC) expand the residual stream of a Transformer into N parallel streams, and manifold-constrained HC (mHC) stabilizes them at scale. However, mHC benefits little from more than four streams, partly bec
Multi-Teacher On-Policy Distillation (MOPD) enables a single student to absorb the skills of several domain teachers by learning from its own generated responses. We present Multimodal Model Multi-Teacher On-Policy Disti
Turning a CAD model into a machining program requires both process knowledge and geometric precision. Language models can generate such workflows by writing geometry and toolpath code for each part, but this makes numeri
This paper develops an open-source tool called CADMAGEDDON that can automatically reconstruct editable CAD designs from 3D meshes, allowing users to work with proprietary parts and run on local systems. Practitioners may care about this because it enables them to preserve their intellectual property and work efficiently with limited resources.
Visual foundation models are central to physical intelligence, yet their development remains constrained by fragmented training datasets, inconsistent annotation quality, and limited multimodal supervision. To address th
In structured Bayesian models, a large latent block is often Gaussian conditional on a smaller set of parameters, yet generic inference methods relearn this geometry during warmup. We study how this structure can be used
This paper introduces a new type of streaming video model that can handle user requests in a more interactive way, by being able to proactively monitor the situation, coordinate tasks, and ask for feedback. Practitioners might care because this could lead to more efficient and effective human-AI interactions in applications like customer service or virtual assistants.
Visible artifacts are an unreliable cue for detecting generated videos, because newer generators learn to remove them. We instead detect a trace left by the generation pipeline itself. Most video generators decode latent
This paper develops a framework to balance responding quickly with providing useful information in proactive streaming video agents, which is crucial for real-world interactive systems. Practitioners may care about how to improve the responsiveness and usefulness of their video agents.
High-resolution brain MRI is important for neuroradiological assessment, yet routine clinical protocols often rely on fast, anisotropic 2D acquisitions, particularly in resource-constrained settings. These acquisitions m
This paper investigates how large language models withhold responses under uncertainty or safety risks, and why this withholding (called abstention) is important for trustworthy models. Practitioners might care about understanding abstention to improve the reliability of language models in applications where they must make decisions that could have safety implications.
This paper develops a system that allows frozen models to control robots by incorporating contact forces, enabling manipulation tasks that rely on these forces. Practitioners may care about this research as it can improve the capabilities of frozen models in real-world applications.
This paper introduces MedJev, a vision-language decision model that improves medical image understanding by reducing unnecessary tokens and latency, allowing for faster and more accurate decision-making. Practitioners might care about this research if they work on medical image analysis and want to develop more efficient models.
Asynchronous reinforcement learning increases training throughput, but data from stale policies and numerical mismatch between training and inference can destabilize learning. To stabilize off-policy learning, score cent
This paper introduces a new framework called VideoNexus that allows for efficient video perception and planning by separating detailed video perception from evidence-driven planning and reasoning. Practitioners might care about how VideoNexus can improve the accuracy and efficiency of video perception and planning tasks.
Formal specifications define intended program behavior and underpin verification, but writing them in theorem provers such as Lean is slow and expertise-intensive, making specification a major bottleneck to scaling forma