DENIM: Ablating Catalog Access and Visual Grounding in an Agentic Fashion Shopping Assistant

Large Language Models now power e-commerce shopping agents that converse with customers, call retrieval tools, and recommend from live product catalogs. Building one means deciding how the product catalog reaches the agent: what it can query or read at inference time, and in what representation. We isolate two of these decisions: what kind of catalog access the agent needs, and in what form visual item information should reach it. To answer both questions, we first implemented an agentic conversational recommender for fashion e-commerce. It is designed as a single orchestrated system with tool-augmented retrieval over a large fashion catalog. We then ran controlled single-change ablations on it, evaluated with a leakage-controlled simulated shopper in a cold-start setting. First, structured catalog access is the dominant factor: removing it costs 31 points of success rate, and most of that value (27 points) comes from a catalog inspection tool that lets the agent see what the catalog contains and which attribute values are valid before filtering, a capability that recent systems include in various forms but whose effect on retrieval had not been isolated. Second, what matters about visual information is the form in which it reaches the agent: injecting item images at runtime adds no measurable success over offline-extracted visual attributes, while removing those attributes' structured rendering costs a further 11 points even though the same information remains available in prose descriptions. At matched item representation, the orchestrated pipeline also outperforms ReAct and Reflexion-style baselines built on the same backbone and tools by 12–20 points, sustaining preference elicitation where they stall. All findings replicate in direction across two simulator LLM families, one of which shares the recommender's own model family. We distill the results into practical guidance for building e-commerce shopping agents.