What AI recommends when you pick software: popularity, not fit
Hand selection to an AI and you often get a popularity chart rather than a fit for your context — dependencies get ignored and evidence stays opaque.
Key numbers
- The study frames software package selection as multi-criteria decision-making (MCDM), with automated pipelines continuously collecting metadata, usage trends, vulnerability data and developer sentiment from GitHub, PyPI and Stack Overflow.
- A decision model explicitly represents relationships among packages, domain features and quality attributes; an LLM interprets user intent and queries the model for candidate packages.
- Evaluation scale: 798,669 Python scripts from 16,887 GitHub repositories, plus a user study based on the Technology Acceptance Model.
- Results show high extraction precision and better recommendation quality than generative-AI baselines — while the paper names the common failure modes: ignored dependency evaluation, popularity standing in for suitability, and results that cannot be reproduced.
Method and sample
arXiv paper “Empirical Evaluation of AI-Assisted Software Package Selection: A Knowledge Graph Approach” (2025-08-06): the PySelect decision-support system structures software metadata, trends, vulnerabilities and sentiment into a knowledge graph and combines MCDM with LLMs, evaluated on 16,887 repositories, 798,669 scripts and a user study.
What it means for brands
An AI selection suggestion is not a review: it often treats popularity as fit and a single source as evidence. For SaaS and tool vendors that cuts both ways — if verifiable evidence about your product exists in directories, reviews and communities, the model is far more likely to shortlist you.
How SourceSight handles it
SourceSight groups prompts by how customers ask (which to choose, compare, pricing, credentials and compliance) to find the question types where you are entirely absent, and the off-site list covers the verifiable third-party evidence.
Limits
The study covers open-source package selection, which is not the same as enterprise SaaS procurement; its conclusions depend on specific data sources and evaluation methods and do not transfer to every software category.
References
Run it on your own project
What the same prompts return across 10 engines is only knowable by running them.
Related research
Cross-category recommendations: +36% add-to-cart on item pages, +15% on cart pages
Recommending general merchandise to grocery shoppers — a milk frother with the milk — has long been hard; LLM retrieval plus real-time cart context moved production A/B metrics.
Industries · 2026-10-07Home services: 22% of homeowners already use AI to find a provider, and 80% of businesses don't know how to be found
The local services funnel is moving upstream: before a homeowner opens a website or calls, AI has already synthesised reviews, reputation and service descriptions into one answer.