Isn't this the very definition of the Wikidata-Query-Service, which is now deployed in production, therefore meaning this is resolved?

This is about being able to use the query service from a wiki via a Lua module.

In T67626#1900322, @JanZerebecki wrote:

This is about being able to use the query service from a wiki via a Lua module.

I've updated the task title to reflect this.

As one of the possible solutions for this: https://commons.wikimedia.org/wiki/User:TabulistBot

Not sure whether it closes the task, but at least it implements a major part of it.

@Lydia_Pintscher - I wonder if this may be enough?

I am noticing that there's no use of the TabulistBot at all. So I wonder if the use case is there...

Isn't this mostly supported by https://www.wikidata.org/wiki/Wikidata:Listeria? What are the features missing?

Listeria is a 3rd party tool, and we need a feature built-in Wikimedia. Also Listeria does not store the result in any structured way.

Listeria is a 3rd party tool, and we need a feature built-in Wikimedia.

What makes you think so?

In T67626#7620447, @Aklapper wrote:

Listeria is a 3rd party tool, and we need a feature built-in Wikimedia.

What makes you think so?

Another issue to consider is by updating Listeriabot make an edit to each page, and this clutters page histories for pages with embedded lists. It is much worse if the wiki host thousands of Listeria-generated pages, many of them are barely viewed but refreshed frequently, like cywiki. Snapshots are sometimes useful (T291091: Snapshots for saved queries), but in most times, they are not.

For some queries frequently used, there are various types: one is those that should be regularly refreshed (currently ListeriaBot do it but many pages are not refreshed as frequent as the page defined); one is those that may be refreshed on demand; one is those relatively stable, such as a list of chemical elements or US presidents. For the last type, it is useful to introduce a semi-permanent result and users who want to refresh it may see the diff between result sets. This also keep off most vandalisms, and may even become a way to moniter vandalisms (with some mechanism to regularly check query results). These three types are stored queries.

Also, they may be a parser function to invoke an arbitrary query on-the-fly, probably via some caching mechanism. This may be part of Wikilambda. In addition, there may be a "query template" mechanism to handle some sets of queries whose only difference are some parameters (one example: people died on <date>).

In T67626#7620208, @Bugreporter wrote:

Listeria is a 3rd party tool, and we need a feature built-in Wikimedia. Also Listeria does not store the result in any structured way.

I want to make one point deeper: The current Listeriabot is 100% not a good design - it does multiple things at once.

First we can run a SPARQL query and we have a (raw) result set. secondly Listeria does some basic transform - both automatic and manual: You can look up labels, descriptions, aliases, some statements/qualifiers (this is not relied on SPARQL) and then since 2.0 the values are automatically formated (i.e. turned to links) which (it seems) there are no parameter to disable. Thirdly the results are formated into some table or template calls.

Ideally we should made them three independent part: the query part may be cached in ways described above; result the transform part may be cached, possibly per language if results in interface language is requested (if no terms is requested, or only monolingual terms is required which can already queried in SPARQL, this step can be skipped entirely); and the content is displayed using the cached result.

(If there are multiple possible transforms of one query, a further optimalization may be caching query results by some sorts of hash, or make transforms a top-level object, though the latter may be confuing to some users.)

They are entered in a page in the Query namespace and internally saved as JSON.

Why JSON? We have WDQS, which uses SPARQL, so it seems straightforward for me to store these on-wiki queries in SPARQL as well. (The cached results should probably be JSON, as it’s easier to consume and format for display, but for the queries themselves we don’t need to reinvent the wheel by creating a JSON schema.)

Another feature Listeriabot missing: Listeriabot always assumes that each entry is a item. Thus it is not easily usable for lists of properties, lexemes or non-entities (such as dates). Hacks exist, but they are merely hacks, not proper solution.

If we want to support this in 3rd party wikis, a solution that is not highly coupled with Wikifunctions is recommended. This may be:

• A feature to store specific query on a query namespace and cached results in some storage outside of WikiLambda, plus a feature to run arbitrary query on-the-fly via Wikilambda, and/or
• Extracting the async rendering feature of Wikilambda to some other extension and the complex query service will be based on such extension

In https://wikitech.wikimedia.org/wiki/User:DCausse/WDQS_Graph_Split_Impact_Analysis it is mentioned that Listeria is heavily throttled by WDQS (only 1.33% success rate). It is therefore not a scalable approach.