MediaWiki:Scribunto/Central guideline

La bibliothèque libre.
THIS COOPERATION FOR A CENTRAL MODULES GUIDELINE IS IN A ROUGH DESCRIPTION STATE
Some parts are empty or missing
This document will be the result of a cooperation and contains first more questions than aswers.

Documentations

Central libraries in brief

In this document, Library: is not a namespace, only a way to shortly express that an object is understood as a library.

The central system permits to convert any module as central.

The Lua part of central modules support contains some libraries used by central modules.

The Module:Central installs central libraries (waiting a stable Scribunto central repository).

Central libraries are intimately dependent of each others. Examples: the Library:versions installs others and the Library:viewer is usefull anywhere.

Stability

Central modules specially must be stable. Some supports for stability:

  • Central modules must follow S.O.L.I.D recommendations. Lua-coders can ask help from their pairs if necessary, see https://en.wikipedia.org/wiki/SOLID_(object-oriented_design)
  • Which kind and level of stability do we need to begin to use central modules?
  • Which kind and level of stability do we need to really centralize?
  • How to insure a hight level of stability?
    • Run many tests for key functions.
    • Display groups of tests for Lua-coders. Then they can detect errors or missing tests cases.
    • Notify and alert central Lua-coders using errors and categories.
    • Document all functions supporting stability and form a list of all actual fonctions to not miss some one's.
    • Save and restore main configurations for each group of tests to better avoid mutual perturbations between them.
    • Display in live tests in any documentations pages.
    • Search other means to increase the stability and report them.

Stability guideline

Some questions are especially important for all developers in the early phase of the deployment of central modules.

Libraries and functions names must be stable before a large deployment to reduce the renaming of them. That could imply heavy corrections after the begin phase.

Stability questions:

  • How to name central libraries ?
  • How to name functions in these libraries ?
  • How to distribute the functions in one or some libraries ?

Stability tasks

  • Same tests of functions are available for Mediawiki testframework and for Lua-coders in tests mode.
  • The versions library installs sought sub-modules using versions management.
  • Drop boxes and tables views help to easy form new tests.
  • The activity library help to manage sub-tasks and phabricator tasks, between Lua-coders, projects and languages.
  • The tracker object supports parametrable track objects to help to debug the Lua code.
  • The events library forms events as errors, warnings and categories.",
  • The testframework library reports differencies in attended results from functions. It reports recursively at tests cases level for groups of groups of tests cases.
  • The deprecated functions support the detection of their uses.
  • Any detail in any page can crash whole the page. An antiCrash function can protect against that fail.
  • In many parts in the Lua code, strict type checks reduce errors risks.
  • The initial development in the same text file enhance the cohesion in values, variables and functions.
  • The function save_configs() saves global configuration before eventual changes in tests. And restore_configs() restore it.

Services management

The central system permit to support some new services.

Languages management

Central modules support translations in many languages, using i18n tables.

They support also the content, page and user languages, depending of complex reasons in each case.

The system can also suggest a default local language depending where the module is used.

For these goals, central modules get these languages from the system, modes, options, arguments and tests cases.

Translations management

The langs library supports i18n translations.

  • It also detects missing translations between languages.
  • It merges translations together from libraries, sub-modules and the main module.
  • It also detects the translations changes in the main module compared to more inner others translations.

In i18n tables, each translation key begin with the module or library name, to not disturb other and future translations.

An alternative version can modify any translation of the normal version.

A main module can modify any translation from any submodule to adapt it to a new context.

A /I18N submodule can adapt any translation from a normal or new version.

  • Each module or sub-module has its own /I18N submodule.

Then the versions.bind_modules(p)function follows these rules:

  • Starts with the libraries translations,
  • starts with the internal translations,
  • then add or mix translations from normal /I18N,
  • then add or mix translations from the /I18N alternative versions,
  • Continue with the modules translations,
  • starts with the internal translations,
  • then add or mix translations from normal /I18N,
  • then add or mix translations from the /I18N alternative versions.
  • In doc and tests modes, a report display missing or modified translations.

Versions management

To activate the versions management of a module, start it with a definition of alternative versions of its sub-modules, like this:

p.versions = { -- Modules dependencies. Dependencias del módulo. Dépendances du module.
	versionName = "Main", versionNumber = "1.03", versionDate = "2016-04-27 23:23",
	sought = "Main, Box 2, Item.3", -- Sought module and submodules versions
	known = " Main,Main2 * Box,Box 2 * Item,Item.2,Item.3 ", -- Known module and submodules versions
}

Proposed guideline for versions numbers: They are of the form major.minor.release.

The major number is incremented when there's a big change in functionality.

Template or invoke arguments can replace soughtversions and knownversions to try them without record, or in a test template.

In known versions, the first of each group of versions is the normal one and can replace any other if not available.

The versions management require() adequate versions of submodules, following known and sought versions, like Module:Name.2.4.

This versions management reports versions: used, errors, missing or replaced by normal ones... then Lua-coders can better select and understand what happens.

Any Lua-coder of main module can select versions of its submodules from internal arguments or from template arguments.

When the developer copies or rename this file in a new version, he can forget to adapt that definition and then disturb the versions management.

To reduce this risk, each module need to know its own name.

To document the version date-time, we need that for each module or sub-module.

See the task T119978: Get the own module name and last record date-time from each module Erreur de script : le module « Centralizer » n’existe pas.


Some related local pages

Draft documentation: Central modules .
Draft central libraries: Module:Central .
Simple examples of uses: Module:Central/Documentation
Full example: Victor_Hugo .

Cooperate for central modules happening

We can already build a cooperation convention from following elements:

Many discussions have already happened in many places.
As for many other projects we probably need a Space:Central modules, a Project:Central modules in each project, a main task and some others, a user manual of central modules, and descriptions of these elements.

Subjects to build and stabilize:

Is the starting point here? Under which name?
How to share tasks?
Inform each other Lua-coders in discussions pages about the part/s each user take in charge. Perhaps synthesis tables?
Main dedicated task: T20170202 Begin to use and support central modules
Discussion
Link in all documents and discussions to all other documents and discussions.
What happens to begin for central modules? Until when?
What happens waiting a central repository? Until when?
What is in the central repository? Running objects? Documentations? Discussions? Organisation?
How coordinate the content of the central repository with the versions in all other projects (tests, try, enhancements...)?

Shared Task T135845 : Convert any module as central or centralisable

This task started as a one-man task but becomes a shared task to stabilize and enhance it in a long term perspective.
Then some developpers can start de diffusion and many people can benefit form central modules, including developpers themselves.
Scribunto can convert any module as central, or even centralisable before an efficient central repository exists.
A central module can help users and helpers, in their own languages and the wiki languages for arguments, errors, warnings, categories, documentations and tests.
The main part of this task is to support new services:
  • Translations management for modules arguments, errors, warnings, categories, dropboxes titles, tables headers...
  • Events management to collect, display and activate errors, warnings and categories.
  • Arguments management[1] of modules: required, limited to multiple values, error if re-defined, unknown to rename, auto-rename if not ambiguous...
  • Display management to adapt the result of modules at some cases: normal read, edit, document or test of modules.
  • Versions management for modules and libraries, based on sought, availables and used versions, with errors messages.
  • Tests cases management, for stable modules and libraries, available for gerrit but also at modules level. These tests permit at any LUA coder to display, better understand functions, verify and even define missing or failing tests.
  • Easy viewers availability for parameterized texts, dropboxes, tables with columns and rows, easy styles changes...
[1] The argument management probably needs a reciprocal coherence/compatibility with mw:Extension:TemplateData . To do.

Shared Task T20170202 : Begin to use and support central modules

This task is not a one-man task but a shared task to begin a long term use and support of central modules. It will finish only when there will be at least 10 central modules realy used, 10 Lua-coders who use them, 10 large projects using them, small projects in 10 languages using them.
For these survey, the support of central modules:
  • Displays the translated languages management and their counts.
  • Detects and display changed and missing translations.
  • Displays a link to the count of admins in each project.
  • Detects and displays arguments missing of translations.
  • Records in categories their uses.
The draft proposition for this task:T20170202 is in fr.wikisource Central modules.

Central repository for central modules

Waiting a central repository
In central repository
These new services permit to use central modules in many languages, in many projects, even in many small wikis without enough helpers.
  • That provides an easier and more efficient peer review.
  • That enhances the mean abilities of modules for a same global effort.
  • That reduces the duplication of efforts for a same result.
  • That increases the mean availability of helpers and LUA coders.
On novembre 2016, the task is implemented in libraries centre, events, luaTable, tools, string, testsCases, viewer. Other parts permit to essay or to use them. All these libraries are in the standalone Module:Central.
 
The community can implement these new functions at any level, or any mixes: mediawiki, extensions, libraries, modules, gadgets, user scripts.
Compared to git, the implemention in modules is more easy and available for many more LUA coders.
 
The proposition / description for this task:T135845 is in Extension:Scribunto/Central_modules .
A prototype tries these services from january 2013 in some modules.

Share tasks

  • This task T135845 is enough advanced to share it.
  • That means enough defined, explicit and debug to permit at any developer to take a part of this task and to work on it in coordination with others.
  • Then we can/must choose how to cooperate .
  • That means we can/must choose pages to discuss our works and changes .
  • This process also is a long term task for many central modules.
  • As a first step, on 2016-06-02, the initial developper creates this draft page to begin this process, and leave to others to take their parts of it.
  • On 2016-06-02, the initial developer Rical works on MW TestsCases .
  • Any developer can already Convert a module as centralisable.

See also the tool_policy in MW.

For these goals, we can offer some new services, which are the main of this task:
  • Translation management for modules arguments, categories, error and warning messages, table headers...
  • Collect and display management for same ones.
  • Arguments management of modules: needed, limited values, limited at multiple values, error if re-defined, unknown to rename, auto-rename if not ambiguous...
  • Display management to adapt the result of modules at some cases: normal read, edit, document or test modules.
  • Easy viewers availability for parametric texts, dropboxes, tables with columns and rows, easy styles changes...
  • Versions management for modules and libraries, based on sought, availables and used versions, with alert messages.
  • Tests cases management, for stable modules and libraries, available in gerrit but also at modules level. That permit at any LUA coder to display, verify and even define missing tests.
We can implement these services at any level, or mixes: mediawiki, gadgets, user script, libraries, modules.
A prototype try that services from january 2013 at module level.


This task try to distribute implementations using mediawiki, scribunto, modules and interactions between them.
LUA coders need to define services, libraries, objects, fonctions and options.
LUA coders need ways in modules to form and manage parametrable translations, viewers, catégories, errors.
Each module has its own translations, catégories, viewers, and chooses for each text a language(wiki, user or fix) for each object.


Tests cases stabilize libraries and modules; LUA coders (with adhoc rights) can also view, verify, change and extend them.
Any LUA coder can try any change on actual libraries, at any time.


In actual development, 2016-06, all new services come from new centre and tools libraries.
But of course, a more advanced implementation will distribute services in mediawiki, scribunto and modules.


This task began in january 2013 to enhance the module fr.wikisource//Module:Author3.

Task: Convert any module as central or centralisable


:Scribunto permit to convert any module as central, or even centralisable before an efficient central repository exists.
:That permits to use central modules in many languages, in many projects, even in many small wikis without enought helpers.
:*That provides an easier and more efficient peer review.
:*That enhanses the mean abilities of modules for a same global effort.
:*That reduces the duplication of effort for a same result.
:*That increases the mean availability of coders and helpers.
:A central module can help users and helpers, in their own languages and the wiki languages, by errors, warnings, categories, reports and tests.


:*Example: display a category in the helper language and link to the category in wiki language.
:*Example: for each project and each language, choose detailed categories names or only categories "users modules errors" and "internal modules errors".

A central module must be very stable, understandable and strong against small usual misuses.
For that it permits managements of arguments, wikidata, errors, messages and modules versions.
The draft proposition for this task:T135845 is in fr.wikisource Central modules reference manual.

What is a central module?

From actual live developments we can at least say:

It is able to use translations, in some or many languages, of its arguments names, catégories names, and errors messages that it produces.
Translations are in a /I18N submodule to not disturb it, except for minimal internal translations.
It is able to automaticly use any available translated languages.
It is able to use the local wiki language and the user language at the same time.
It can use a version management ability, at local and at central repository.
It can work at local like at central repository, for local tests without disturb central version, or for uses waiting central availability.
It is stable and contains necessary MW tests cases.
It is protected against too easy edit for a better stability.
If sought, it is able to help readers, helpers and developers in doc, edit and tests mode, showing internal documentation, reports and tests of complex basic functions.
These abilities come from one or some scribunto libraries, depending of the complexity of the module an its submomules.
Its uses of central libraries must be very simple in any simple or complex modules.
Functions verify parameters types and replace missing parameters by default values.
Functions are strong enough to accept little calling faults when non ambiguous.

How to support central modules?

Like modules in talk pages? And like phab tasks? And sharing tasks?
Use a common sharing page: Talk:Central modules?
How to name a Lua developer? Developer? Coder? Lua coder?
Use the versions management, in tests places or in final places.
The rights of Global interface editors could give global ability.
To ask in Requests for other global permissions could give global ability.
The rights of Interface editors could give global ability only from local login.
Waiting an efficient central repository, some bots could repeat validated centralisable modules in some places where active coders experiment them.

Easier, faster and stable development

Keep in one module the example of central module, and its libraries until they will be stable:
  • to easy change groups of functions in libraries, or in object,
  • to easy coordinate changes in several parts of the code,
  • to easy install and update all central libraries in a new place where use the code or where change it.


: In development phase, to show the abilities of central modules, keep some languages, errors, messages, categories...


: Alias of functions insure the continuity of service across change in sub-module, then in calling module, then in sub-module to delete the alias.

function tools.maini18n_lang_v_t(i18n, txt) -- Deprecated name of the function
	return tools.form_i18n_counts(i18n, txt) -- Better name or change in 
end -- Count available translations and languages
In development phase, when many functions or even some libraries are not well defined, put them in the same module and re-distribute functions and parameters.
function centre.init(frame, args_known, mode_name, options_for_modes, itemid) -- In an easier function, group some others, before change in calling module.
	tools.init_tools(frame, options_for_modes)
	centre.bindlibraries() -- Install central libraries, waiting true central libraries.
	centre.bindmodules(p)
	tools.get_arg_mode(mode_name, source_key, try_lang, args_source)
end

: To try evolutions in a module or restructuration of some ones, use the [[#Versions management|versions management]].

<source lang=lua>
p.versions = { -- Modules dependencies. Dependencias del módulo. Dépendances du module.
	versionName = "Central", versionNumber = "1.03", versionDate = "2016-02-21 23:53",
	sought = "Central01,Central 2,mathroman", -- Sought module and submodules versions
	known = " Central01,Central 2 * mathroman,mathroman3 ", -- Known module and submodules versions
}

Short story of development

  • The previous templates was based on the oldest LISP programmation langage style.
  • On 2012-04-16, Mediawiki offered Lua in Scribunto. This language is very simple to use, but very complete, all in objects, with the "meta" table to replace the words "inherited" or "class" (even sub-threads out of Mediawiki).
  • Of course then, many users began to convert each old templates in new Lua modules which was simple due to the already existing LISP-like templates.
  • But the previous French Template:Author was already a complex template, needing 6 sub-templates, see "liés".
  • On 2013-01-18, i18n tables was used to translate errors.
  • On 2013-01-25, in the Module:Author, the generic management of arguments, and the table to group their definitions, came from its large number of arguments, see "args_known".
  • On 2013-03-12, the Module:ControlArgs groups complex translations and management of arguments to support other modules to become centralisable.
  • On 2015-02-25, the Module:Author3 include the versions management for a better stability.
  • On 2016-01-30, the Module:Central convert modules in libraries.
  • On 2016-05-20, T135845: a task propose to "Convert any module as central or centralisable".
  • On 2017-02-02, T20170202: a task propose to "Begin to use and support central modules".

State of development

In phase of development, some errors are needed to show them in some reports and to choose their better places.
Do not correct these errors, or even activate some ones.
Central translatable libraries are: centre and tools.
Each translatable library contains internal translations in english, french and spanish.
The library mathroman is a very small exemple of translatable library, only detects errors and has no mode nor options.
The centre library binds modules in sought versions management and easy translates parametrable strings.
The tools library support simple to complex modules.
Their supports are to translate, collect, manage and display views.
Translate is for arguments names from template, for images and texts on page and for categories and errors.
Waiting final true libraries, for local or central developments, the libraries and their translations can come from Module:Central and Module:Central/I18N.
Where to place the i18n translations out of the final libraries to not disturb it when translations change?
Proposition: put the centre library in Module:Library/centre/I18N.
The arguments management collects and keeps strings to translate and their parameters. Later also display lists of translated strings in right places (warning, errors, categories, categorize).
Manage arguments means also coming from wikidata, help for better argument name orthograph, auto correction of arguments names when non ambiguous, alias arguments names, verify if value is in the parametrable list of values, needed but absent, warning when twice defined, arguments descriptions.
The versions management require() adequate versions of submodules to test, following known and sought versions, like Module:Vers2.4.
This versions management reports versions used, errors, missing or replaced by normal ones... then helpers can better select and understand what happens.
Any helper of main module can select versions of its submodules from internal arguments or from template arguments.
Central libraries permit to easy generates categories, single or in group, translatable dropboxs, tables views, tests views...

Unavailable or sought abilities

on 2016-05-20

  • Central libraries are NOT stable and need to contain necessary MW tests cases.
  • For MW tests cases, we can also display the results in tableviews, like for many other tests.
  • Auto correction of arguments names when they are non ambiguous (see available levenstein).
  • Waiting the tasks debug, config arguments simulate informations depending of blocking tasks.
  • In the list of available languages, when it is large, we could display first a list of "near languages", like tools_near_languages = "fr,es,it", defined in i18n translations for each language. Others are in alphabetic order of languages codes. See tools.maini18n_languages_list() which returns "This module can translate 171 texts in 3 languages: English(en=English), español(es=Spanish), français(fr=French),".
  • For MW tests cases of the logic of centre.bindmodules(), replace existing modules by a simulation of package.loaded.
  • Make a testviewer based on MW tests cases.
  • Main options for dropbox (title, content, image) also could be in the style options.
  • Tests of multiple values arguments must be explicit.

Erreur de script : le module « Central » n’existe pas.

Tasks to open?
  • T00000: Add module and library types in scribunto ?
  • T00001: Sorted tableview do not sort date-time, in template and in module.
  • T00002: In a dropbox after \n* the content is after the box. Better define the case. Insert "\n------" before correct that. Try that out of fr.wikisource, in test2 or mediawiki...
  • T00003: bindI18N() after bindmodules to protect sought translatios against un-sought ones.
  • T00011: Sorted tableview do not sort date-time
  • T00012: In a dropbox after \n* the content is not inside the box. Better define the case. Insert "\n------" before to correct that.
  • T00013: Insert and mask structured datas and errors in resulting wikitext for bots. In xml or json or template style.
  • T00014: Display changes in test_init() arguments, in a dropbox.
  • T00015: Convert some groups of functions as new objects types: tableview with its headers and options, luatable() and its luatablecount(), dropbox() and dropdownfunc() and their options, get{} output from centre.get_module_and_time() for modules and libraries.

Blocked tasks

T41610: Scribunto should support global module invocations
T127354: Consider helping some smaller Wikipedias' communities
T122086: RFC: Sharing templates and modules between wikis - poor man's version (investigation)
T89598: Convert Template:Authority_control to Lua and use Wikidata

Blocking tasks

Some blocking tasks are detected by the module itself.

{{#invoke:Central | doc | tools_tasks_table_report_title }} Erreur de script : le module « Central » n’existe pas.

Important support tasks

T53660: Detect the edit state to adapt help messages to user errors.
  • To show edit or test mode only in edit state.
T68051: Get the user language to help admins to maintain a module or a template
  • Permit to use the same module in many languages in many projects. This enlarge the use of modules in many more small wikis. This also enhanse the mean abilities of modules for a same global effort.
  • Help helpers, from large wikis in their own langages, to help small wikis in other languages. Example: display a category in the helper language and link to the category in wiki language.
T119978: Get the own module name and last record date-time from each module
  • This task is necessary to reduce human errors:
  • to rightly collect /I18N translations,
  • and for a more confident versions management.
  • search this task id below for details.
T122752: #invoke seems do not record the module
  • This task is a true bug which impacts the versions management.
  • But the centre library correct it for the main module. Other modules come by require() and are not impacted.
  • In most other uses, this bug has no impact.
See
  • tools.task_T122752_get_state() -- Detect T122752 : #invoke seems do not record the module
  • tools.task_T122752_correct_once() -- Debug T122752 : #invoke seems do not record the module

Central repository tasks

  • These tasks impact the final stage of true central usage.
  • But the full development of central libraries is possible without central repository.
  • And the use of central libraries is possible in any local wiki with exactly the same modules, and only new i18n tranlations.
T121470: Central Global Repository for Templates, Lua modules, and Gadgets
T52329: We need a common repository for Scribunto modules and templates
T41610: Scribunto should support global module invocations

Marginal tasks

  • These tasks are interesting to optimize somefunctions but have no link to any central aspects.
T48160: Lua: Allow using labels in mw.wikibase.entity.formatPropertyValues
T63958: Use existing $dateFormats to format dates on Wikidata
T85412: The formatPropertyValues needs an option to format data from date-time, angle or geographic coordinates
T75460: [Story] Make Luafunctions default to the user's language on multilingual wikis
T85419: User groups should be exposed to Scribunto ( This permit to adapt errors and warning for administrators or modules coders )

Actual debug or detection in code waiting resolved tasks

Some bug are temporary debug in actual code, staring with .debug:
  • centre.debugT68051_status, "Give at modules the user language to display errors and categories names for him"
  • centre.debugT122752_status, "#invoke do not record the main module in package.loaded"
  • centre.debugT133498_status, "Any title in wikibase can give its item id to modules"
  • testsCases.debugT20160616_status, "A function call lost a table argument in a multilevel table"