| cldr.py |
Digesting the CLDR's data.
Provides two classes:
CldrReader -- driver for reading CLDR data
CldrAccess -- used by the reader to access the tree of data files
The former should normally be all you need to access.
See individual classes for further detail.
|
49315 |
- |
| cldr2qlocalexml.py |
Convert CLDR data to QLocaleXML
The CLDR data can be downloaded as a zip-file from CLDR_, which has a
sub-directory for each version; you need the ``core.zip`` file for
your version of choice (typically the latest), which you should then
unpack. Alternatively, you can clone the git repo from github_, which
has a tag for each release and a maint/maint-$ver branch for each
major version. Either way, the CLDR top-level directory should have a
subdirectory called common/ which contains (among other things)
subdirectories main/ and supplemental/.
This script has had updates to cope up to v44.1; for later versions,
we may need adaptations. Pass the path of the CLDR top-level directory
to this script as its first command-line argument. Pass the name of
the file in which to write output as the second argument; either omit
it or use '-' to select the standard output. This file is the input
needed by ``./qlocalexml2cpp.py``
When you update the CLDR data, be sure to also update
src/corelib/text/qt_attribution.json's entry for unicode-cldr. Check
this script's output for unknown language, territory or script messages;
if any can be resolved, use their entry in common/main/en.xml to
append new entries to enumdata.py's lists and update documentation in
src/corelib/text/qlocale.qdoc, adding the new entries in alphabetic
order.
Both of the scripts mentioned support --help to tell you how to use
them.
.. _CLDR: https://unicode.org/Public/cldr/
.. _github: https://github.com/unicode-org/cldr
|
4673 |
- |
| dateconverter.py |
Conversion between CLDR and Qt datetime formats.
Keep in sync with qlocale_mac.mm's macToQtFormat().
The definitive source of truth is:
https://www.unicode.org/reports/tr35/tr35-68/tr35-dates.html#Date_Field_Symbol_Table
See convert() for explanation of the approach taken. Each method
with a single-letter name is used to scan a prefix of a text,
presumed to begin with that letter (or one Qt treats as equivalent
to it) and returns a pair (Qt format, length), to use the given Qt
format in place of text[:length]. In all cases, length must be
positive. |
7655 |
- |
| enumdata.py |
Assorted enumerations implicated in public API.
The numberings of these enumerations can only change at major
versions. When new CLDR data implies adding entries, the new ones must
go after all existing ones. See also zonedata.py for enumerations
related to timezones and CLDR, which can more freely be changed
between versions.
A run of cldr2qlocalexml.py will produce output reporting any
language, script and territory codes it sees, in data, for which it
can find a name (taken always from en.xml) that could potentially be
used. There is no point adding a mapping for such a code unless the
CLDR's common/main/ contains an XML file for at least one locale that
exercises it (and little point, even then, absent substantial data,
ignoring draft='unconfirmed' entries).
Each *_map reflects the current values of its enums in qlocale.h; if
new xml language files are available in CLDR, these languages and
territories need to be *appended* to this list (for compatibility
between versions). Include any spaces and dashes present in names
(they'll be squished out for the enum entries) in *_map, but use the
squished forms of names in the *_aliases mappings. The squishing also
turns the first letter of each word into a capital so you can safely
preserve the case of en.xml's name; but omit (or replace with space)
any punctuation aside from dashes and map any accented letters to
their un-accented plain ASCII. The two tables, for each enum, have
the forms:
* map { Numeric value: ("Proper name", "ISO code") }
* alias { "OldName": "CurrentName" }
TODO: add support for marking entries as deprecated from a specified
version. For aliases that merely deprecates the name. Where we have a
name for which CLDR offers no data, we may also want to deprecate
entries in the map - although they may be worth keeping for the
benefit of QLocaleSelector (see QTBUG-112765), if other
locale-specific resources might have use of them.
For a new major version (and only then), we can change the numbering,
so re-sort each list into alphabetic order (e.g. using sort -k2); but
keep the Any and C entries first. That's why those are offset with a
blank line, below. After doing that, regenerate locale data as usual;
this will cause a binary-incompatible change.
Note on 'macrolanguage' comments: see QTBUG-107781 and 'ISO 639
macrolanguage' on Wikipedia. A 'macrolanguage' is (loosely-speaking) a
group of languages so closely related to one another that they could
also be regarded as divergent dialects of the macrolanguage. In some
cases this may mean a resource (such as translation or text-to-speech
data) may describe itself as pertaining to the macrolanguage, implying
its suitability for use in any of the languages within the
macrolanguage. For example, no_NO might be used for a generic
Norwegian resource, embracing both nb_NO and nn_NO.
|
48562 |
- |
| formattags.txt |
|
580 |
- |
| iso639_3.py |
Representation of ISO639-2 language code data.
|
3257 |
- |
| ldml.py |
Parsing the Locale Data Markup Language
It's an XML format, so the raw parsing of XML is, of course, delegated
to xml.dom.minidom; but it has its own specific schemata and some
funky rules for combining data from various files (inheritance between
locales). The use of it we're interested in is extraction of CLDR's
data, so some of the material here is specific to CLDR; see cldr.py
for how it is mainly used.
Provides various classes to wrap xml.dom's objects, specifically those
returned by minidom.parse() and their child-nodes:
Node -- wraps any node in the DOM tree
XmlScanner -- wraps the root element of a stand-alone XML file
Supplement -- specializes XmlScanner for supplemental data files
LocaleScanner -- wraps a locale's inheritance-chain of file roots
See individual classes for further detail.
|
35115 |
- |
| localetools.py |
Utilities shared among the CLDR extraction tools.
Functions:
unicode2hex() -- converts unicode text to UCS-2 in hex form.
wrap_list() -- map list to comma-separated string, 20 entries per line.
Classes:
Error -- A shared error class.
Transcriber -- edit a file by writing a temporary file, then renaming.
SourceFileEditor -- adds standard prelude and tail handling to Transcriber.
|
8933 |
- |
| qlocalexml.py |
Shared serialization-scanning code for QLocaleXML format.
Provides classes:
Locale -- common data-type representing one locale as a namespace
QLocaleXmlWriter -- helper to write a QLocaleXML file
QLocaleXmlReader -- helper to read a QLocaleXML file back in
Support:
Spacer -- provides control over indentation of the output.
RelaxNG schema for the used file format can be found in qlocalexml.rnc.
QLocaleXML files can be validated using:
jing -c qlocalexml.rnc <file.xml>
You can download jing from https://relaxng.org/jclark/jing.html if your
package manager lacks the jing package.
|
48937 |
- |
| qlocalexml.rnc |
|
5746 |
- |
| qlocalexml2cpp.py |
Script to generate C++ code from CLDR data in QLocaleXML form
See ``cldr2qlocalexml.py`` for how to generate the QLocaleXML data itself.
Pass the output file from that as first parameter to this script; pass the ISO
639-3 data file as second parameter. You can optionally pass the root of the
qtbase check-out as third parameter; it defaults to the root of the qtbase
check-out containing this script.
The ISO 639-3 data file can be downloaded from the SIL website:
https://iso639-3.sil.org/sites/iso639-3/files/downloads/iso-639-3.tab
|
52426 |
- |
| README |
locale_database is used to generate qlocale data from CLDR. |
727 |
- |
| testlocales |
|
|
- |
| zonedata.py |
Data relating to timezones and CLDR.
This is not implicated in public APIs, so may safely be changed.
Contrast the enumdata.py, where public API is implicated.
Scripts digesting CLDR data shall report the updates you need to make,
if any arise.
The windowsIdList is a list of twoples (ID, offset), associating each
Windows-specific ID for a zone with that zone's offset from UTC, in
seconds. Entries are sorted in case-insensitive lexical order by
ID. If a script reports that it has found a Windows ID not listed
here, research the relevant zone's offset and add a new entry to the
list of twoples, preserving the ordering. Internet search engines and
timeanddate.com can help with researching the offset. Note that some
UTC offset zones (giving only the hour) are present in windowsIdList.
When adding an entry to windowsIdList, check whether its offset
corresponds to that of some entry in utcIdList; if not, add such an
entry.
The utcIdList is a simple list of various UTC-offset names. Aside from
'UTC' itself, shared with windowsIdList, these include minutes in
their offsets even when they are whole hour offsets. The list contains
the UTC-equivalents of all offsets seen in the windowsIdList, plus the
whole hours out to ±14 hours, the two verbose forms of UTC±00:00 and
any legacy entries from past Windows zone offsets. Entries should not
be removed, even if the relevant Windows ID becomes obsolete or
switches to a different offset, as they make up the available zones of
the UTC back-end. (That recognizes other offset zones, and its
is-available check will accept them, but it leaves them out of its
list. There are, after all, thousands of possible offset zones, but
relatively few are widely used.)
Note: -00:00 (without the UTC prefix) was introduced in RFC3339 as a
way to indicate that a date-time has been converted to UTC but its use
should not be understood to say anything about the local time of the
origin of the message using it. However, ISO 8601 has, since 2000,
forbidden this as an offset suffix. The more recent compromise is to
use Z to convey the meaning RFC3339 gave to -00:00. So the use of
-00:00 as offset suffix should be avoided (and, by extension, likewise
for UTC-00:00 as a zone ID), but this suffix (and ID) should be
recognized when consuming data generated by other sources, for
backwards compatibility.
|
10472 |
- |