.. ================================================== .. Header hierarchy .. == .. -- .. ^^ .. "" .. ;; .. ,, .. .. --------------------------------------------used to the update the records specified ------ .. Best Practice T3 reST: https://docs.typo3.org/m/typo3/docs-how-to-document/master/en-us/WritingReST/CheatSheet.html .. Reference: https://docs.typo3.org/m/typo3/docs-how-to-document/master/en-us/WritingReST/Index.html .. Italic *italic* .. Bold **bold** .. Code ``text`` .. External Links: `Bootstrap `__ .. Internal Link: :ref:`downloadButton` (default url text) or :ref:`Download Button` (explicit url text) .. Anker for internal link: .. _`download`: .. Add Image: .. image:: ./Images/a4.jpg .. .. Add image with caption: .. figure:: ./Images/black_dot.png .. :class: with-border .. :width: 20px .. Add Attachment (place in Documentation/_static): `Download CSV <_static/yourfile.csv>`__ .. .. Admonitions .. .. note:: .. important:: .. tip:: .. warning:: .. Color: (blue) (orange) (green) (red) .. .. Definition: .. some text becomes strong (only one line) .. description has to indented .. .. Code Block: .. code-block:: sql .. .. -*- coding: utf-8 -*- with BOM. .. include:: Includes.txt .. _`system`: System ====== .. _`autocron`: AutoCron -------- The `AutoCron` service fires periodically jobs like `open a webpage` (typically a QFQ page which does some database actions) or `send mail`. * AutoCron will be triggered via system cron. Minimal time distance therefore is 1 minute. If this is not sufficient, any process who starts `.../typo3conf/ext/qfq/Classes/External/auto-cron.php` via `/usr/bin/php` frequently might be used. * Custom start time and frequency. * Per job: * If a job still runs and receives the next trigger, the running job will be completed first. * If more than one trigger arrives during a run, only one trigger will be processed. * If the system misses a run, it will be played as soon as the system is online again. * If multiple runs are missed, only one run is fired as soon as the system is online again. * Running and processed jobs can easily be monitored via *lastRun, lastStatus, nextRun, inProgress*. Setup ^^^^^ * Setup a system cron entry, typically as the webserver user ('www-data' on debian). * Necessary privileges: * Read for `.../typo3conf/ext/qfq/*` * Write, if a logfile should be written (specified per cron job) in the custom specified directory. Cron task (user cron tab): :: * * * * * /usr/bin/php /var/www/html/typo3conf/ext/qfq/Classes/External/auto-cron.php AutoCron Jobs of type 'website' needs the php.ini setting: :: allow_url_fopen = On Remember: if a cron job fails for whatever reason, the cron daemon will send a mail to the userid who started the cron job. E.g. www-data. Setup a email forward of such account to a frequently read email account. Create / edit `AutoCron` jobs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Create a T3 page with a QFQ record (similar to the formeditor). Such page should be access restricted and is only needed to edit `AutoCron` jobs: .. code-block:: sql dbIndex={{indexQfq:Y}} form={{form:S}} { # Table header. sql = SELECT CONCAT('p:{{pageSlug:T}}?form=cron') AS _pagen, 'id', 'Next run','Frequency','Comment' , 'Last run','In progress', 'Status', 'Auto generated' head = tail =
rbeg = rend = fbeg = fend = { # All Cron Jobs sql = SELECT CONCAT('') , '', CONCAT('p:{{pageSlug:T}}?form=cron&r=', c.id) AS _pagee, '' , c.id, '' , IF( c.nextrun=0,"", DATE_FORMAT(c.nextrun, "%d.%m.%y %H:%i:%s")), '' , c.frequency, '' , c.comment, '' , IF(c.lastrun=0,"", DATE_FORMAT(c.lastrun,"%d.%m.%y %H:%i:%s")), '' , IF(c.inProgress=0,"", DATE_FORMAT(c.inProgress,"%d.%m.%y %H:%i:%s")), '' , LEFT(c.laststatus,40) AS '_+pre', '' , c.autoGenerated , CONCAT('U:form=cron&r=', c.id) AS _paged, '' FROM Cron AS c ORDER BY c.id } } Or you can use the following code in a separate QFQ record for the twig version of autoCron:: file=_autoCronTwig Usage ^^^^^ The system `cron` service will call the `QFQ AutoCron` every minute. `QFQ AutoCron` checks if there is a pending job, by looking for jobs with `nextRun` <= `NOW()`. All found jobs will be fired - depending on their type, such jobs will send mail(s) or open a `webpage`. A `webpage` will mostly be a local T3 page with at least one QFQ record on it. Such a QFQ record might do some manipulation on the database or any other task. A job with `nextRun`=0 or `inProgress`!=0 won't never be started. Due to checking `inProgress`, jobs will never run in parallel, even if a job needs more than 1 minute (interval system cron). Job: repeating """""""""""""" * frequency: '1 MINUTE', '2 DAY', '3 MONTH', .... After finishing a job, `nextRun` will be increased by `frequency`. If `nextRun` still points in the past, it will be increased by `frequency` again, until it points to the future. Job: asynchronous """"""""""""""""" * frequency: An 'AutoCron' job becomes 'asynchronous' if `frequency` is empty. Then, auto repeating is switched off. If `nextRun` is > 0 and in the past, the job will be fired. After the job has been done, `nextRun` will be set to 0. This is useful for jobs which have to be fired from time to time. To fire such an asynchronous job, just set `nextRun=NOW()` and wait for the next system cron run. If such a job is running and a new `nextRun=NOW()` is applied, the 'AutoCron' job will be fired again during the next system cron run. Type: Mail """""""""" Currently QFQ uses a special sendmail notation - this will change in the future. * `Mail`: .. code-block:: sql {{!SELECT 'john@doe.com' AS sendMailTo, 'Custom subject' AS sendMailSubject, 'jane@doe.com' AS sendMailFrom, 123 AS sendMailGrId, 456 AS sendMailXId}} AutoCron will send as many mails as records are selected by the SQL query in field `Mail`. Field `Mail body` provides the mail text. All columns of the SQL are available in STORE_PARENT. Type: Website """"""""""""" The page specified in `URL` will be opened. Optional the output of that page can be logged to a file (take care to have write permissions on that file). * `Log output to file`=`output.log` - creates a file in the Typo3 host directory. * `Log output to file`=`/var/log/output.log` - creates a file in `/var/log/` directory. Also `overwrite` or `append` can be selected for the output file. In case of `append` a file rotation should be setup on OS level. To check for a successful DB connection, it's a good practice to report a custom token on the T3 page / QFQ record like 'DB Connect: ok'. Such a string can be checked via `Pattern to look for on output=/DB Connect: ok/`. The pattern needs to be written in PHP PCRE syntax. For a simple search string, just surround them with '/'. If the pattern is found on the page, the job get's 'Ok' - else 'Error - ...'. Access restriction ;;;;;;;;;;;;;;;;;; To protect AutoCron pages not to be triggered accidental or by unprivileged access, access to those page tree might be limited to localhost. Some example Typoscript:: # Access allowed for any logged in user or via 'localhost' [usergroup = *] || [IP = 127.0.0.1] page.10 < styles.content.get [else] # Error Message page.10 = TEXT page.10.value =

Access denied

Please log in or access this page from an authorized host. Your current IP address:  page.20 = TEXT page.20.data = getenv : REMOTE_ADDR [global] AutoCron / website: HTTPS protocol ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; * For `https` the PHP extension `php_openssl` has to be installed. * All certificates are accepted, even self signed without a correct chain or hostnames, not listed in the certificate. This is useful if there is a general 'HTTP >> HTTPS' redirection configured and the website is accessed via `https://localhost/...` .. _`versionApi`: Version API ----------- The Version API provides system information about the QFQ installation, including TYPO3 version, installed extensions, configuration details, and backend/frontend user information. This endpoint is useful for monitoring and administration purposes. .. tip:: Endpoint: ``/typo3conf/ext/qfq/Classes/Api/version.php`` Setup ^^^^^ The API requires token-based authorization. Configure the API token in the QFQ extension settings: * In TYPO3 backend: **Admin Tools > Settings > Configure extensions > qfq > Security** * Set the ``qfqVersionApiToken`` to a secure random string. * There are no minimum requirements for the token, but it is recommended to use a 30-character alphanumeric value. Example: ``aB3xK9mP2qR7sT4uV6wY8zN1cD5eF0gH`` .. important:: The API will return an error if no token is configured on the server or if the provided token doesn't match. Error codes see below. Authorization ^^^^^^^^^^^^^ Every request must include the API token in the ``Authorization`` HTTP header. Example: :: Authorization: your-secret-token-here Response ^^^^^^^^ The API returns a JSON object containing: +----------------------------+---------------------------------------------------------------------+ | Field | Description | +============================+=====================================================================+ | url | Base URL of the TYPO3 installation | +----------------------------+---------------------------------------------------------------------+ | hostname | Server hostname | +----------------------------+---------------------------------------------------------------------+ | project | TYPO3 site name from ``LocalConfiguration.php`` (SYS/sitename) | +----------------------------+---------------------------------------------------------------------+ | typo3 | TYPO3 version | +----------------------------+---------------------------------------------------------------------+ | deb-packages | Debian package versions from whitelist [#debpkgs]_ | +----------------------------+---------------------------------------------------------------------+ | check-beUser | JSON encoded list of all TYPO3 backend users. Also writable, see | | | below. | +----------------------------+---------------------------------------------------------------------+ | check-feUser | JSON encoded frontend user activity information | +----------------------------+---------------------------------------------------------------------+ | check-extSetting | Extension settings (values from ext_conf_template.txt). Also | | | writable, see below. | +----------------------------+---------------------------------------------------------------------+ | check-extModule | Extension module data (e.g., OIDC providers). Also writable, see | | | below. | +----------------------------+---------------------------------------------------------------------+ | | Version for each enabled extension (e.g. ``qfq``, ``frontend``) | +----------------------------+---------------------------------------------------------------------+ | file-qfqJson | QFQ configuration file content (sensitive data masked) | +----------------------------+---------------------------------------------------------------------+ | file-LocalConfigurationPhp | TYPO3 LocalConfiguration content (sensitive data masked) | +----------------------------+---------------------------------------------------------------------+ .. note:: Sensitive data like passwords, encryption keys, and secrets are automatically masked in the response. A configuration key is considered sensitive if it matches (case-insensitive) one of the predefined blacklist keywords [#sensitive]_. Matching values are replaced with ``***`` in the output. .. rubric:: Footnotes .. [#sensitive] Blacklist keywords for sensitive data masking: ``DB_1_PASSWORD``, ``DB_2_PASSWORD``, ``DB_3_PASSWORD``, ``DB_4_PASSWORD``, ``LDAP_1_PASSWORD``, ``ENCRYPTION_KEY``, ``installToolPassword``, ``password``, ``secretKey``, ``forwardServiceSecret``, ``encryptionKey`` .. [#debpkgs] Whitelist for Debian packages (defined in ``Config::getServerInfo()``): ``apache2``, ``nginx``, ``mariadb-client``, ``mariadb-server``, ``wkhtmltopdf`` Example ^^^^^^^ Request: :: curl -X GET -H "Authorization: your-secret-token-here" \ "https://example.com/typo3conf/ext/qfq/Classes/Api/version.php" Response: :: { "url": "https://example.com/", "hostname": "webserver01", "project": "My TYPO3 Project", "typo3": "11.5.30", "deb-packages": { ... }, "check-beUser": "[{\"username\":\"admin\", ...}]", "check-feUser": "[{...}]", "qfq": "24.10.0", "fluid_styled_content": "11.5.30", ... } Backend User Management ^^^^^^^^^^^^^^^^^^^^^^^ The API can optionally manage backend users by sending a JSON body with a ``check-beUser`` object. Each key is a username, and the value defines the desired state. .. note:: If a user specified in the request does not exist in TYPO3, it will be created automatically. +-------------------+------------------------------------------------------------------------------+ | Property | Description | +===================+==============================================================================+ | isAdmin | User should have admin privileges | +-------------------+------------------------------------------------------------------------------+ | isMaintainer | User should be a system maintainer | +-------------------+------------------------------------------------------------------------------+ | isDisabled | User should be disabled | +-------------------+------------------------------------------------------------------------------+ | delete | User should be deleted | +-------------------+------------------------------------------------------------------------------+ Request: :: curl -X POST -H "Authorization: your-secret-token-here" \ -H "Content-Type: application/json" \ -d '{ "check-beUser": { "admino": { "isAdmin": true, "isMaintainer": true, "isDisabled": false, "delete": false }, "olduser": { "isAdmin": false, "isMaintainer": false, "isDisabled": false, "delete": true } } }' \ "https://example.com/typo3conf/ext/qfq/Classes/Api/version.php" HTML Status Responses ^^^^^^^^^^^^^^^^^^^^^ +--------+----------------------------------------------------------+ | Code | Description | +========+==========================================================+ | 200 | OK | +--------+----------------------------------------------------------+ | 401 | No API token provided or invalid token | +--------+----------------------------------------------------------+ | 500 | Server configuration error (API token not set on server) | +--------+----------------------------------------------------------+ Extension Settings Management ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The API can read and write TYPO3 extension settings by using the ``check-extSetting`` field. Extension settings are the values configured in **Admin Tools > Settings > Configure extensions**. Reading Extension Settings """""""""""""""""""""""""" A standard GET request returns the ``check-extSetting`` field containing the current configuration for all extensions that have an ``ext_conf_template.txt`` file. The response includes both the template structure and current values. Response structure: :: { "check-extSetting": { "qfq": { "basic": { "database": { "value": "1" }, "flagProduction": { "value": "yes" }, ... }, "security": { "qfqVersionApiToken": { "value": "***" }, ... } }, "oidc_client": { "general": { "defaultScopes": { "value": "openid email" }, ... } } } } Writing Extension Settings """""""""""""""""""""""""" To update extension settings, send a POST request with the ``check-extSetting`` object. Only the values you want to change need to be included. Request: :: curl -X POST -H "Authorization: your-secret-token-here" \ -H "Content-Type: application/json" \ -d '{ "check-extSetting": { "qfq": { "basic": { "flagProduction": { "value": "no" } } } } }' \ "https://example.com/typo3conf/ext/qfq/Classes/Api/version.php" .. note:: Changes to extension settings are written to ``LocalConfiguration.php`` and the TYPO3 cache is automatically cleared. OIDC Provider Management (check-extModule) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The API can manage OIDC client providers through the ``check-extModule`` field. This requires the ``oidc_client`` extension to be installed. .. important:: The ``check-extModule`` endpoint operates in two modes: * **Isolated mode**: When only ``check-extModule`` is in the request (no ``check-beUser``), only OIDC data is returned. * **Full mode**: When both ``check-extModule`` and ``check-beUser`` are in the request, all data including OIDC is returned. Reading OIDC Providers """""""""""""""""""""" Request: :: curl -X POST -H "Authorization: your-secret-token-here" \ -H "Content-Type: application/json" \ -d '{"check-extModule": {}}' \ "https://example.com/typo3conf/ext/qfq/Classes/Api/version.php" Response structure: :: { "check-extModule": { "oidc_client": { "installed": true, "tableExists": true, "providers": { "microsoft": { "general": { "uid": { "value": 1, "__label": "UID", "__type": "number", "__readonly": true }, "name": { "value": "microsoft", "__label": "Internal identifier", "__type": "text", "__readonly": true }, "title": { "value": "Microsoft SSO", "__label": "The name of the provider", "__type": "text" } }, "client": { "client_id": { "value": "abc123", "__label": "Client identifier", "__type": "text" }, "client_secret": { "value": "***", "__label": "Client secret", "__type": "text" }, ... }, "processors_ldap": { "enabled": { "value": 0, "__label": "Enable", "__type": "checkbox" }, "filter": { "value": "", "__label": "Query filter", "__type": "text" }, "server_1": { "value": 1, "__label": "LDAP Server 1", "__type": "checkbox", "__cat": "servers" }, ... }, ... } } } } } The response uses the **inputCollectionJson** format with metadata for each field: +---------------+------------------------------------------------------------------+ | Metadata Key | Description | +===============+==================================================================+ | value | Current field value | +---------------+------------------------------------------------------------------+ | __label | Human-readable field label | +---------------+------------------------------------------------------------------+ | __desc | Field description/help text | +---------------+------------------------------------------------------------------+ | __type | Field type: ``text``, ``textarea``, ``checkbox``, ``select``, | | | ``number`` | +---------------+------------------------------------------------------------------+ | __readonly | If true, field cannot be modified | +---------------+------------------------------------------------------------------+ | __cat | Sub-category for grouping (e.g., ``servers`` for LDAP servers) | +---------------+------------------------------------------------------------------+ | __options | Available options for select fields | +---------------+------------------------------------------------------------------+ Provider Categories """"""""""""""""""" Each provider is organized into categories: +---------------------+------------------------------------------------------------------+ | Category | Description | +=====================+==================================================================+ | general | Basic provider info (uid, name, title) | +---------------------+------------------------------------------------------------------+ | authentication | Authentication settings (MFA, backend/frontend enable flags) | +---------------------+------------------------------------------------------------------+ | client | OAuth client credentials (client_id, client_secret, scopes) | +---------------------+------------------------------------------------------------------+ | provider | Provider endpoints (auth, token, userinfo, logout URLs) | +---------------------+------------------------------------------------------------------+ | mapping | Claim mapping rules (id_claim, usernames, emails, mapping) | +---------------------+------------------------------------------------------------------+ | notes | Description field | +---------------------+------------------------------------------------------------------+ | processors_ldap | LDAP processor settings (enable, filter, mapping, servers) | +---------------------+------------------------------------------------------------------+ | processors_qfq | QFQ processor settings (enable, prefix, store, report) | +---------------------+------------------------------------------------------------------+ Writing OIDC Providers """""""""""""""""""""" To create or update OIDC providers, send a POST request with the provider data: Request: :: curl -X POST -H "Authorization: your-secret-token-here" \ -H "Content-Type: application/json" \ -d '{ "check-extModule": { "oidc_client": { "providers": { "google": { "general": { "title": { "value": "Google SSO" } }, "client": { "client_id": { "value": "google-client-id" }, "client_secret": { "value": "google-secret" } }, "provider": { "auth_endpoint": { "value": "https://accounts.google.com/o/oauth2/auth" }, "token_endpoint": { "value": "https://oauth2.googleapis.com/token" } } } } } } }' \ "https://example.com/typo3conf/ext/qfq/Classes/Api/version.php" Deleting OIDC Providers """"""""""""""""""""""" To soft-delete a provider, include the ``delete`` flag: Request: :: curl -X POST -H "Authorization: your-secret-token-here" \ -H "Content-Type: application/json" \ -d '{ "check-extModule": { "oidc_client": { "providers": { "old-provider": { "delete": true } } } } }' \ "https://example.com/typo3conf/ext/qfq/Classes/Api/version.php" .. note:: Deleted providers are soft-deleted (``deleted=1``) in the database. If you create a new provider with the same name as a previously deleted one, the deleted provider will be reactivated and updated with the new values.