Quantcast
Channel: Archives des PostgreSQL - dbi Blog
Viewing all 522 articles
Browse latest View live

PostgreSQL 14: LZ4 compression for TOAST

$
0
0

In PostgreSQL a row or tuple can not span multiple pages (a page is typically 8kB), but of course you can store larger rows and PostgreSQL brakes and compresses these rows into smaller chunks by using a technique called TOAST. Once your table contains a toast-able data type a so-called toast table is created automatically. Up to PostgreSQL 13 you had no choice how the data is compressed, but a recent commit brings the option to use LZ4 as compression method for TOAST. As you can read in the Wikipedia article, LZ4 is all about compression and decompression speed, so let’s have a look.

If you want to make use of this new feature, you need to compile PostgreSQL with support for it:

postgres@debian10pg:/home/postgres/postgresql/ [pgdev] ./configure --help | grep LZ4
  --with-lz4              build with LZ4 support
  LZ4_CFLAGS  C compiler flags for LZ4, overriding pkg-config
  LZ4_LIBS    linker flags for LZ4, overriding pkg-config

For that to work, the operating system needs to have the corresponding libraries installed. For Debian this is:

postgres@debian10pg:/home/postgres/postgresql/ [pgdev] apt search liblz4-dev
Sorting... Done
Full Text Search... Done
liblz4-dev/stable,now 1.8.3-1 amd64 [installed]
  Fast LZ compression algorithm library - development files

Having that ready and PostgreSQL compiled and installed, lets create two tables: One with the default compression and one with the new LZ4 compression:

postgres=# create table t1 ( a text );
CREATE TABLE
postgres=# create table t2 ( a text compression LZ4 );
CREATE TABLE
postgres=# \d+ t1
                                          Table "public.t1"
 Column | Type | Collation | Nullable | Default | Storage  | Compression | Stats target | Description 
--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
 a      | text |           |          |         | extended | pglz        |              | 
Access method: heap

postgres=# \d+ t2
                                          Table "public.t2"
 Column | Type | Collation | Nullable | Default | Storage  | Compression | Stats target | Description 
--------+------+-----------+----------+---------+----------+-------------+--------------+-------------
 a      | text |           |          |         | extended | lz4         |              | 
Access method: heap

Both tables automatically got a toast table attached:

postgres=# select reltoastrelid from pg_class where relname in ('t1','t2');
 reltoastrelid 
---------------
         16387
         16392
(2 rows)

postgres=# select oid,relname from pg_class where oid in (16387,16392 );
  oid  |    relname     
-------+----------------
 16387 | pg_toast_16384
 16392 | pg_toast_16389
(2 rows)

postgres=# \d+ pg_toast.pg_toast_16384
TOAST table "pg_toast.pg_toast_16384"
   Column   |  Type   | Storage 
------------+---------+---------
 chunk_id   | oid     | plain
 chunk_seq  | integer | plain
 chunk_data | bytea   | plain
Owning table: "public.t1"
Indexes:
    "pg_toast_16384_index" PRIMARY KEY, btree (chunk_id, chunk_seq)
Access method: heap

postgres=# \d+ pg_toast.pg_toast_16389
TOAST table "pg_toast.pg_toast_16389"
   Column   |  Type   | Storage 
------------+---------+---------
 chunk_id   | oid     | plain
 chunk_seq  | integer | plain
 chunk_data | bytea   | plain
Owning table: "public.t2"
Indexes:
    "pg_toast_16389_index" PRIMARY KEY, btree (chunk_id, chunk_seq)
Access method: heap

Let’s check if there is a difference if we populate those tables with some dummy data:

postgres=# \timing on
Timing is on.
postgres=# insert into t1(a) select lpad('a',1000000,'a') from generate_series(1,1000);
INSERT 0 1000
Time: 4643.583 ms (00:04.644)
postgres=# insert into t2(a) select lpad('a',1000000,'a') from generate_series(1,1000);
INSERT 0 1000
Time: 314.107 ms
postgres=# truncate table t1,t2;
TRUNCATE TABLE
Time: 4143.579 ms (00:04.144)
postgres=# insert into t1(a) select lpad('a',1000000,'a') from generate_series(1,1000);
INSERT 0 1000
Time: 4759.809 ms (00:04.760)
postgres=# insert into t2(a) select lpad('a',1000000,'a') from generate_series(1,1000);
INSERT 0 1000
Time: 1011.221 ms (00:01.011)
postgres=# truncate table t1,t2;
TRUNCATE TABLE
Time: 41.449 ms
postgres=# insert into t1(a) select lpad('a',1000000,'a') from generate_series(1,1000);
INSERT 0 1000
Time: 4507.416 ms (00:04.507)
postgres=# insert into t2(a) select lpad('a',1000000,'a') from generate_series(1,1000);
INSERT 0 1000
Time: 299.458 ms

This is a huge difference, and I’ve repeated that test several times and got almost the same numbers. That’s really a great improvement regarding speed. But what about the size on disk?

postgres=# select relname,reltoastrelid from pg_class where relname in ('t1','t2');
 relname | reltoastrelid 
---------+---------------
 t1      |         16387
 t2      |         16392
(2 rows)

Time: 3.091 ms
postgres=# select oid,relname from pg_class where oid in (16387,16392);
  oid  |    relname     
-------+----------------
 16387 | pg_toast_16384
 16392 | pg_toast_16389
(2 rows)

Time: 0.836 ms
postgres=# select pg_size_pretty(pg_relation_size('pg_toast.pg_toast_16384'));
 pg_size_pretty 
----------------
 12 MB
(1 row)

Time: 0.506 ms
postgres=# select pg_size_pretty(pg_relation_size('pg_toast.pg_toast_16389'));
 pg_size_pretty 
----------------
 4000 kB
(1 row)

Time: 0.743 ms

Quite impressive. In addition to the speed, we also get a great reduction on disk. Because compression is better with LZ4, we see fewer rows in the toast table for t2:

postgres=# select count(*) from pg_toast.pg_toast_16384;
 count 
-------
  6000
(1 row)

Time: 5.807 ms
postgres=# select count(*) from pg_toast.pg_toast_16389;
 count 
-------
  2000
(1 row)

Of course the string I used here is not very representative, but this new feature really looks promising. There is also a new parameter if you want to change to this behavior globally;

postgres=# show default_toast_compression ;
 default_toast_compression 
---------------------------
 pglz
(1 row)
postgres=# alter system set default_toast_compression = 'lz4';
ALTER SYSTEM
postgres=# select pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

Cet article PostgreSQL 14: LZ4 compression for TOAST est apparu en premier sur Blog dbi services.


temBoard on SLES15 – 1 – The WebUI

$
0
0

During our last SwissPUG online meeting there was a presentation about monitoring PostgreSQL with temBoard. A question that came up afterwards was: How to install that on SLES 15 as there are only packages for Debian and CentOS/RHEL. As temBoard is written in Python you can also install it using pip. That requires a bit more work compared to the package method, but it is perfectly fine. The only downside (but this is true for the packaged version as well) is, that you need Python 2.7 for the WebUI, and this is already end of life. So, lets give it a try on a fresh SLES 15 minimal installation.

As mentioned just before we’ll start with a minimal installation, as this is intended to be a step by step guide:

sles15webui:~ $ cat /etc/os-release 
NAME="SLES"
VERSION="15-SP2"
VERSION_ID="15.2"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP2"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp2"

The minimal installation of SLES 15 comes with Python 3.6.13 and this is not what we need:

sles15webui:~ $ python3 --version
Python 3.6.13

For getting Python 2.7 onto the system, the corresponding repository needs to be enabled:

sles15webui:~ $ SUSEConnect --product sle-module-python2/15.2/x86_64
Registering system to SUSE Customer Center

Updating system details on https://scc.suse.com ...

Activating sle-module-python2 15.2 x86_64 ...
-> Adding service to system ...
-> Installing release package ...

Successfully registered system

Having that ready, we can install the requirements for temBoard:

sles15webui:~ $ zypper in python2 python2-pip
Refreshing service 'Basesystem_Module_15_SP2_x86_64'.
Refreshing service 'Python_2_Module_15_SP2_x86_64'.
Refreshing service 'SUSE_Linux_Enterprise_Server_15_SP2_x86_64'.
Refreshing service 'Server_Applications_Module_15_SP2_x86_64'.
...
sles15webui:~ $ python2 --version
Python 2.7.18

temBoard requires a PostgreSQL instance up and running as a repository database, and SLES 15 comes with PostgreSQL 13 already packaged:

sles15webui:~ $ zypper search postgres | grep 13
  | postgresql13                 | Basic Clients and Utilities for PostgreSQL                              | srcpackage
  | postgresql13                 | Basic Clients and Utilities for PostgreSQL                              | package
  | postgresql13-contrib         | Contributed Extensions and Additions to PostgreSQL                      | package
  | postgresql13-devel           | PostgreSQL client development header files and libraries                | package
  | postgresql13-docs            | HTML Documentation for PostgreSQL                                       | package
  | postgresql13-plperl          | The PL/Tcl, PL/Perl, and  PL/Python procedural languages for PostgreSQL | package
  | postgresql13-plpython        | The PL/Python Procedural Languages for PostgreSQL                       | package
  | postgresql13-pltcl           | PL/Tcl Procedural Language for PostgreSQL                               | package
  | postgresql13-server          | The Programs Needed to Create and Run a PostgreSQL Server               | package
  | postgresql13-server-devel    | PostgreSQL server development header files and utilities                | package

Either use that to install PostgreSQL and create a cluster, or install PostgreSQL from source code. I’ll skip these steps here and my PostgreSQL instance is already running:

postgres@sles15webui:/home/postgres/ [tb] psql
psql (13.2)
Type "help" for help.

postgres=# select version();
                                      version                                       
------------------------------------------------------------------------------------
 PostgreSQL 13.2 on x86_64-pc-linux-gnu, compiled by gcc (SUSE Linux) 7.5.0, 64-bit
(1 row)

The installation of temBoard is quite simple, all you have to do is this:

postgres@sles15webui:/home/postgres/ [tb] sudo pip2 install temboard psycopg2-binary
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting temboard
...
    Running setup.py install for tornado ... done
  WARNING: The script mako-render is installed in '/usr/local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script alembic is installed in '/usr/local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts temboard and temboard-migratedb are installed in '/usr/local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed Mako MarkupSafe alembic backports-abc configparser contextlib2 futures importlib-metadata pathlib2 psycopg2-binary python-dateutil python-editor scandir singledispatch sqlalchemy temboard tornado zipp

As you can see from the last lines above, temBoard gets installed into “/usr/local/bin”:

postgres@sles15webui:/home/postgres/ [tb] ls /usr/local/bin/
alembic  mako-render  temboard  temboard-migratedb

The other important directory is “/usr/local/share/temboard/”, this one contains all the scripts:

postgres@sles15webui:/home/postgres/ [tb] ls /usr/local/share/temboard/
auto_configure.sh  create_repository.sh  purge.sh  quickstart  sql

The “auto_configure.sh” script is the one which does all the work. You can either go with the default or adjust the parameters as you like, e.g.:

postgres@sles15webui:/home/postgres/ [tb] sudo su -
sles15webui:~ $ export ETCDIR=/u01/app/postgres/local/dmk/etc
sles15webui:~ $ export VARDIR=/u01/app/postgres/local/dmk/bin
sles15webui:~ $ export LOGDIR=/u01/app/postgres/local/dmk/log
sles15webui:~ $ export LOGFILE=/u01/app/postgres/local/dmk/log/temboard-auto-configure.log

The auto configuration script needs to connect to your PostgreSQL instance, so make sure that this works before starting:

sles15webui:~ $ export PATH=/u01/app/postgres/product/13/db_2/bin:$PATH
sles15webui:~ $ export PGHOST=/tmp
sles15webui:~ $ sudo -Eu postgres psql
could not change directory to "/root": Permission denied
psql (13.2)
Type "help" for help.

If that works just execute the script:

sles15webui:~ $ /usr/local/share/temboard/auto_configure.sh
Creating system user temBoard.
Configuring temboard in /u01/app/postgres/local/dmk/etc.
Generating self-signed certificate.
Creating Postgres user, database and schema.
Failure. See /u01/app/postgres/local/dmk/log/temboard-auto-configure.log for details.

… and that fails. Checking the log file, the reason is that “temboard-migratedb” is not found:

CREATE DATABASE
./create_repository.sh: line 17: temboard-migratedb: command not found
./create_repository.sh: line 18: temboard-migratedb: command not found

The easy fix is to update the create_repository.sh script:

if ! /usr/local/bin/temboard-migratedb check ; then
    /usr/local/bin/temboard-migratedb upgrade
    psql="psql -aw --set ON_ERROR_STOP=on --pset pager=off"
    if [ -n "${DEV-}" ] ; then
        $psql -f $SQLDIR/dev-fixture.sql
    fi
fi

Running the auto configuration once more, and it succeeds:

sles15webui:~ $ /usr/local/share/temboard/auto_configure.sh
Configuring temboard in /u01/app/postgres/local/dmk/etc.
Creating Postgres user, database and schema.

Success. You can now start temboard using:

    systemctl start temboard

Remember to replace default admin user!!!

A systemd service was created automatically:

sles15webui:~ $ systemctl status temboard
● temboard.service - temBoard Web UI
   Loaded: loaded (/usr/local/lib/systemd/system/temboard.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
sles15webui:~ $ cat /usr/local/lib/systemd/system/temboard.service
[Unit]
Description=temBoard Web UI
After=network.target

[Service]
Type=simple
User=temboard
Group=temboard
ExecStart=/usr/bin/env SYSTEMD=1 temboard -c /etc/temboard/temboard.conf

[Install]
WantedBy=multi-user.target

The issue with that service is, that the configuration file does not exist:

sles15webui:~ $ cat /etc/temboard/temboard.conf
cat: /etc/temboard/temboard.conf: No such file or directory

Because we’ve set the environment variables before doing the temBoard installation, our configuration file is here:

sles15webui:~ $ cat /u01/app/postgres/local/dmk/etc/temboard.conf 
# Configuration initiated by /usr/local/share/temboard/auto_configure.sh on Thu Apr  1 09:17:58 CEST 2021
#
# See https://temboard.rtfd.io/ for details on configuration
# possibilities.

[temboard]
ssl_cert_file = /etc/ssl/certs/temboard.pem
ssl_key_file = /etc/ssl/private/temboard.key
cookie_secret = d11bcef48a3f6bc6de09cb3e39b88b99b2fae98b4d3dbfe501019efdcac681044be29eed3907b0cbe12dc64d49923f50f262b33fe1d154cf9b8609f2770edaef
home = /u01/app/postgres/local/dmk/bin

[repository]
host = /tmp
port = 5432
user = temboard
password = 4ebe5cd9715718780702420436cbd268
dbname = temboard

[logging]
method = stderr
level = INFO

[monitoring]
# purge_after = 365

[statements]
# purge_after = 7

Let’s try to start temBoard using this configuration manually as root:

sles15webui:~ $ /usr/bin/env SYSTEMD=1 temboard -c /u01/app/postgres/local/dmk/etc/temboard.conf
 INFO: Starting temboard 7.6.
 INFO: Running on SLES 15-SP2.
 INFO: Using Python 2.7.18 (/usr/bin/python2).
 INFO: Using Psycopg2 2.8.6 (dt dec pq3 ext lo64), Tornado 5.1.1 and SQLAlchemy 1.4.4
Loaded plugin 'activity'.
Loaded plugin 'dashboard'.
Loaded plugin 'monitoring'.
Loaded plugin 'pgconf'.
...

This works and accessing the UI over https://[IP]:8888 works as well:

So, we need to fix the systemd unit file to use the correct temBoard configuration file like this:

sles15webui:~ $ cat /usr/local/lib/systemd/system/temboard.service
[Unit]
Description=temBoard Web UI
After=network.target

[Service]
Type=simple
User=temboard
Group=temboard
ExecStart=/usr/bin/env SYSTEMD=1 temboard -c /u01/app/postgres/local/dmk/etc/temboard.conf

[Install]
WantedBy=multi-user.target
sles15webui:~ $ systemctl daemon-reload

Lets try to start with systemd:

sles15webui:~ $ systemctl start temboard
sles15webui:~ $ systemctl status temboard
● temboard.service - temBoard Web UI
   Loaded: loaded (/usr/local/lib/systemd/system/temboard.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2021-04-01 09:49:17 CEST; 3s ago
  Process: 28629 ExecStart=/usr/bin/env SYSTEMD=1 temboard -c /u01/app/postgres/local/dmk/etc/temboard.conf (code=exited, status=1/FAILURE)
 Main PID: 28629 (code=exited, status=1/FAILURE)

Apr 01 09:49:16 sles15webui systemd[1]: Started temBoard Web UI.
Apr 01 09:49:17 sles15webui env[28629]:  INFO: Starting temboard 7.6.
Apr 01 09:49:17 sles15webui env[28629]: ERROR: Invalid temboard_ssl_key_file from config: /etc/ssl/private/temboard.key: File not found...
Apr 01 09:49:17 sles15webui env[28629]: CRITI: Failed to load configuration.
Apr 01 09:49:17 sles15webui systemd[1]: temboard.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:49:17 sles15webui systemd[1]: temboard.service: Unit entered failed state.
Apr 01 09:49:17 sles15webui systemd[1]: temboard.service: Failed with result 'exit-code'.

Next issue, the key file can not be found (the service uses the tembaord user to start tembaord). The file actually is there, but the temboard user does not have access to it:

sles15webui:~ $ ls -la /etc/ssl/private/temboard.key
-rw------- 1 root root 1704 Apr  1 08:29 /etc/ssl/private/temboard.key

As I did not want to open permissions to the SSL configuration in /etc, I’ve copied that over:

sles15webui:~ $ mv /etc/ssl/private/temboard.key /u01/app/postgres/local/dmk/etc/
sles15webui:~ $ chown temboard:temboard /u01/app/postgres/local/dmk/etc/temboard.key 

Adjusted the temboard configuration with the new location of the key file:

sles15webui:~ $ cat /u01/app/postgres/local/dmk/etc/temboard.conf | grep key
ssl_key_file = /u01/app/postgres/local/dmk/etc/temboard.key

Started again:

sles15webui:~ $ systemctl start temboard
sles15webui:~ $ systemctl status temboard
● temboard.service - temBoard Web UI
   Loaded: loaded (/usr/local/lib/systemd/system/temboard.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2021-04-01 09:54:50 CEST; 2s ago
  Process: 28792 ExecStart=/usr/bin/env SYSTEMD=1 temboard -c /u01/app/postgres/local/dmk/etc/temboard.conf (code=exited, status=1/FAILURE)
 Main PID: 28792 (code=exited, status=1/FAILURE)

Apr 01 09:54:50 sles15webui env[28792]:     now=datetime.utcnow()
Apr 01 09:54:50 sles15webui env[28792]:   File "/usr/local/lib/python2.7/site-packages/temboardui/toolkit/tasklist/sqlite3_engine.py", line 253, in recover
Apr 01 09:54:50 sles15webui env[28792]:     raise StorageEngineError("Could not recover tasks.")
Apr 01 09:54:50 sles15webui env[28792]: StorageEngineError: Could not recover tasks.
Apr 01 09:54:50 sles15webui env[28792]: temboardui version is 7.6.
Apr 01 09:54:50 sles15webui env[28792]: This is a bug!
Apr 01 09:54:50 sles15webui env[28792]: Please report traceback to https://github.com/dalibo/temboard/issues! Thanks!
Apr 01 09:54:50 sles15webui systemd[1]: temboard.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:54:50 sles15webui systemd[1]: temboard.service: Unit entered failed state.
Apr 01 09:54:50 sles15webui systemd[1]: temboard.service: Failed with result 'exit-code'.

… and it again fails. Lets try to start manually with the temBoard user:

sles15webui:~ $ su - temboard
This account is currently not available.
sles15webui:~ $ usermod -s /bin/bash temboard
sles15webui:~ $ su - temboard
attempt to write a readonly database
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/temboardui/toolkit/tasklist/sqlite3_engine.py", line 244, in recover
    (st_aborted, datetime_to_epoch(now), st_doing)
OperationalError: attempt to write a readonly database
Unhandled error:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/temboardui/toolkit/app.py", line 298, in entrypoint
    retcode = self.main(argv, environ)
  File "/usr/local/lib/python2.7/site-packages/temboardui/__main__.py", line 395, in main
    self.scheduler.apply_config()
  File "/usr/local/lib/python2.7/site-packages/temboardui/__main__.py", line 143, in apply_config
    super(SchedulerService, self).apply_config()
  File "/usr/local/lib/python2.7/site-packages/temboardui/toolkit/taskmanager.py", line 724, in apply_config
    self.scheduler.setup_task_list()
  File "/usr/local/lib/python2.7/site-packages/temboardui/toolkit/taskmanager.py", line 482, in setup_task_list
    self.task_list.recover()
  File "/usr/local/lib/python2.7/site-packages/temboardui/toolkit/taskmanager.py", line 196, in recover
    now=datetime.utcnow()
  File "/usr/local/lib/python2.7/site-packages/temboardui/toolkit/tasklist/sqlite3_engine.py", line 253, in recover
    raise StorageEngineError("Could not recover tasks.")
StorageEngineError: Could not recover tasks.
temboardui version is 7.6.
This is a bug!
Please report traceback to https://github.com/dalibo/temboard/issues! Thanks!

The issue is this: “attempt to write a readonly database”. As there is no information where that database (seems to be SQLite) is located, lets use strace to spot the location:

temboard@sles15webui:~> strace -f /usr/bin/env SYSTEMD=1 temboard -c /u01/app/postgres/local/dmk/etc/temboard.conf 

Digging through the strace output, the database is this one: /u01/app/postgres/local/dmk/bin/server_tasks.db. Because we started as root before, this file has the woring permissions (this is not the fault of temBoard):

sles15webui:~$  ls -la /u01/app/postgres/local/dmk/bin/server_tasks.db
-rw-r--r-- 1 root root 12288 Apr  1 09:45 /u01/app/postgres/local/dmk/bin/server_tasks.db

Lets fix this, and try again:

sles15webui:~ $ chown temboard:temboard /u01/app/postgres/local/dmk/bin/server_tasks.db
sles15webui:~ $ systemctl status temboard
● temboard.service - temBoard Web UI
   Loaded: loaded (/usr/local/lib/systemd/system/temboard.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-04-01 10:15:22 CEST; 3s ago
 Main PID: 1064 (temboard)
    Tasks: 5
   CGroup: /system.slice/temboard.service
           ├─1064 temboard: web
           ├─1071 temboard: worker pool
           └─1072 temboard: scheduler

Apr 01 10:15:22 sles15webui env[1064]: Loaded plugin 'pgconf'.
Apr 01 10:15:22 sles15webui env[1064]: Loaded plugin 'maintenance'.
Apr 01 10:15:22 sles15webui env[1064]: Loaded plugin 'statements'.
Apr 01 10:15:22 sles15webui env[1064]: temBoard database is up-to-date.
Apr 01 10:15:22 sles15webui env[1064]: Starting web.
Apr 01 10:15:22 sles15webui env[1064]: Serving temboardui on https://0.0.0.0:8888
Apr 01 10:15:22 sles15webui env[1064]: Starting scheduler.
Apr 01 10:15:22 sles15webui env[1064]: Starting worker pool.
Apr 01 10:15:23 sles15webui env[1064]: Starting collector scheduler worker.
Apr 01 10:15:23 sles15webui env[1064]: End of collector scheduler worker.

Now we are fine, temBoard is started with the correct user with systemd, login (admin/admin) works as well:

This was kind of a brute force method for getting it installed, but if you plan that well, it should not be an issue if you follow these steps:

  • Prepare your own self signed certificate and make sure the temBoard user can access it
  • Prepare your own systemd configuration file
  • Prepare your own temBoard configuration file

In the next post we’ll install the temBoard agent on another SLES 15 machine, and actually want to monitor a PostgreSQL instance.

Cet article temBoard on SLES15 – 1 – The WebUI est apparu en premier sur Blog dbi services.

temBoard on SLES15 – 2 – The agent

$
0
0

In the last post we’ve installed the temBoard repository and the UI. As this alone is not so much fun, we’ll be installing the temBoard agent on another SLES 15 machine in this post. We’ll end up with one PostgreSQL instance registered in the temBoard repository and this instance is constantly monitored. With the repository and the UI we had the requirement to use Python 2.7, but this is not anymore required for the temBoard agent. Lets go.

What I’ve already prepared, is the PostgreSQL instance we want the agent to monitor:

postgres@localhost:/u02/pgdata/13/PG1/ [PG1] psql -l
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   
-----------+----------+----------+-------------+-------------+-----------------------
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
(3 rows)

As pip3 is already available we can go straight ahead and install the temBoard agent:

ostgres@localhost:/u02/pgdata/13/PG1/ [PG1] sudo sudo pip3 install temboard-agent
Collecting temboard-agent
  Downloading https://files.pythonhosted.org/packages/58/6e/3d5c772587e48b17cc6fa5a1569daab0d879845828c792f16ecec49564ae/temboard_agent-7.6-py2.py3-none-any.whl (115kB)
    100% |████████████████████████████████| 122kB 1.5MB/s 
pygobject 3.34.0 requires pycairo>=1.11.1, which is not installed.
Installing collected packages: temboard-agent
Successfully installed temboard-agent-7.6
You are using pip version 10.0.1, however version 21.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

First issue, but easy to fix:

postgres@localhost:/u02/pgdata/13/PG1/ [PG1] sudo zypper in cairo-devel
postgres@localhost:/u02/pgdata/13/PG1/ [PG1] sudo pip3 install pycairo
Collecting pycairo
  Using cached pycairo-1.20.0.tar.gz (344 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: pycairo
  Building wheel for pycairo (PEP 517) ... done
  Created wheel for pycairo: filename=pycairo-1.20.0-cp36-cp36m-linux_x86_64.whl size=246164 sha256=e25a94a3a0801e25db8eee9b1080d793b7a6dd313b12b011e56888509e411362
  Stored in directory: /root/.cache/pip/wheels/09/91/ec/f970f7c0cd3fba778c7150409181074fdfc2af42148486561d
Successfully built pycairo
Installing collected packages: pycairo
Successfully installed pycairo-1.20.0

Once again:

postgres@localhost:/u02/pgdata/13/PG1/ [PG1] sudo sudo pip3 install temboard-agent
Requirement already satisfied: temboard-agent in /usr/lib/python3.6/site-packages (7.6)
postgres@localhost:/u02/pgdata/13/PG1/ [PG1] which temboard-agent
/usr/bin/temboard-agent
postgres@localhost:/u02/pgdata/13/PG1/ [PG1] temboard-agent --version
Traceback (most recent call last):
  File "/usr/bin/temboard-agent", line 7, in 
    from temboardagent.scripts.agent import main
  File "/usr/lib/python3.6/site-packages/temboardagent/scripts/agent.py", line 12, in 
    from ..cli import Application
  File "/usr/lib/python3.6/site-packages/temboardagent/cli.py", line 5, in 
    from .postgres import Postgres
  File "/usr/lib/python3.6/site-packages/temboardagent/postgres.py", line 8, in 
    import psycopg2.extensions
ModuleNotFoundError: No module named 'psycopg2'

For connecting to PostgreSQL psycopg2 must be there as well:

postgres@localhost:/u02/pgdata/13/PG1/ [PG1] sudo pip3 install psycopg2-binary
Collecting psycopg2-binary
  Downloading psycopg2_binary-2.8.6-cp36-cp36m-manylinux1_x86_64.whl (3.0 MB)
     |████████████████████████████████| 3.0 MB 1.3 MB/s 
Installing collected packages: psycopg2-binary
Successfully installed psycopg2-binary-2.8.6
postgres@localhost:/u02/pgdata/13/PG1/ [PG1] temboard-agent --version
 INFO: Starting temboard-agent 7.6.
7.6

The agent is ready, lets proceed with the configuration. As with the UI in the last post, there is a configuration script:

postgres@localhost:/u02/pgdata/13/PG1/ [PG1] ls -l /usr/share/temboard-agent/
total 24
-rwxr-xr-x 1 root root 7318 Apr  1 11:32 auto_configure.sh
-rwxr-xr-x 1 root root  682 Apr  1 11:32 purge.sh
drwxr-xr-x 1 root root  276 Apr  1 11:32 quickstart
-rwxr-xr-x 1 root root  674 Apr  1 11:32 restart-all.sh
-rw-r--r-- 1 root root 2843 Apr  1 11:32 temboard-agent.conf
-rw-r--r-- 1 root root  176 Apr  1 11:32 temboard-agent.logrotate

Again, make sure that psql can be executed and all the environment variable match your environment:

postgres@localhost:/u02/pgdata/13/PG1/ [PG1] sudo su -
localhost:~ $ export ETCDIR=/u01/app/postgres/local/dmk/etc/
localhost:~ $ export VARDIR=/u01/app/postgres/local/dmk/bin
localhost:~ $ export LOGDIR=/u01/app/postgres/local/dmk/log
localhost:~ $ export SYSUSER=postgres
localhost:~ $ export PGHOST=/tmp
localhost:~ $ export PATH=/u01/app/postgres/product/13/db_2/bin:$PATH
localhost:~ $ sudo -Eu postgres psql 
could not change directory to "/root": Permission denied
psql (13.2)
Type "help" for help.

postgres=# 

Once you’ve that ready run the configuration:

localhost:~ $ /usr/share/temboard-agent/auto_configure.sh
FQDN is not properly configured. Set agent hostname with TEMBOARD_HOSTNAME env var..
Failure. See /var/log/temboard-agent-auto-configure.log for details.

Ok, this error message is clear:

localhost:~ $ export TEMBOARD_HOSTNAME=sles15tbagent.it.dbi-services.com
localhost:~ $ /usr/share/temboard-agent/auto_configure.sh
Using hostname sles15tbagent.it.dbi-services.com.
Configuring for PostgreSQL user postgres.
Configuring for cluster on port 5432.
Configuring for cluster at /u02/pgdata/13/PG1.
Configuring temboard-agent in /u01/app/postgres/local/dmk/etc//13/pg5432/temboard-agent.conf .
Saving auto-configuration in /u01/app/postgres/local/dmk/etc//13/pg5432/temboard-agent.conf.d/auto.conf
Configuring temboard-agent to run on port 2345.
Enabling systemd unit temboard-agent@13-pg5432.service.

Success. You can now start temboard-agent using:

    systemctl start temboard-agent@13-pg5432.service

For registration, use secret key deb91ab1322007536e2e8284b26bdb5c .
See documentation for detailed instructions.

As we had troubles with the systemd unit file in the last post, lets have a look at it:

localhost:~ $ systemctl status temboard-agent@13-pg5432.service
● temboard-agent@13-pg5432.service - PostgreSQL Remote Control Agent 13/pg5432
   Loaded: loaded (/usr/lib/systemd/system/temboard-agent@.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
localhost:~ $ cat /usr/lib/systemd/system/temboard-agent@.service
[Unit]
Description=PostgreSQL Remote Control Agent %I
After=network.target postgresql@%i.service
AssertPathExists=/etc/temboard-agent/%I/temboard-agent.conf

[Service]
Type=simple
User=postgres
Group=postgres
ExecStart=/usr/bin/env SYSTEMD=1 temboard-agent -c /etc/temboard-agent/%I/temboard-agent.conf

[Install]
WantedBy=multi-user.target

This again points to the wrong configuration file, ours is here:

localhost:~ $ cat /u01/app/postgres/local/dmk/etc/13/pg5432/temboard-agent.conf
#
#   T E M B O A R D   A G E N T   C O N F I G U R A T I O N
#
# This file contains all possible settings of temBoard agent, with defaults for
# quickstart. temBoard agent overrides values of this file with .conf files
# found in directory named after this filename, with .d suffix. e.g.
# temboard-agent.conf.d/auto.conf is likely to override some values of this
# file.
#

[temboard]
# Bind port.
port = 2345
# Bind address.
address = 192.168.22.211
# User & password file.
users = users
# Plugins in use.
plugins = ["monitoring", "dashboard", "pgconf", "administration", "activity", "maintenance", "statements"]
# SSL: certificat file path (.pem)
ssl_cert_file = /u01/app/postgres/local/dmk/etc/13/pg5432/server.pem
# SSL: private key file path (.key)
ssl_key_file = /u01/app/postgres/local/dmk/etc/13/pg5432/server.key
# Hostname must be an unique and valid FQDN : e.g. db1.mydomain.foo
# If you leave this empty, then the system wide hostname will be used
# Note : `localhost` is not a correct value
hostname = sles15tbagent
# Key
key = SECRETKEYTOBECHANGED

[postgresql]
# Unix socket path.
host = /tmp
# PG port number.
port = 5432
# Super-user name.
user = postgres
# Super-user password.
# password =
# Default database.
dbname = postgres
# Instance name.
instance = main

[logging]
# Available methods for logging: stderr, syslog or file
method = stderr
# Syslog facility.
# facility = local0
# Log destination, should be /dev/log for syslog on Linux system.
# When using file logging method, this is referencing the log file path.
# destination = /var/log/temboard-agent/temboard-agent.log
# Log level, can be: DEBUG, INFO, WARNING, ERROR or CRITICAL.
level = INFO

### PLUGINS ###
[dashboard]
# Dashboard plugin part
# Interval, in second, between each run of the process collecting
# data used to render the dashboard. Default: 2
# scheduler_interval = 2
# Number of record to keep. Default: 150
# history_length = 150

[monitoring]
# Monitoring plugin part.
# DB name list (comma separator) to supervise. * for all.
dbnames = *
# List of probes to run, comma separator, * for all.
# Available probes: bgwriter,blocks,btree_bloat,cpu,db_size,filesystems_size,heap_bloat,loadavg,locks,memory,process,replication_connection,replication_lag,sessions,tblspc_size,temp_files_size_delta,wal_files,xacts
# when monitoring a standby server, 'wal_file' probe shouldn't be loaded.
probes = *
# Interval, in second, between each run of the process executing
# the probes. Default: 60
# scheduler_interval = 60

[administration]
# External command used for start/stop PostgreSQL.
# This commands actually works on Debian jessie.
pg_ctl = '/usr/lib/postgresql/9.4/bin/pg_ctl %s -D /var/lib/postgresql/9.4'

[statements]
# Statements plugin part.
# DB name hosting pg_stat_statements view (the one where the extension has
# been created with "CREATE EXTENSION")
dbname = postgres

Once more, adjust the the systemd service file (I am creating a static one here):

localhost:~ $ cat /usr/lib/systemd/system/temboard-agent@.service
[Unit]
Description=PostgreSQL Remote Control Agent %I
After=network.target postgresql@%i.service
AssertPathExists=/u01/app/postgres/local/dmk/etc/13/pg5432/temboard-agent.conf

[Service]
Type=simple
User=postgres
Group=postgres
ExecStart=/usr/bin/env SYSTEMD=1 temboard-agent -c /u01/app/postgres/local/dmk/etc/13/pg5432/temboard-agent.conf

[Install]
WantedBy=multi-user.target
localhost:~ $ mv /usr/lib/systemd/system/temboard-agent@.service /usr/lib/systemd/system/temboard-agent.service
localhost:~ $ systemctl daemon-reload
localhost:~ $ systemctl status temboard-agent.service
● temboard-agent.service - PostgreSQL Remote Control Agent 
   Loaded: loaded (/usr/lib/systemd/system/temboard-agent.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

Before starting with systemd, lets try to start with the postgres user. Before we do that we need to create a user which will be used for authentication later on:

localhost:~ $ su - postgres
postgres@localhost:/home/postgres/ [pg132] temboard-agent-adduser -c /u01/app/postgres/local/dmk/etc/13/pg5432/temboard-agent.conf
 INFO: Starting temboard-agent-adduser 7.6.
ERROR: Invalid temboard_ssl_key_file from config: /etc/ssl/private/temboard-agent-13-pg5432.key: File not found...
CRITI: Failed to load configuration.
postgres@localhost:/home/postgres/ [pg132] sudo ls -l /etc/ssl/private/temboard-agent-13-pg5432.key
-rw------- 1 root root 1704 Apr  1 11:46 /etc/ssl/private/temboard-agent-13-pg5432.key

Again the issue with the key file, so:

postgres@localhost:/home/postgres/ [pg132] sudo mv /etc/ssl/private/temboard-agent-13-pg5432.key /u01/app/postgres/local/dmk/etc/
postgres@localhost:/home/postgres/ [pg132] sudo chown postgres:postgres /u01/app/postgres/local/dmk/etc/temboard-agent-13-pg5432.key
postgres@localhost:/home/postgres/ [pg132] cat /u01/app/postgres/local/dmk/etc/13/pg5432/temboard-agent.conf | grep key
# SSL: private key file path (.key)
ssl_key_file = /u01/app/postgres/local/dmk/etc/temboard-agent-13-pg5432.key
key = SECRETKEYTOBECHANGED

Once more:

postgres@localhost:/home/postgres/ [pg132] temboard-agent-adduser -c /u01/app/postgres/local/dmk/etc/13/pg5432/temboard-agent.conf
 INFO: Starting temboard-agent-adduser 7.6.
ERROR: Invalid temboard_ssl_key_file from config: /etc/ssl/private/temboard-agent-13-pg5432.key: File not found...
CRITI: Failed to load configuration.

Seems the location of the key file is somehow hard coded, so workaround this like this:

postgres@localhost:/home/postgres/ [pg132] sudo ln -s /u01/app/postgres/local/dmk/etc/13/pg5432/temboard-agent.key /etc/ssl/private/temboard-agent-13-pg5432.key
postgres@localhost:/home/postgres/ [pg132] sudo chmod 755 /etc/ssl/private
postgres@localhost:/home/postgres/ [pg132] ls /etc/ssl/private/temboard-agent-13-pg5432.key
/etc/ssl/private/temboard-agent-13-pg5432.key
postgres@localhost:/home/postgres/ [pg132] temboard-agent-adduser -c /u01/app/postgres/local/dmk/etc/13/pg5432/temboard-agent.conf
 INFO: Starting temboard-agent-adduser 7.6.
Username: admin
Password: 
Retype password: 
Done.

The user/password combination will be written to the users file:

postgres@localhost:/home/postgres/ [pg132] cat /u01/app/postgres/local/dmk/etc/13/pg5432/users 
admin:xiet7KLumux50Q2dmV/FBue2zqtexElavkVAKnEYN/rw2YLIn51TQsXb3u8FPm4wiuHUTtEjjvBIrtBPAgRMsA==

That should be all and we should be able to start the agent:

postgres@localhost:/home/postgres/ [pg132] temboard-agent --config /u01/app/postgres/local/dmk/etc/13/pg5432/temboard-agent.conf 
 INFO: Starting temboard-agent 7.6.
 INFO: Found plugin monitoring = temboardagent.plugins.monitoring:MonitoringPlugin.
 INFO: Found plugin dashboard = temboardagent.plugins.dashboard:DashboardPlugin.
 INFO: Found plugin pgconf = temboardagent.plugins.pgconf:PgConfPlugin.
...
 INFO: Starting .
 INFO: Starting .
 INFO: Starting .

Looks fine, lets try with systemd:

sles15tbagent:/home/postgres $ systemctl start temboard-agent.service
sles15tbagent:/home/postgres $ systemctl status temboard-agent.service
● temboard-agent.service - PostgreSQL Remote Control Agent 
   Loaded: loaded (/usr/lib/systemd/system/temboard-agent.service; disabled; vendor preset: disabled)
   Active: active (running) since Sat 2021-04-03 14:42:21 CEST; 4s ago
 Main PID: 9299 (temboard-agent)
    Tasks: 5
   CGroup: /system.slice/temboard-agent.service
           ├─9299 temboard-agent: 13/pg5432: main process
           ├─9301 temboard-agent: 13/pg5432: worker pool
           └─9302 temboard-agent: 13/pg5432: scheduler

sles15tbagent:/home/postgres $ ps -ef | grep temboard
postgres  9299     1  0 14:42 ?        00:00:00 temboard-agent: 13/pg5432: main process
postgres  9301  9299  0 14:42 ?        00:00:00 temboard-agent: 13/pg5432: worker pool
postgres  9302  9299  0 14:42 ?        00:00:00 temboard-agent: 13/pg5432: scheduler
root      9804  9249  0 14:44 pts/1    00:00:00 grep --color=auto temboard

The final step is to register the agent to the WebUI:

postgres@sles15tbagent:/home/postgres/ [PG1] temboard-agent-register -c /u01/app/postgres/local/dmk/etc/13/pg5432/temboard-agent.conf --host $(hostname --fqdn) --port 2345 --groups default https://192.168.22.210:8888
 INFO: Starting temboard-agent-register 7.6.
 INFO: Getting system & PostgreSQL informations from the agent (https://sles15tbagent:2345/discover) ...
Login at https://192.168.22.210:8888 ...
 Username: admin
 Password: 
Registering instance/agent to https://192.168.22.210:8888 ...
Done.

Heading over to the WebUI confirms, that the agent is registered and the PostgreSQL instance is monitored:

So, even if there are no pre-build packages for your distribution, it should be quite easy to get temBoard up and running.

Cet article temBoard on SLES15 – 2 – The agent est apparu en premier sur Blog dbi services.

Pass a variable to a trigger in PostgreSQL

$
0
0

By Franck Pachot

.
With Oracle there are many ways to set a state in the session: context variables, set with DBMS_SESSION.SET_CONTEXT and retrieved with SYS_CONTEXT, or package global variables, or global or private temporary tables with ON COMMIT PRESERVE ROWS,…

How would you do it in PostgreSQL? I’m taking an example from a Yugabyte slack question, as Yugabyte SQL layer is the PostgreSQL one.


CREATE TABLE employees (
  employee_no integer PRIMARY KEY,
  name text,
  department text
);

INSERT INTO employees (employee_no, name, department) VALUES 
(1221, 'John Smith', 'Marketing'),
(1222, 'Bette Davis', 'Sales'),
(1223, 'Lucille Ball', 'Operations'),
(1224, 'John Zimmerman', 'Sales');

CREATE TABLE mgr_table (
  mgr_id integer references employees,
  employee_no integer references employees,
  primary key(mgr_id, employee_no)
);

insert into mgr_table values(1223,1222);

We have an employee/manager table example:


CREATE TABLE employee_dept_changes (
  employee_no integer NOT NULL references employees,
  name text,
  department text,
  changed_on TIMESTAMP(6) NOT NULL,
  changed_by integer
);

This table is there to log the changes when an employee is transferred to another manager. You can see that we log the values, but also the context about who is transferring because this operation is allowed only by a manager.

Ideally, all database operations are encapsulated in a microservice. The application calls a stored procedure that does all the logic: update the manager and log the change. And this procedure has all context. However, the idea here is to do this logging through a trigger. The trigger has access to the table values, but how can we pass the context of who is doing this change? Without context variables, or package variables, this requires something else.

If you look at the stackoverflow for this question, you will see Frits Hoogland and Bryn Llewellyn answers with workaround and recommendations. I’m just adding here a possibility (with the same recommendation: don’t have procedures calling SQL calling triggers, but all logic encapsulated in procedural code calling SQL).


ALTER TABLE employees ADD COLUMN
  "_trigger_state_mgr_id" integer --> this is added to pass the state variable

I’m adding a column to my table. This will not take any storage but will be used only to pass some values to the trigger


CREATE OR REPLACE FUNCTION record_dept_changes()
RETURNS TRIGGER AS
$$
BEGIN
 IF NEW.department  OLD.department
  THEN INSERT INTO employee_dept_changes(employee_no, name, department, changed_on, changed_by)
   VALUES(OLD.employee_no, OLD.name, OLD.department, now(), NEW."_trigger_state_mgr_id"  --> passing the state to the added column
   );
 END IF;
 NEW."_trigger_state_mgr_id" := null; --> setting null not to persist anything (or maybe a real "last_update_by" column makes sense?)
 RETURN NEW;
END;
$$
LANGUAGE 'plpgsql';

The trigger logs the change into the “changes” table, reading this added column value. But I set this value to null once used because I don’t want to waste any storage (disk and memory) for it.


CREATE TRIGGER dept_changes
BEFORE UPDATE ON employees
FOR EACH ROW
EXECUTE PROCEDURE record_dept_changes();

In PostgreSQL the trigger calls the procedure which has access to the NEW. valies


CREATE OR REPLACE PROCEDURE transfer_employee(integer, integer, text)
LANGUAGE plpgsql
AS $$
BEGIN
  -- IF employee reports to mgr, allow mgr to transfer person
  IF EXISTS (SELECT employee_no FROM mgr_table where mgr_id = $1 and employee_no = $2)
  THEN
   UPDATE employees
   SET department = $3, "_trigger_state_mgr_id" = $1 --> passing the state through the column
   WHERE employee_no = $2;
   COMMIT;
  END IF;
END;
$$;

Here is my procedure that implements the transfer service. In addition to checking if the manager is allowed to do the transfer, and doing the update, I set the additional column with the context value.

This column is declared in the metadata, is used by the update statement but is discarded by the trigger. If PostgreSQL had a feature like the Oracle invisible column, I would set this column as invisible as I don’t want to see it in SELECT * or INSERT to it. I can do the same with a view on top of the table.

In summary:

  • you may have less feature in open source software than commercial ones, but this probably means that there’s something to do differently in the application design. Commercial vendors can implement anything when important customers pay for it. An open source community will not add more complexity to workaround a problem that should be addressed by the application design. Don’t call SQL DML with trigger magic behind. Call a procedure that has all the logic, including all DML (main change and audit) and verifications
  • workarounds are valid as long as you document them (what they do, and why) and verify all side effects (like setting it to null afterwards). There are often many workarounds and choose the one that makes more sense. Here, maybe the “_trigger_state_mgr_id” will become a real column to persist one day, if there’s a need to store the latest manager that assigned the employee to a department

Cet article Pass a variable to a trigger in PostgreSQL est apparu en premier sur Blog dbi services.

Building a PostgreSQL EC2 AMI for testing the latest commits

$
0
0

Testing the latest features or bug fixes in PostgreSQL can easily be done by pulling the latest commits from the git repository and compile PostgreSQL from source code. I am doing that frequently on my local sandbox VM but this becomes limited when I need more CPUs or more memory. This is where the public cloud really shines, as you can easily deploy new VMs, do your tests and get rid of it afterwards with just a view clicks or commands. Bringing up a Debian VM in AWS, which already has the latest development snapshot up and running is something that can be done quite fast.

As a starting point I’ve used the official Debian AMI:

For creating your own customized AMI you could go with a micro or even a nano instance, but if you want to compile fast, more CPUs are better:

Your VPC should already be configured (or just use the default one) and if you want to connect from outside AWS you’ll need a public IP address:

How much storage you’ll need, depends on the tests you want to do. For now it is not really important as you can change that later on, once the AMI is ready:

Tags are always a good idea, especially if you have many instances running:

Of course you should configure the security group to allow SSH access:

Review, provide your key pair and launch the instance:


Once it is running grab the public IP address and connect (the user for Debian is “admin”, not “ec2-user”):

dwe@ltdwe:~$ ssh -i /home/dwe/Documents/aws/dwe-key-pair.pem admin@3.122.206.167
The authenticity of host '3.122.206.167 (3.122.206.167)' can't be established.
ECDSA key fingerprint is SHA256:E/W08OSWhym0k2SVQafxu3rCljqEVg/VC744sz3ilog.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '3.122.206.167' (ECDSA) to the list of known hosts.
Linux ip-10-0-1-35 4.19.0-16-cloud-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

Updating the OS to the latest release is always something I am doing as the first step:

admin@ip-10-0-1-35:~$ sudo apt update && sudo apt dist-upgrade -y

Install all packages you need for the compilation from source code (and some more in the list below):

admin@ip-10-0-1-35:~$ sudo apt install libldap2-dev libpython-dev libreadline-dev libssl-dev bison flex libghc-zlib-dev libcrypto++-dev libxml2-dev libxslt1-dev tcl tclcl-dev bzip2 wget screen libpam0g-dev libperl-dev make unzip libpam0g-dev python libsystemd-dev sudo llvm-7 llvm-7-dev clang pkg-config gcc g++ liblz4-dev pkg-config git -y

Configure the OS group and user you’ll use for the PostgreSQL installation:

admin@ip-10-0-1-35:~$ sudo groupadd postgres
admin@ip-10-0-1-35:~$ sudo useradd -g postgres -m -s /bin/bash postgres

Clone the PostgreSQL git repository:

admin@ip-10-0-1-35:~$ sudo su - postgres -c "git clone git://git.postgresql.org/git/postgresql.git"
Cloning into 'postgresql'...
remote: Enumerating objects: 836887, done.
remote: Counting objects: 100% (836887/836887), done.
remote: Compressing objects: 100% (121635/121635), done.
remote: Total 836887 (delta 719772), reused 828577 (delta 712107), pack-reused 0
Receiving objects: 100% (836887/836887), 255.06 MiB | 32.09 MiB/s, done.
Resolving deltas: 100% (719772/719772), done.

This is the small script that will do all the work for us:

admin@ip-10-0-1-35:~$ sudo su - postgres
postgres@ip-10-0-1-35:~$ cat compile.sh 
#!/bin/bash

rm -rf /home/postgres/pgdata
cd /home/postgres/postgresql
git pull
make distclean
./configure --prefix=/home/postgres/pgdev
make -j4 all
make install
cd contrib
make -j4 install
/home/postgres/pgdev/bin/initdb -D /home/postgres/pgdata
/home/postgres/pgdev/bin/pg_ctl -D /home/postgres/pgdata start -l /dev/null
/home/postgres/pgdev/bin/psql -c "select version()"

Once PostgreSQL is installed create a new AMI from the EC2 instance:

Having the new AMI ready we can easily launch an instance based on the AMI:

In the “Advanced Details” section provide the call to the script:

When the instance started up, you can follow the progress in the cloud init log file:

Happy testing 🙂

Cet article Building a PostgreSQL EC2 AMI for testing the latest commits est apparu en premier sur Blog dbi services.

PostgreSQL: set_config and current_setting

$
0
0

PostgreSQL comes with a rich set of administrative functions for various use cases. There are functions to control parameters, there are functions for index maintenance, others are for recovery and or backup control and there is much more. In this post we’ll look at how you can use the build-in functions to set and get parameters without touching the configuration file (or setting parameters at the cluster, database and user level). In the posts that follow, we’ll walk through various other functions, PostgreSQL provides by default.

If you want to set parameters at the session level, you’ll usually use the “set” command, e.g.:

postgres=# show work_mem;
 work_mem 
----------
 4MB
(1 row)

postgres=# set work_mem='6MB';
SET
postgres=# show work_mem;
 work_mem 
----------
 6MB
(1 row)

As this is on the session level, you’ll get back to the default setting, once you create a new session:

postgres=# \c postgres
You are now connected to database "postgres" as user "postgres".
postgres=# show work_mem;
 work_mem 
----------
 4MB
(1 row)

Another possibility you have for this requirement, is to use the build-in set_config function:

postgres=# select set_config('work_mem','6MB',true);
 set_config 
------------
 6MB
(1 row)

postgres=# show work_mem;
 work_mem 
----------
 4MB
(1 row)

The third parameter specifies the scope, and in this case the scope is the transaction. As psql by default auto commits and you’re not in a transaction, you fall back to the default immediately. Doing the same in an explicit transactions makes this more clear:

postgres=# begin;
BEGIN
postgres=*# select set_config('work_mem','6MB',true);
 set_config 
------------
 6MB
(1 row)

postgres=*# show work_mem;
 work_mem 
----------
 6MB
(1 row)

postgres=*# end;
COMMIT
postgres=# show work_mem;
 work_mem 
----------
 4MB
(1 row)

If you want to set a parameter for the scope of the whole session, just switch the third parameter:

postgres=# select set_config('work_mem','6MB',false);
 set_config 
------------
 6MB
(1 row)

postgres=# show work_mem ;
 work_mem 
----------
 6MB
(1 row)

postgres=# \c postgres
You are now connected to database "postgres" as user "postgres".
postgres=# show work_mem ;
 work_mem 
----------
 4MB
(1 row)

The opposite of the set_config function is current_setting:

postgres=# select current_setting('work_mem',false);
 current_setting 
-----------------
 4MB
(1 row)

The reason for the second parameter is, to give you choice what happens if you ask for a parameter that does not exist:

postgres=# select current_setting('work_memX',false);
ERROR:  unrecognized configuration parameter "work_memX"
postgres=# select current_setting('work_memX',true);
 current_setting 
-----------------
 
(1 row)

Depending on what behavior you want, you’ll either get NULL or an error message.

You might ask yourself: Why do I need these functions when I can do the same with set and show? The answer is easy: Functions can easily be used in other functions or procedures:

postgres=# \! cat a.sql
create or replace function f1() returns text as
$$
declare
  lv text;
begin
  select current_setting('work_mem')
    into lv;
  return lv;
end;
$$ language plpgsql;
postgres=# \i a.sql
CREATE FUNCTION
postgres=# select f1();
 f1  
-----
 4MB
(1 row)

This gives you the flexibility to ask for, and set parameters while you’re doing business logic in the database.

Cet article PostgreSQL: set_config and current_setting est apparu en premier sur Blog dbi services.

Getting started with Citus – Setting up a four node cluster

$
0
0

One issue with traditional database systems like PostgreSQL is, that you cannot easily scale vertically. Of course you could add read replicas and offload read operations, but that either requires changes in the application, or putting something in front that understands the PostgreSQL dialect and automatically routes writes to the primary and spreads reads across one or more replicas (e.g. pgpool-II). But even if you have something in place, you need to deal with replications lags or you need to go for synchronous replication (which comes with it’s own downsides). Another answer to vertically scaling is Citus. By using Citus you can have sharding in PostgreSQL by simple installing an extension.

The official Citus documentation is really good, so I will not repeat the concepts here. Basically you need one coordinator: This is the node receiving all the traffic from the application. All other nodes are so called worker nodes which perform the actual work, send the results back to the coordinator which finally accumulates the results.

We’ll start simple with one node, and will end up with one coordinator node and three worker nodes at the end of this post:

Important right from the beginning: Citus is not a fork of PostgreSQL. Citus comes as an extension and you can use it with plain community PostgreSQL. Nothing else is required. Of course you need to install the extension and there are pre-build packages for that, either for flavors of Debian of flavors of Red Hat/Fedora. In this post will walk through installing the Citus extension from source code, as I’ve installed PostgreSQL 13.2 from source code as well (this is already done, search this blog for “install postgres source code”, if you want to know how to do that).

The first thing to do is to check your environment. pg_config needs to be in your $PATH so the Citus extension can find it when it configures/compiles:

postgres@ip-10-0-1-23:/home/postgres/ [pg132] which pg_config 
/u01/app/postgres/product/13/db_2/bin/pg_config

Once that is ready the procedure for getting the Citus extension installed is quite simple. Either clone the git repository (which will give you the latest development snapshot) or download the latest release. For the scope of this post we’ll clone the repository:

postgres@ip-10-0-1-23:/home/postgres/ [pg132] git clone https://github.com/citusdata/citus.git
Cloning into 'citus'...
remote: Enumerating objects: 66167, done.
remote: Counting objects: 100% (1447/1447), done.
remote: Compressing objects: 100% (638/638), done.
remote: Total 66167 (delta 942), reused 1166 (delta 804), pack-reused 64720
Receiving objects: 100% (66167/66167), 33.13 MiB | 25.28 MiB/s, done.
Resolving deltas: 100% (47834/47834), done.

Having sources in place, configure, compile and install (I am installing the packages for Debian here, you need to adjust this for other Linux distributions):

postgres@ip-10-0-1-23:/home/postgres/ [pg132] cd citus/
postgres@ip-10-0-1-23:/home/postgres/citus/ [pg132] sudo apt install libghc-curl-dev
postgres@ip-10-0-1-23:/home/postgres/citus/ [pg132] sudo apt install libzstd-dev
postgres@ip-10-0-1-23:/home/postgres/citus/ [pg132] ./configure 
checking for a sed that does not truncate output... /usr/bin/sed
checking for gawk... no
checking for mawk... mawk
...
config.status: creating src/include/citus_config.h
config.status: creating src/include/citus_version.h

Ready to compile …

postgres@ip-10-0-1-23:/home/postgres/citus/ [pg132] make
Makefile:51: warning: overriding recipe for target 'check'
/u01/app/postgres/product/13/db_2/lib/pgxs/src/makefiles/pgxs.mk:433: warning: ignoring old recipe for target 'check'
make -C src/backend/distributed/ all
...
 -D_GNU_SOURCE -I/usr/include/libxml2  -I/u01/app/postgres/product/13/db_2/include -I/home/postgres/citus/vendor/safestringlib/include -flto=thin -emit-llvm -c -o ../columnar/write_state_management.bc ../columnar/write_state_management.c
make[1]: Leaving directory '/home/postgres/citus/src/backend/distributed'

.. and install:

postgres@ip-10-0-1-23:/home/postgres/citus/ [pg132] make install
Makefile:51: warning: overriding recipe for target 'check'
/u01/app/postgres/product/13/db_2/lib/pgxs/src/makefiles/pgxs.mk:433: warning: ignoring old recipe for target 'check'
make -C src/backend/distributed/ all
make[1]: Entering directory '/home/postgres/citus/src/backend/distributed'
...
/usr/bin/install -c -m 644 ./src/include/citus_version.h '/u01/app/postgres/product/13/db_2/include/server/'
/usr/bin/install -c -m 644 /home/postgres/citus/./src/include/distributed/*.h '/u01/app/postgres/product/13/db_2/include/server/distributed/'

From now on the Citus extension will be available for every PostgreSQL cluster which is initialized with these binaries. As usual, we’ll need to initialize a new cluster:

postgres@ip-10-0-1-23:/home/postgres/citus/ [pg132] mkdir -p /u02/pgdata/13/citus
postgres@ip-10-0-1-23:/home/postgres/citus/ [pg132] initdb -D /u02/pgdata/13/citus
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "C.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /u02/pgdata/13/citus ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    pg_ctl -D /u02/pgdata/13/citus -l logfile start

Start the cluster and install the Citus extension:

postgres@ip-10-0-1-23:/home/postgres/citus/ [pg132] echo "shared_preload_libraries = 'citus,pg_stat_statements'" >> /u02/pgdata/13/citus/postgresql.auto.conf 
postgres@ip-10-0-1-23:/home/postgres/citus/ [pg132] pg_ctl -D /u02/pgdata/13/citus/ start
postgres@ip-10-0-1-23:/home/postgres/citus/ [pg132] psql -c "select * from pg_available_extensions where name like '%citus%'" postgres
 name  | default_version | installed_version |          comment           
-------+-----------------+-------------------+----------------------------
 citus | 10.1-1          |                   | Citus distributed database
(1 row)

Installing the extension into a database works like any other extension (you’ll notice that the extension creates a self signed certificate by default):

postgres@ip-10-0-1-23:/home/postgres/citus/ [pg132] psql -c "create database citus" postgres
CREATE DATABASE
postgres@ip-10-0-1-23:/home/postgres/citus/ [pg132] psql -c "create extension citus" citus
2021-05-05 17:44:02.467 UTC [7306] LOG:  citus extension created on postgres without ssl enabled, turning it on during creation of the extension
2021-05-05 17:44:02.467 UTC [7306] CONTEXT:  SQL statement "SELECT citus_setup_ssl()"
        PL/pgSQL function inline_code_block line 5 at PERFORM
2021-05-05 17:44:02.467 UTC [7306] STATEMENT:  create extension citus
2021-05-05 17:44:02.477 UTC [7306] LOG:  no certificate present, generating self signed certificate
2021-05-05 17:44:02.477 UTC [7306] CONTEXT:  SQL statement "SELECT citus_setup_ssl()"
        PL/pgSQL function inline_code_block line 5 at PERFORM
2021-05-05 17:44:02.477 UTC [7306] STATEMENT:  create extension citus
2021-05-05 17:44:02.586 UTC [7289] LOG:  received SIGHUP, reloading configuration files
2021-05-05 17:44:02.587 UTC [7289] LOG:  parameter "ssl" changed to "on"
2021-05-05 17:44:02.587 UTC [7289] LOG:  parameter "ssl_ciphers" changed to "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384"
CREATE EXTENSION
postgres@ip-10-0-1-23:/home/postgres/citus/ [pg132] ls -latr /u02/pgdata/13/citus/server*
-rw------- 1 postgres postgres 1708 May  5 17:44 /u02/pgdata/13/citus/server.key
-rw------- 1 postgres postgres  981 May  5 17:44 /u02/pgdata/13/citus/server.crt

The Citus extension is ready:

postgres@ip-10-0-1-23:/home/postgres/citus/ [pg132] psql -c "select citus_version();" citus
2021-05-05 17:51:13.927 UTC [7343] LOG:  starting maintenance daemon on database 16384 user 10
2021-05-05 17:51:13.927 UTC [7343] CONTEXT:  Citus maintenance daemon for database 16384 user 10
                                                     citus_version                                                     
-----------------------------------------------------------------------------------------------------------------------
 Citus 10.1devel on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit gitref: master(sha: d0ba12206)
(1 row)

Cheking the processes, there is new background worker:

postgres@ip-10-0-1-23:/home/postgres/citus/ [pg132] ps ax | grep Citus
 7343 ?        Ss     0:00 postgres: Citus Maintenance Daemon: 16384/10 
 7351 pts/0    S+     0:00 grep Citus

Time to create some tables and data using pgbench:

postgres@ip-10-0-1-23:/home/postgres/ [citus] pgbench -i -s 10 citus
dropping old tables...
NOTICE:  table "pgbench_accounts" does not exist, skipping
NOTICE:  table "pgbench_branches" does not exist, skipping
NOTICE:  table "pgbench_history" does not exist, skipping
NOTICE:  table "pgbench_tellers" does not exist, skipping
creating tables...
generating data (client-side)...
1000000 of 1000000 tuples (100%) done (elapsed 4.18 s, remaining 0.00 s)
vacuuming...
creating primary keys...
done in 6.35 s (drop tables 0.00 s, create tables 0.03 s, client-side generate 4.43 s, vacuum 0.32 s, primary keys 1.56 s).

As of now these are standard tables and we can turn them into distributed tables by using the “create_distributed_table” function and provide the column we we want to shard on:

postgres@ip-10-0-1-23:/home/postgres/ [citus] psql -c "SELECT create_distributed_table('pgbench_accounts', 'aid')" citus
NOTICE:  Copying data from local table...
NOTICE:  copying the data has completed
DETAIL:  The local data in the table is no longer visible, but is still on disk.
HINT:  To remove the local data, run: SELECT truncate_local_data_after_distributing_table($$public.pgbench_accounts$$)
 create_distributed_table 
--------------------------
 
(1 row)
postgres@ip-10-0-1-23:/home/postgres/ [citus] psql -c "SELECT truncate_local_data_after_distributing_table(\$\$public.pgbench_accounts\$\$)" citus
 truncate_local_data_after_distributing_table 
----------------------------------------------
 
(1 row)

This created a distributed table and you can query citus_tables to get more information:

postgres@ip-10-0-1-23:/home/postgres/ [citus] psql -c "select * from citus_tables" citus
    table_name    | citus_table_type | distribution_column | colocation_id | table_size | shard_count | table_owner | access_method 
------------------+------------------+---------------------+---------------+------------+-------------+-------------+---------------
 pgbench_accounts | distributed      | aid                 |             1 | 151 MB     |          32 | postgres    | heap
(1 row)

Running a simple query against that table, you’ll notice that the execution plan shows other plan nodes than usually:

citus=# explain (analyze) select * from pgbench_accounts where aid=100;
                                                                                     QUERY PLAN                                                                                     
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Custom Scan (Citus Adaptive)  (cost=0.00..0.00 rows=0 width=0) (actual time=12.633..12.635 rows=1 loops=1)
   Task Count: 1
   Tuple data received from nodes: 89 bytes
   Tasks Shown: All
   ->  Task
         Tuple data received from node: 89 bytes
         Node: host=localhost port=5432 dbname=citus
         ->  Index Scan using pgbench_accounts_pkey_102024 on pgbench_accounts_102024 pgbench_accounts  (cost=0.29..8.30 rows=1 width=97) (actual time=0.019..0.021 rows=1 loops=1)
               Index Cond: (aid = 100)
             Planning Time: 0.407 ms
             Execution Time: 0.046 ms
 Planning Time: 0.481 ms
 Execution Time: 12.658 ms
(13 rows)

As we currently only have one node, all data is coming from this node. Time to add more nodes to the cluster. Before doing this, repeat the Citus installation on the remaining nodes, initiliaze a new PostgreSQL cluster in the same way as above, and make sure that authentication is configured so the nodes can talk to each other (pg_hba.conf). Before we can add worker nodes we need to specify which of the nodes is the coordinator node, and we’ll use the current one for this

postgres@ip-10-0-1-23:/home/postgres/ [citus] psql -c "SELECT citus_set_coordinator_host('10.0.1.23') " citus
 citus_set_coordinator_host 
----------------------------
 
(1 row)

Now we can add the workers:

postgres@ip-10-0-1-23:/home/postgres/ [citus] psql -c "SELECT * from citus_add_node('10.0.1.16', 5432)" citus
NOTICE:  shards are still on the coordinator after adding the new node
HINT:  Use SELECT rebalance_table_shards(); to balance shards data between workers and coordinator or SELECT citus_drain_node('10.0.1.23',5432); to permanently move shards away from the coordinator.
 citus_add_node 
----------------
              2
(1 row)

postgres@ip-10-0-1-23:/home/postgres/ [citus] psql -c "SELECT * from citus_add_node('10.0.1.220', 5432)" citus
 citus_add_node 
----------------
              3
(1 row)

postgres@ip-10-0-1-23:/home/postgres/ [citus] psql -c "SELECT * from citus_add_node('10.0.1.27', 5432)" citus
 citus_add_node 
----------------
              4
(1 row)

Distributing the data across the nodes is something you need to kick of manually:

postgres@ip-10-0-1-23:/home/postgres/ [citus] psql -c "SELECT rebalance_table_shards()" citus
NOTICE:  Moving shard 102008 from 10.0.1.23:5432 to 10.0.1.16:5432 ...
NOTICE:  Moving shard 102009 from 10.0.1.23:5432 to 10.0.1.220:5432 ...
NOTICE:  Moving shard 102010 from 10.0.1.23:5432 to 10.0.1.27:5432 ...
NOTICE:  Moving shard 102011 from 10.0.1.23:5432 to 10.0.1.16:5432 ...
NOTICE:  Moving shard 102012 from 10.0.1.23:5432 to 10.0.1.220:5432 ...
NOTICE:  Moving shard 102013 from 10.0.1.23:5432 to 10.0.1.27:5432 ...
NOTICE:  Moving shard 102014 from 10.0.1.23:5432 to 10.0.1.16:5432 ...
NOTICE:  Moving shard 102015 from 10.0.1.23:5432 to 10.0.1.220:5432 ...
NOTICE:  Moving shard 102016 from 10.0.1.23:5432 to 10.0.1.27:5432 ...
NOTICE:  Moving shard 102017 from 10.0.1.23:5432 to 10.0.1.16:5432 ...
NOTICE:  Moving shard 102018 from 10.0.1.23:5432 to 10.0.1.220:5432 ...
NOTICE:  Moving shard 102019 from 10.0.1.23:5432 to 10.0.1.27:5432 ...
NOTICE:  Moving shard 102020 from 10.0.1.23:5432 to 10.0.1.16:5432 ...
NOTICE:  Moving shard 102021 from 10.0.1.23:5432 to 10.0.1.220:5432 ...
NOTICE:  Moving shard 102022 from 10.0.1.23:5432 to 10.0.1.27:5432 ...
NOTICE:  Moving shard 102023 from 10.0.1.23:5432 to 10.0.1.16:5432 ...
NOTICE:  Moving shard 102024 from 10.0.1.23:5432 to 10.0.1.220:5432 ...
NOTICE:  Moving shard 102025 from 10.0.1.23:5432 to 10.0.1.27:5432 ...
NOTICE:  Moving shard 102026 from 10.0.1.23:5432 to 10.0.1.16:5432 ...
NOTICE:  Moving shard 102027 from 10.0.1.23:5432 to 10.0.1.220:5432 ...
NOTICE:  Moving shard 102028 from 10.0.1.23:5432 to 10.0.1.27:5432 ...
NOTICE:  Moving shard 102029 from 10.0.1.23:5432 to 10.0.1.16:5432 ...
NOTICE:  Moving shard 102030 from 10.0.1.23:5432 to 10.0.1.220:5432 ...
NOTICE:  Moving shard 102031 from 10.0.1.23:5432 to 10.0.1.27:5432 ...
 rebalance_table_shards 
------------------------
 
(1 row)

Finally remove the data from the coordinator node by draining the node:

postgres@ip-10-0-1-23:/home/postgres/ [citus] psql -c "SELECT citus_drain_node('10.0.1.23',5432)" citus
NOTICE:  Moving shard 102032 from 10.0.1.23:5432 to 10.0.1.16:5432 ...
NOTICE:  Moving shard 102033 from 10.0.1.23:5432 to 10.0.1.220:5432 ...
NOTICE:  Moving shard 102034 from 10.0.1.23:5432 to 10.0.1.27:5432 ...
NOTICE:  Moving shard 102035 from 10.0.1.23:5432 to 10.0.1.16:5432 ...
NOTICE:  Moving shard 102036 from 10.0.1.23:5432 to 10.0.1.220:5432 ...
NOTICE:  Moving shard 102037 from 10.0.1.23:5432 to 10.0.1.27:5432 ...
NOTICE:  Moving shard 102038 from 10.0.1.23:5432 to 10.0.1.16:5432 ...
NOTICE:  Moving shard 102039 from 10.0.1.23:5432 to 10.0.1.220:5432 ...
 citus_drain_node 
------------------
 
(1 row)

Doing the same query again, the execution plan shows that the data is not coming anymore from the local node:

postgres@ip-10-0-1-23:/home/postgres/ [citus] psql citus
psql (13.2)
Type "help" for help.

citus=# explain (analyze) select * from pgbench_accounts where aid=100;
                                                                                     QUERY PLAN                                           
------------------------------------------------------------------------------------------------------------------------------------------
 Custom Scan (Citus Adaptive)  (cost=0.00..0.00 rows=0 width=0) (actual time=20.801..20.803 rows=1 loops=1)
   Task Count: 1
   Tuple data received from nodes: 89 bytes
   Tasks Shown: All
   ->  Task
         Tuple data received from node: 89 bytes
         Node: host=10.0.1.220 port=5432 dbname=citus
         ->  Index Scan using pgbench_accounts_pkey_102024 on pgbench_accounts_102024 pgbench_accounts  (cost=0.29..8.30 rows=1 width=97) 
               Index Cond: (aid = 100)
             Planning Time: 1.794 ms
             Execution Time: 1.124 ms
 Planning Time: 2.193 ms
 Execution Time: 20.841 ms
(13 rows)

All the shards are evenly distributed across the nodes:

postgres@ip-10-0-1-23:/home/postgres/ [citus] psql -c "\d+" -h 10.0.1.16 citus
                                     List of relations
 Schema |          Name           | Type  |  Owner   | Persistence |  Size   | Description 
--------+-------------------------+-------+----------+-------------+---------+-------------
 public | citus_tables            | view  | postgres | permanent   | 0 bytes | 
 public | pgbench_accounts_102008 | table | postgres | permanent   | 4176 kB | 
 public | pgbench_accounts_102011 | table | postgres | permanent   | 4128 kB | 
 public | pgbench_accounts_102014 | table | postgres | permanent   | 4072 kB | 
 public | pgbench_accounts_102017 | table | postgres | permanent   | 4120 kB | 
 public | pgbench_accounts_102020 | table | postgres | permanent   | 4136 kB | 
 public | pgbench_accounts_102023 | table | postgres | permanent   | 4144 kB | 
 public | pgbench_accounts_102026 | table | postgres | permanent   | 4152 kB | 
 public | pgbench_accounts_102029 | table | postgres | permanent   | 4136 kB | 
 public | pgbench_accounts_102032 | table | postgres | permanent   | 4104 kB | 
 public | pgbench_accounts_102035 | table | postgres | permanent   | 4104 kB | 
 public | pgbench_accounts_102038 | table | postgres | permanent   | 4136 kB | 
(12 rows)

postgres@ip-10-0-1-23:/home/postgres/ [citus] psql -c "\d+" -h 10.0.1.220 citus
                                     List of relations
 Schema |          Name           | Type  |  Owner   | Persistence |  Size   | Description 
--------+-------------------------+-------+----------+-------------+---------+-------------
 public | citus_tables            | view  | postgres | permanent   | 0 bytes | 
 public | pgbench_accounts_102009 | table | postgres | permanent   | 4152 kB | 
 public | pgbench_accounts_102012 | table | postgres | permanent   | 4144 kB | 
 public | pgbench_accounts_102015 | table | postgres | permanent   | 4136 kB | 
 public | pgbench_accounts_102018 | table | postgres | permanent   | 4128 kB | 
 public | pgbench_accounts_102021 | table | postgres | permanent   | 4128 kB | 
 public | pgbench_accounts_102024 | table | postgres | permanent   | 4136 kB | 
 public | pgbench_accounts_102027 | table | postgres | permanent   | 4128 kB | 
 public | pgbench_accounts_102030 | table | postgres | permanent   | 4144 kB | 
 public | pgbench_accounts_102033 | table | postgres | permanent   | 4160 kB | 
 public | pgbench_accounts_102036 | table | postgres | permanent   | 4136 kB | 
 public | pgbench_accounts_102039 | table | postgres | permanent   | 4152 kB | 
(12 rows)

postgres@ip-10-0-1-23:/home/postgres/ [citus] psql -c "\d+" -h 10.0.1.27 citus
                                     List of relations
 Schema |          Name           | Type  |  Owner   | Persistence |  Size   | Description 
--------+-------------------------+-------+----------+-------------+---------+-------------
 public | citus_tables            | view  | postgres | permanent   | 0 bytes | 
 public | pgbench_accounts_102010 | table | postgres | permanent   | 4136 kB | 
 public | pgbench_accounts_102013 | table | postgres | permanent   | 4152 kB | 
 public | pgbench_accounts_102016 | table | postgres | permanent   | 4128 kB | 
 public | pgbench_accounts_102019 | table | postgres | permanent   | 4136 kB | 
 public | pgbench_accounts_102022 | table | postgres | permanent   | 4128 kB | 
 public | pgbench_accounts_102025 | table | postgres | permanent   | 4144 kB | 
 public | pgbench_accounts_102028 | table | postgres | permanent   | 4136 kB | 
 public | pgbench_accounts_102031 | table | postgres | permanent   | 4136 kB | 
 public | pgbench_accounts_102034 | table | postgres | permanent   | 4128 kB | 
 public | pgbench_accounts_102037 | table | postgres | permanent   | 4112 kB | 
(11 rows)

That’s it for the initial setup. In a future post we’ll dive into the system in more detail.

Cet article Getting started with Citus – Setting up a four node cluster est apparu en premier sur Blog dbi services.

PostgreSQL, pg_ctl and signals

$
0
0

I believe most people use pg_ctl nowadays to start, stop, or restart a PostgreSQL instance. You can do all these tasks without using pg_ctl, but you’ll notice in a minute that pg_ctl really is your friend. Even if you’re using pg_ctl already, is is good to know what happens in the background and this is the scope of this post.

For a moment, let’s imagine there is not pg_ctl: How can you start a PostgreSQL instance without it? The most simple solution to that is this:

postgres@debian10pg:/home/postgres/ [pgdev] postgres -D /u02/pgdata/DEV/
2021-05-14 15:00:47.309 CEST - 1 - 2793 -  - @ LOG:  redirecting log output to logging collector process
2021-05-14 15:00:47.309 CEST - 2 - 2793 -  - @ HINT:  Future log output will appear in directory "pg_log".

The downside with this is, that PostgreSQL will run in the foreground and shutdown as soon as the session ends or you abort the command:

postgres@debian10pg:/home/postgres/ [pgdev] CRTL-C
postgres@debian10pg:/home/postgres/ [pgdev] psql
psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?

Does this behavior make sense? It might not make sense when you start PostgreSQL manually, but it does make sense if:

  • You want to start PostgreSQL with systemd and you’re using Type=notify
  • You run PostgreSQL in a container. In this case you do not want to detach, as the container is considered dead when the postmaster dies.

There are probably other cases, but for me those are the two important ones.

If you want to start PostgreSQL in the background you would need to do something like this:

postgres@debian10pg:/home/postgres/ [pgdev] postgres -D /u02/pgdata/DEV/ >/var/tmp/pg.log 2>&1 &
[1] 2806
postgres@debian10pg:/home/postgres/ [pgdev] psql
psql (14devel)
Type "help" for help.

Nothing is wrong with that, but how do you stop PostgreSQL without using pg_ctl? The answer are signals. Way back in 2012 I’ve written a small post about signals and I think it is still worth reading. In very much the same way as written in this old post, you can control PostgreSQL by sending signals to the postmaster. For shutting down in “fast” mode, this is:

postgres@debian10pg:/home/postgres/ [pgdev] head -1 /u02/pgdata/DEV/postmaster.pid 
2806
postgres@debian10pg:/home/postgres/ [pgdev] kill -SIGINT 2806
postgres@debian10pg:/home/postgres/ [pgdev] psql
psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
[1]+  Done                    postgres -D /u02/pgdata/DEV/ > /var/tmp/pg.log 2>&1

The other signals you can use for shutting down, are SIGTERM(smart) and SIGQUIT(immediate). pg_ctl is doing nothing different in the background, but of course it is much easier to use the switches of pg_ctl, than to remember the signals all the time. Here is a fragment of the source code from pg_ctl, where you can see the mapping of the signals to the shutdown modes:

        if (strcmp(modeopt, "s") == 0 || strcmp(modeopt, "smart") == 0)
        {
                shutdown_mode = SMART_MODE;
                sig = SIGTERM;
        }
        else if (strcmp(modeopt, "f") == 0 || strcmp(modeopt, "fast") == 0)
        {
                shutdown_mode = FAST_MODE;
                sig = SIGINT;
        }
        else if (strcmp(modeopt, "i") == 0 || strcmp(modeopt, "immediate") == 0)
        {
                shutdown_mode = IMMEDIATE_MODE;
                sig = SIGQUIT;
        }

The same is true for starting up: pg_ctl is starting PostgreSQL in the background automatically and you do not need to care about this:

postgres@debian10pg:/home/postgres/ [pgdev] pg_ctl start
waiting for server to start....2021-05-14 15:18:07.159 CEST - 1 - 2856 -  - @ LOG:  redirecting log output to logging collector process
2021-05-14 15:18:07.159 CEST - 2 - 2856 -  - @ HINT:  Future log output will appear in directory "pg_log".
 done
server started

There is another signal you can use: SIGHUP. When you change a configuration in postgresql.conf, postgresql.auto.conf or pg_hba.conf you need to tell the postmaster about that change. Given the parameter can be changed online, or a change in pg_hba.conf (which can always be done online) you can tell the postmaster to re-read it’s configuration files:

postgres@debian10pg:/home/postgres/ [pgdev] head -1 /u02/pgdata/DEV/postmaster.pid 
2856
postgres@debian10pg:/home/postgres/ [pgdev] kill -SIGHUP 2856

Take a look at the log file and you’ll see the exact same messages, as if you would have used “pg_ctl reload”:

2021-05-14 15:21:02.623 CEST - 8 - 2856 -  - @ LOG:  received SIGHUP, reloading configuration files

Instead of using pg_terminate_backend to terminate a process, you can send a signal instead:

postgres@debian10pg:/home/postgres/ [pg14] ps -ef | grep psql
postgres  2908  2289  0 15:30 pts/1    00:00:00 psql
postgres  3086  2918  0 15:31 pts/0    00:00:00 grep psql
postgres@debian10pg:/home/postgres/ [pg14] kill -SIGTERM 2908
-- the session of the backend reports:
psql (14devel)
Type "help" for help.

[local]:5432 postgres@postgres=# Terminated

You see, pg_ctl and pg_terminate_backend are your friends, as it is more clear what happens if you can use something descriptive, instead of sending signals directly.

Cet article PostgreSQL, pg_ctl and signals est apparu en premier sur Blog dbi services.


PostgreSQL on AWS Graviton2: CFLAGS

$
0
0

By Franck Pachot

.

In a previous post I’ve compiled PostgreSQL with GCC 7 and GCC 11 with the default options and checked that the ARM v8.2 features were used. However it may not be so simple:

  • PostgreSQL ./configure defines gcc flags to produce a binary that is compatible with older versions of ARM. This means that it may not use the LSE feature introduced in ARM v8.1
  • With this, the default compilation (-march=armv8-a) doesn’t use LSE (Large System Extensions) for atomic instructions, because they were introduced in v8.1
  • This means that we need to call ./configure with CFLAGS=”-march=armv8-a+lse” or CFLAGS=”-march=armv8.1-a” or CFLAGS=”-march=armv8.2-a” to get a binary using LSE atomics
  • When I compiled with GCC 11, without adding this CFLAGS, I had the LSE instructions in my binary because GCC 10 introduced the possibility to detect at run-time the support of LSE, and it is enabled by default
  • When I compiled with GCC 7 I had the LSE instructions, despites using the default configure, because I was running on Amazon Linux 2 where this run-time detection has been backported, and enabled by default
  • However, this run-time detection has been backported, but is not enabled by default on all versions, or distributions, and then may require the -moutline-atomics to use LSE

So, the most important is to understand if your binary will use LSE (which shows great performance improvement on spinlock). And it is also good to know if LSE is inlined, or outlined by this run-time detection (which has a limited overhead but allows the binary to be compatible with all ARM versions).

I have started a t4g.micro EC2 instance which runs on ARM v8.2 Neoverse N1, the Amazon Graviton2. I’m using it because we have 750 hours per month free until June 30th, 2021 and I try to get my demos easy and free to reproduce. But of course this instance size is too small for running a database and compiling gcc.

[ec2-user@ip-172-31-83-114 postgresql-14devel]$ PS1="$(curl -s http://169.254.169.254/latest/meta-data/instance-type)# "
t4g.micro#

I’ve set my prompt to the instance type because I’ve a lot of tty opened and I like to see which one is still running (for the non-free ones especially).

I am running on Amazon Linux 7 with GCC 7


t4g.micro# cat /etc/system-release
Amazon Linux release 2 (Karoo)

t4g.micro# gcc --version
gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-12)
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

But here the runtime detection and outlining atomics has been backported and enabled by default.

I’ll will verify that by building the PostgreSQL binaries.


sudo yum install -y git gcc readline-devel bison-devel zlib-devel
curl -s https://ftp.postgresql.org/pub/snapshot/dev/postgresql-snapshot.tar.gz | tar -zxvf -
cd postgresql-14devel
./configure  --enable-debug
make clean
make
sudo make install

This downloads PostgreSQL source (version 14 from latest development snapshot here) and compiles with all default options.


t4g.micro# file /usr/local/pgsql/bin/postgres

/usr/local/pgsql/bin/postgres: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 3.7.0, BuildID[sha1]=37c48090a793b8ee8b62e46e361ac4c24646732f, not stripped

I’m checking the load and store exclusives, the only ones available before ARM v8.1, and LSE which can replace them with atomic instructions starting from ARM v8.1


t4g.micro# objdump -d /usr/local/pgsql/bin/postgres | awk '/\t(ldxr|ldaxr|stxr|stlxr)/{print $3"\t(load and store exclusives)"}/\t(cas|casp|swp|ldadd|stadd|ldclr|stclr|ldeor|steor|ldset|stset|ldsmax|stsmax|ldsmin|stsmin|ldumax|stumax|ldumin|stumin)/{print $3"\t(large-system extensions)"}' | sort -k2 | uniq -c

      2 casal   (large-system extensions)
      2 ldaddal (large-system extensions)
      1 ldclral (large-system extensions)
      1 ldsetal (large-system extensions)
      1 swpa    (large-system extensions)
      7 ldaxr   (load and store exclusives)
      6 stlxr   (load and store exclusives)
      1 stxr    (load and store exclusives)

I see both here. Instructions like CASAL (compare and swap) and SWPA (swap words) are LSE atomics. LDAXR and STXR are the instructions used before LSE.

Both are there because I compiled with the GCC versions that builds a binary with run-time detection to use one or the other. I’ll take one function that makes use of atomics. If you have a high rate of writes in your PostgreSQL databases, from multiple sessions, you may have seen contention on WAL generation. Especially in case of CPU starvation because XLogInsert->XLogInsertRecord->ReserveXLogInsertLocation uses a spinlock to access the shared memory structure.
Let’s see what the compiler has generated for this:


t4g.micro# gdb --batch /usr/local/pgsql/bin/postgres -ex 'disas ReserveXLogInsertLocation'

Dump of assembler code for function ReserveXLogInsertLocation:
   0x000000000055ff18 :     stp     x29, x30, [sp, #-80]!
   0x000000000055ff1c :     mov     x29, sp
   0x000000000055ff20 :     str     w0, [sp, #44]
   0x000000000055ff24 :    str     x1, [sp, #32]
   0x000000000055ff28 :    str     x2, [sp, #24]
   0x000000000055ff2c :    str     x3, [sp, #16]
   0x000000000055ff30 :    adrp    x0, 0xde0000 
   0x000000000055ff34 :    add     x0, x0, #0x550
   0x000000000055ff38 :    ldr     x0, [x0]
   0x000000000055ff3c :    str     x0, [sp, #72]
   0x000000000055ff40 :    ldr     w0, [sp, #44]
   0x000000000055ff44 :    add     w0, w0, #0x7
   0x000000000055ff48 :    and     w0, w0, #0xfffffff8
   0x000000000055ff4c :    str     w0, [sp, #44]
   0x000000000055ff50 :    ldr     x0, [sp, #72]
   0x000000000055ff54 :    bl      0x55f9b0 
...

This TAS is an atomic Test And Set function. There is a PostgreSQL wiki page about atomics implementation (https://wiki.postgresql.org/wiki/Atomics) but not really up to date. It says that the ARM for TAS is LDREX/STREX load and store instructions. Let’s look at this in my binary:


t4g.micro# gdb --batch /usr/local/pgsql/bin/postgres -ex 'disas tas'

Dump of assembler code for function tas:
   0x0000000000514784 :     stp     x29, x30, [sp, #-32]!
   0x0000000000514788 :     mov     x29, sp
   0x000000000051478c :     str     x0, [sp, #24]
   0x0000000000514790 :    ldr     x1, [sp, #24]
   0x0000000000514794 :    mov     w0, #0x1                        // #1
   0x0000000000514798 :    bl      0xb4f080 
   0x000000000051479c :    ldp     x29, x30, [sp], #32
   0x00000000005147a0 :    ret
End of assembler dump.

What is interesting here is the call to __aarch64_swp4_acq which is the outlined atomics.


t4g.micro# gdb --batch /usr/local/pgsql/bin/postgres -ex 'disas __aarch64_swp4_acq'

Dump of assembler code for function __aarch64_swp4_acq:
   0x0000000000b4f080 :     hint    #0x22
   0x0000000000b4f084 :     adrp    x16, 0xe11000 
   0x0000000000b4f088 :     ldrb    w16, [x16, #1848]
   0x0000000000b4f08c :    cbz     w16, 0xb4f098 
   0x0000000000b4f090 :    swpa    w0, w0, [x1]
   0x0000000000b4f094 :    ret
   0x0000000000b4f098 :    mov     w16, w0
   0x0000000000b4f09c :    ldaxr   w0, [x1]
   0x0000000000b4f0a0 :    stxr    w17, w16, [x1]
   0x0000000000b4f0a4 :    cbnz    w17, 0xb4f09c 
   0x0000000000b4f0a8 :    ret
End of assembler dump.

This is where we can see both LSE (SWPA) for ARM v8.1 or later and load store exclusives (LDAXR STRX) for previous versions. The latter is in a loop (CBNZ branches to LDAXR) but the former is one atomic instruction. The decision is done by CBZ to branch to one or the other.

I have installed GCC 11 where outlined atomics is the default, in all Linux distributions (and doing that on a larger size instance).


CFLAGS="" ./configure ./configure && make clean && make && sudo make install

This shows the same as above: a __aarch64_swp4_acq function with run-time detection and two branches, LSE or loop on load/store.

However if I explicitely compile for a version of ARM which has LSE atomics, there’s no need for this runtime detection:


CFLAGS="-march=armv8.2-a" ./configure && make clean && make && sudo make install

c6gn.xlarge# gdb --batch /usr/local/pgsql/bin/postgres -ex 'disas tas'

Dump of assembler code for function tas:
   0x00000000005146b4 :     sub     sp, sp, #0x10
   0x00000000005146b8 :     str     x0, [sp, #8]
   0x00000000005146bc :     ldr     x0, [sp, #8]
   0x00000000005146c0 :    mov     w1, #0x1                        // #1
   0x00000000005146c4 :    swpa    w1, w1, [x0]
   0x00000000005146c8 :    mov     w0, w1
   0x00000000005146cc :    add     sp, sp, #0x10
   0x00000000005146d0 :    ret
End of assembler dump.

The TAS doesn’t call a __aarch64_swp4_acq function and the SWPA is in the function (not outlined).

I have the same result with:


CFLAGS="-mcpu=neoverse-n1" ./configure && make clean && make && sudo make install

as GCC 11 knows that Neoverse N1 is ARM v8.2 and -mcpu defines -march and -mtune if not specified otherwise

Note, however, that:


CFLAGS="-mtune=neoverse-n1" ./configure && make clean && make && sudo make install

still generates __aarch64_swp4_acq to be compatible with pre v8.1 ARM (use -mcpu and not -mtune if you don’t want them, or add -mno-outline-atomics to -mtune)

Note also that:


c6gn.xlarge# CFLAGS="-march=armv8-a+lse" ./configure && make clean && make && sudo make install

generates the LSE only (without runtime detection)

But:


c6gn.xlarge# CFLAGS="-march=armv8-a -mno-outline-atomics" ./configure && make clean && make && sudo make install

c6gn.xlarge# gdb --batch /usr/local/pgsql/bin/postgres -ex 'disas tas'
Dump of assembler code for function tas:
   0x0000000000514730 :     sub     sp, sp, #0x10
   0x0000000000514734 :     str     x0, [sp, #8]
   0x0000000000514738 :     ldr     x0, [sp, #8]
   0x000000000051473c :    mov     w1, #0x1                        // #1
   0x0000000000514740 :    ldxr    w2, [x0]
   0x0000000000514744 :    stxr    w3, w1, [x0]
   0x0000000000514748 :    cbnz    w3, 0x514740 
   0x000000000051474c :    dmb     ish
   0x0000000000514750 :    mov     w0, w2
   0x0000000000514754 :    add     sp, sp, #0x10
   0x0000000000514758 :    ret
End of assembler dump.

generates only load store exclusives.

You can find many blog posts with benchmarks. But, from what we have seen, the optimizations of Graviton2 depend on many parameters: gcc version, compilation flags, host cpu usage, and of course how spinlock is a bottleneck for your workload. The most important to understand is where the optimization shows up. In order to avoid severe performance degradation when processes are in spinlock contention and scheduled out of the CPU, we need to provision the instance size, or the autoscaling threshold, to avoid those peaks of CPU starvation. With those ARM optimizations, a smaller size may be acceptable for short peaks. This concurs to lower the cost when running on ARM.

So, if you are not sure about outline atomics being the default, like before version 10 of GCC, better add CFLAGS=”-moutline-atomics” to benefit from LSE when available. A patch to add it by default for that was rejected by the PostgreSQL community: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64.
With recent GCC, the best is probably CFLAGS=”-mcpu=neoverse-n1″ for Graviton2. Or CFLAGS=”-mcpu=neoverse-n1 -moutline-atomics” if you want the binary to be compatible with non-LSE processors. And above all, in case of doubt, don’t guess and check the binaries.
All this is quite new, any feedback welcome:

Cet article PostgreSQL on AWS Graviton2: CFLAGS est apparu en premier sur Blog dbi services.

Why is the default postgres database slightly larger than template1?

$
0
0

You probably know that a fresh initialized PostgreSQL cluster comes with three databases by default: template0, template1, and postgres. If you want to know why they are there, and what their purpose is, check this post. In this post we’ll look at something you may never have noticed: Why is the postgres database, even after a fresh initdb, slightly larger than template1 and template0?

Let’s start from scratch and initialize a new cluster:

postgres@debian10pg:/home/postgres/ [pg14] initdb -D /var/tmp/dummy
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

creating directory /var/tmp/dummy ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Europe/Zurich
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    pg_ctl -D /var/tmp/dummy -l logfile start

Once we start start that up and ask for the size of the databases, you’ll notice the difference:

postgres@debian10pg:/home/postgres/ [pg14] export PGPORT=8888
postgres@debian10pg:/home/postgres/ [pg14] pg_ctl -D /var/tmp/dummy start
waiting for server to start....2021-05-20 11:04:49.527 CEST [2708] LOG:  starting PostgreSQL 14devel on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2021-05-20 11:04:49.528 CEST [2708] LOG:  listening on IPv6 address "::1", port 8888
2021-05-20 11:04:49.528 CEST [2708] LOG:  listening on IPv4 address "127.0.0.1", port 8888
2021-05-20 11:04:49.541 CEST [2708] LOG:  listening on Unix socket "/tmp/.s.PGSQL.8888"
2021-05-20 11:04:49.557 CEST [2709] LOG:  database system was shut down at 2021-05-20 11:03:23 CEST
2021-05-20 11:04:49.566 CEST [2708] LOG:  database system is ready to accept connections
 done
server started

postgres@debian10pg:/home/postgres/ [pg14] psql -c "\l+"
                                                                    List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   |  Size   | Tablespace |                Description                 
-----------+----------+----------+-------------+-------------+-----------------------+---------+------------+--------------------------------------------
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |                       | 8265 kB | pg_default | default administrative connection database
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +| 8113 kB | pg_default | unmodifiable empty database
           |          |          |             |             | postgres=CTc/postgres |         |            | 
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +| 8113 kB | pg_default | default template for new databases
           |          |          |             |             | postgres=CTc/postgres |         |            | 
(3 rows)

The results may differ in your environment, depending on what version of PostgreSQL you are using. This is 14devel, but it doesn’t really matter: You will see that the “postgres” database is a bit larger than the other two. But how can this be? The first database which is created by initdb, is template1. The other two are just copies of it. But then they should all have the same size, shouldn’t they?

To understand what’s going on here, we first need to know how the database size is calculated, when you use the “\l+” shortcut. This is easy to check:

postgres=# \set ECHO_HIDDEN on
postgres=# \l+
********* QUERY **********
SELECT d.datname as "Name",
       pg_catalog.pg_get_userbyid(d.datdba) as "Owner",
       pg_catalog.pg_encoding_to_char(d.encoding) as "Encoding",
       d.datcollate as "Collate",
       d.datctype as "Ctype",
       pg_catalog.array_to_string(d.datacl, E'\n') AS "Access privileges",
       CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')
            THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))
            ELSE 'No Access'
       END as "Size",
       t.spcname as "Tablespace",
       pg_catalog.shobj_description(d.oid, 'pg_database') as "Description"
FROM pg_catalog.pg_database d
  JOIN pg_catalog.pg_tablespace t on d.dattablespace = t.oid
ORDER BY 1;
**************************

                                                                    List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   |  Size   | Tablespace |                Description                 
-----------+----------+----------+-------------+-------------+-----------------------+---------+------------+--------------------------------------------
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |                       | 8265 kB | pg_default | default administrative connection database
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +| 8113 kB | pg_default | unmodifiable empty database
           |          |          |             |             | postgres=CTc/postgres |         |            | 
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +| 8113 kB | pg_default | default template for new databases
           |          |          |             |             | postgres=CTc/postgres |         |            | 
(3 rows)

The statement that is used in the background uses the pg_database_size administrative function to calculate the size for each database. But what is pg_database_size actually doing? Time to check the source. pg_database_size is defined in “src/backend/utils/adt/dbsize.c”:

Datum
pg_database_size_oid(PG_FUNCTION_ARGS)
{
        Oid                     dbOid = PG_GETARG_OID(0);
        int64           size;

        size = calculate_database_size(dbOid);

        if (size == 0)
                PG_RETURN_NULL();

        PG_RETURN_INT64(size);
}

This in fact calls “calculate_database_size”, which is defined in the same file. If you check this one you’ll see this:

...
        /* Shared storage in pg_global is not counted */

        /* Include pg_default storage */
        snprintf(pathname, sizeof(pathname), "base/%u", dbOid);
        totalsize = db_dir_size(pathname);

        /* Scan the non-default tablespaces */
        snprintf(dirpath, MAXPGPATH, "pg_tblspc");
        dirdesc = AllocateDir(dirpath);
...

All it does is to calculate the size of the directory in db_dir_size (and any tablespaces if there are any):

/* Return physical size of directory contents, or 0 if dir doesn't exist */
static int64
db_dir_size(const char *path)
...
        while ((direntry = ReadDir(dirdesc, path)) != NULL)
        {

It is looping over all the files and then sums up the size of the files it finds there. Nothing is excluded as you can easily verify:

postgres@debian10pg:/home/postgres/ [pg14] cd /var/tmp/dummy/base/
postgres@debian10pg:/var/tmp/dummy/base/ [pg14] oid2name 
All databases:
    Oid  Database Name  Tablespace
----------------------------------
  13974       postgres  pg_default
  13973      template0  pg_default
      1      template1  pg_default
postgres@debian10pg:/var/tmp/dummy/base/ [pg14] cd 13974
postgres@debian10pg:/var/tmp/dummy/base/13974/ [pg14] dd if=/dev/zero of=dummy bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.0924525 s, 1.1 GB/s

postgres@debian10pg:/var/tmp/dummy/base/13974/ [pg14] psql -c "\l+"
                                                                    List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   |  Size   | Tablespace |                Description                 
-----------+----------+----------+-------------+-------------+-----------------------+---------+------------+--------------------------------------------
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |                       | 108 MB  | pg_default | default administrative connection database
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +| 8113 kB | pg_default | unmodifiable empty database
           |          |          |             |             | postgres=CTc/postgres |         |            | 
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +| 8113 kB | pg_default | default template for new databases
           |          |          |             |             | postgres=CTc/postgres |         |            | 
(3 rows)

postgres@debian10pg:/var/tmp/dummy/base/13974/ [pg14] rm dummy 

All I did here, is to create a file a bit larger than 100MB in the directory of the “postgres” database, and asked for the database size once more. As you can see above, the size of this file is counted.

What information can we get out of this? The difference between the “postgres” and the “template0” and “template1” databases must come from the files on disk. Let’s look at the directories:

postgres@debian10pg:/home/postgres/ [pg14] du -s /var/tmp/dummy/base/13974/
8280    /var/tmp/dummy/base/13974/   ## postgres
postgres@debian10pg:/home/postgres/ [pg14] du -s /var/tmp/dummy/base/1/
8124    /var/tmp/dummy/base/1/       ## template1

Same picture here: “postgres” is larger than “template1” but what is the difference. Actually the difference can easily be spotted:

postgres@debian10pg:/home/postgres/ [pg14] ls -ltr /var/tmp/dummy/base/13974/ | tail -3    # postgres
-rw------- 1 postgres postgres   8192 May 20 11:03 1247_vm
-rw------- 1 postgres postgres 114688 May 20 11:03 1247
-rw------- 1 postgres postgres 156486 May 20 11:06 pg_internal.init
postgres@debian10pg:/home/postgres/ [pg14] ls -ltr /var/tmp/dummy/base/1/ | tail -3        # template1
-rw------- 1 postgres postgres  24576 May 20 11:03 13792_fsm
-rw------- 1 postgres postgres  65536 May 20 11:03 13792
-rw------- 1 postgres postgres 106496 May 20 11:03 1259

To verify that this file really is the reason: Stop the instance, delete the file and compare again:

postgres@debian10pg:/home/postgres/ [pg14] pg_ctl -D /var/tmp/dummy/ stop
waiting for server to shut down.... done
server stopped
postgres@debian10pg:/home/postgres/ [pg14] rm /var/tmp/dummy/base/13974/pg_internal.init
postgres@debian10pg:/home/postgres/ [pg14] du -s /var/tmp/dummy/base/13974/
8124    /var/tmp/dummy/base/13974/
postgres@debian10pg:/home/postgres/ [pg14] du -s /var/tmp/dummy/base/1/
8124    /var/tmp/dummy/base/1/

Exactly the same now. Once you start the instance again, the file will be re-created:

postgres@debian10pg:/home/postgres/ [pg14] pg_ctl -D /var/tmp/dummy/ start
postgres@debian10pg:/home/postgres/ [pg14] ls -ltr /var/tmp/dummy/base/13974/ | tail -3
-rw------- 1 postgres postgres 770048 May 20 11:03 1255
-rw------- 1 postgres postgres   8192 May 20 11:03 1247_vm
-rw------- 1 postgres postgres 114688 May 20 11:03 1247
postgres@debian10pg:/home/postgres/ [pg14] psql -c "\l+"
                                                                    List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   |  Size   | Tablespace |                Description                 
-----------+----------+----------+-------------+-------------+-----------------------+---------+------------+--------------------------------------------
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |                       | 8265 kB | pg_default | default administrative connection database
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +| 8113 kB | pg_default | unmodifiable empty database
           |          |          |             |             | postgres=CTc/postgres |         |            | 
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +| 8113 kB | pg_default | default template for new databases
           |          |          |             |             | postgres=CTc/postgres |         |            | 
(3 rows)

postgres@debian10pg:/home/postgres/ [pg14] ls -ltr /var/tmp/dummy/base/13974/ | tail -3
-rw------- 1 postgres postgres   8192 May 20 11:03 1247_vm
-rw------- 1 postgres postgres 114688 May 20 11:03 1247
-rw------- 1 postgres postgres 156486 May 20 11:45 pg_internal.init

Last question for today: What is that file for? The answer is, once more, in the source code (src/include/utils/relcache.h):

/*
 * Name of relcache init file(s), used to speed up backend startup
 */
#define RELCACHE_INIT_FILENAME  "pg_internal.init"

Cet article Why is the default postgres database slightly larger than template1? est apparu en premier sur Blog dbi services.

PostgreSQL on Oracle free tier ARM

$
0
0

By Franck Pachot

.
This follows the previous post about the Oracle Cloud ARM compute shape just announced on May 25th. The processor is ARM v8.2 with LSE (atomic instructions) and PostgreSQL can benefit from it (see Dramatical Effect of LSE Instructions for PostgreSQL on Graviton2 Instances).

I have installed GCC 11 in the previous post, on a Oracle Linux 7.9 image with comes with GCC 7. If you installed the Ubuntu 20.4 image, you have GCC 9 and I also recommand to install the latest version. Anyway, if you use a version before GCC 11 you must set CFLAGS as below to benefit from ARM v8.1 LSE.

I’ll install PostgreSQL 14 which is currently in beta1:


(
sudo yum install -y git gcc readline-devel bison-devel zlib-devel
curl -s https://ftp.postgresql.org/pub/snapshot/dev/postgresql-snapshot.tar.gz | tar -zxvf -
sudo rm -rf /usr/local/pgsql ; sudo mkdir -p /usr/local/pgsql
cd postgresql-14beta1
make clean
CFLAGS="-march=armv8.2-a+fp16" ./configure --enable-debug
make
sudo make install
cd contrib
make
sudo make install
)

I initially tried to use CFLAGS=”-mcpu=neoverse-n1″ to compile for this processor only (GCC 11 defaults to -m_ouline_atomics to build binaries compatible with ARM v8 but this adds the very small overhead of runtime detection) but I got “unknown architectural extension `rcpc+dotprod+profile'” and that’s why I used CFLAGS=”-march=armv8.2-a+fp16″


[opc@arm-lon ~]$ nm /usr/local/pgsql/bin/postgres | grep -E "aarch64(_have_lse_atomics)?"                                                     

[opc@arm-lon ~]$ objdump -d /usr/local/pgsql/bin/postgres | awk '/\t(ldxr|ldaxr|stxr|stlxr)/{print $3"\t(load and store exclusives)"}/\t(cas|casp|swp|ldadd|stadd|ldclr|stclr|ldeor|steor|ldset|stset|ldsmax|stsmax|ldsmin|stsmin|ldumax|stumax|ldumin|stumin)/{print $3"\t(large-system extensions)"}' | sort | uniq -c | sort -n

      1 ldclral (large-system extensions)
      3 ldsetal (large-system extensions)
     12 ldaddal (large-system extensions)
     15 casal   (large-system extensions)
     31 swpa    (large-system extensions)

I see no __aarch64_ functions to outline atomics and no load and store for pre-v8.1 ARM because I disabled the default -mno-outline-atomics runtime detection. That’s what I wanted to be optimized for this processor. I’ll compile differently later to check the difference.


/usr/local/pgsql/bin/psql --version
/usr/local/pgsql/bin/initdb -D /var/tmp/pgdata
/usr/local/pgsql/bin/pg_ctl -D /var/tmp/pgdata start

This creates a database and starts the instance

[opc@arm-lon contrib]$ free -h

              total        used        free      shared  buff/cache   available
Mem:            22G        1.0G        4.9G         50M         16G         18G
Swap:          8.0G          0B        8.0G

sed -ie "/huge_pages/s/^.*=.*/huge_pages= true/" /var/tmp/pgdata/postgresql.conf
sed -ie "/shared_buffers/s/^.*=.*/shared_buffers= 4800MB/" /var/tmp/pgdata/postgresql.conf
/usr/local/pgsql/bin/pg_ctl -D /var/tmp/pgdata -l postgres.log stop

I have 24 GB RAM here (which is what Oracle gives you for free – this is quite awesome, isn’t it?)
I’ll use enough shared buffers to fit my working set (4 schemas of 1GB) because I want to measure shared buffer cache access (this is where atimics are used).21

[opc@arm-lon contrib]$ $ grep Hugepagesize /proc/meminfo

Hugepagesize:     524288 kB

The huge pages are 512MB here. This processor is build for large systems and that’s perfect for a database. Of course, I’ll define enough huge pages for the shared buffers.


awk '/Hugepagesize.*kB/{print ( 1 + 1024 * MB / $2 ) }' MB=8192 /proc/meminfo | tee /dev/stderr | sudo bash -c  "cat > /proc/sys/vm/nr_hugepages"

/usr/local/pgsql/bin/pg_ctl -D /var/tmp/pgdata -l postgres.log restart

[opc@arm-lon ~]$ free -h

              total        used        free      shared  buff/cache   available
Mem:            22G        9.1G        4.0G         24M        9.4G         10G
Swap:          8.0G        112M        7.9G

Here we are. 8GB for the postgresql shared buffers. 10GB available for the filesystem cache. You need this double buffering because PostgreSQL has no pre-fetching and write ordering optimizations, but still need its own cache to avoid a system call per page access.

I’ll run PGIO to look at the CPU performance, as I did on AWS Graviton2. Because it measures exactly what I want to measure: access to the shared buffers. Without spending all time on client-server calls (see https://amitdkhan-pg.blogspot.com/2020/05/postgresql-on-arm.html for other ways to do it with a pgbench-like workload).


git clone https://github.com/therealkevinc/pgio
tar -zxf pgio/pgio*tar.gz
cat > pgio/pgio.conf <<CAT
 UPDATE_PCT=0
 RUN_TIME=$(( 60 * 60))
 NUM_SCHEMAS=4
 NUM_THREADS=1
 WORK_UNIT=255
 UPDATE_WORK_UNIT=8
 SCALE=1024M
 DBNAME=pgio
 CONNECT_STRING="pgio"
 CREATE_BASE_TABLE=TRUE
CAT
/usr/local/pgsql/bin/psql postgres <<<'create database pgio;'
time ( PATH=/usr/local/pgsql/bin:$PATH ; cd pgio && bash setup.sh )

This setup a database for a read-only workload from 4 sessions reading in their own schema. If you are interesting by atomics improvement for AWL lightweight locks, you can set the PCT_UPDATE. But I like to show something that is often overlooked: a read-only workload on the database is still a read-write workload on the shared memory. And there is no connection, parsing, vacuuming, analyze… here. I use Kevin Closson PGIO, the “SLOB method” to run only what I want to measure: pin and read shared buffers.


( PATH=/usr/local/pgsql/bin:$PATH ; cd pgio && bash runit.sh | sed -e "s/^/$n|\t/" )

Date: Sun May 23 16:56:49 GMT 2021
Database connect string: "pgio".
Shared buffers: 4800MB.
Testing 4 schemas with 1 thread(s) accessing 1024M (131072 blocks) of each schema.
Running iostat, vmstat and mpstat on current host--in background.
Launching sessions. 4 schema(s) will be accessed by 1 thread(s) each.
pg_stat_database stats:
          datname| blks_hit| blks_read|tup_returned|tup_fetched|tup_updated
BEFORE:  pgio    | 10070507688 |   6920153 |   9870736490 |  9862382424 |      988185
AFTER:   pgio    | 10936832180 |   7446586 |  10723766109 | 10715208260 |      988185
DBNAME:  pgio. 4 schemas, 1 threads(each). Run time: 600 seconds. RIOPS >877< CACHE_HITS/s >1443874<

objdump -d /usr/local/pgsql/bin/postgres | awk '/\t(ldxr|ldaxr|stxr|stlxr)/{print $3"\t(load and store exclusives)"}/\t(cas|casp|swp|ldadd|stadd|ldclr|stclr|ldeor|steor|ldset|stset|ldsmax|stsmax|ldsmin|stsmin|ldumax|stumax|ldumin|stumin)/{print $3"\t(large-system extensions)"}' | sort | uniq -c | sort -n

      1 ldclral (large-system extensions)
      3 ldsetal (large-system extensions)
     12 ldaddal (large-system extensions)
     15 casal   (large-system extensions)
     31 swpa    (large-system extensions)

1443874 page read per second from the 4 sessions. This is 360969 LIOPS/thread, with CFLAGS=”-march=armv8.2-a” which calls the CASAL atomic in order to acquire the share mode LWLock to lookup for the block in the shared buffer partition.

I mentioned that I compiled for ARM v8.2 only rather than relying on the defaults that try to build an ARM v8 compatible binary. Let’s do the same with CFLAGS=”-march=armv8-a -moutline-atomics”.


( PATH=/usr/local/pgsql/bin:$PATH ; cd pgio && bash runit.sh | sed -e "s/^/$n|\t/" )

Date: Sun May 23 16:31:43 GMT 2021
Database connect string: "pgio".
Shared buffers: 4800MB.
Testing 4 schemas with 1 thread(s) accessing 1024M (131072 blocks) of each schema.
Running iostat, vmstat and mpstat on current host--in background.
Launching sessions. 4 schema(s) will be accessed by 1 thread(s) each.
pg_stat_database stats:
          datname| blks_hit| blks_read|tup_returned|tup_fetched|tup_updated
BEFORE:  pgio    | 9196454698 |   6919962 |   9010603761 |  9002477779 |      988150
AFTER:   pgio    | 10070502579 |   6919975 |   9870712088 |  9862380498 |      988183
DBNAME:  pgio. 4 schemas, 1 threads(each). Run time: 600 seconds. RIOPS >0< CACHE_HITS/s >1456746<

objdump -d /usr/local/pgsql/bin/postgres | awk '/\t(ldxr|ldaxr|stxr|stlxr)/{print $3"\t(load and store exclusives)"}/\t(cas|casp|swp|ldadd|stadd|ldclr|stclr|ldeor|steor|ldset|stset|ldsmax|stsmax|ldsmin|stsmin|ldumax|stumax|ldumin|stumin)/{print $3"\t(large-system extensions)"}' | sort | uniq -c | sort -n

      1 ldclral (large-system extensions)
      1 ldsetal (large-system extensions)
      1 stxr    (load and store exclusives)
      1 swpa    (large-system extensions)
      2 casal   (large-system extensions)
      2 ldaddal (large-system extensions)
      6 stlxr   (load and store exclusives)
      7 ldaxr   (load and store exclusives)

1456746 page read per second from the 4 sessions. This is 364187 LIOPS/thread, with CFLAGS=”-march=armv8 -moutline-atomics” but with LSE outlined (which is the default in GCC 11). This uses a a runtime detection to use CASAL is LSE are available or LDAXR/STXX if not. I would expect a little overhead for this detection but my test shows that I was able to read nearly 1% more buffers with there. Surprising, isn’t it?

Here are some screenshots fro perf top when those were running:

You can see that I’ve also run with the ARM v8 without LSE:


( PATH=/usr/local/pgsql/bin:$PATH ; cd pgio && bash runit.sh | sed -e "s/^/$n|\t/" )

Date: Sun May 23 17:17:35 GMT 2021
Database connect string: "pgio".
Shared buffers: 4800MB.
Testing 4 schemas with 1 thread(s) accessing 1024M (131072 blocks) of each schema.
Running iostat, vmstat and mpstat on current host--in background.
Launching sessions. 4 schema(s) will be accessed by 1 thread(s) each.
pg_stat_database stats:
          datname| blks_hit| blks_read|tup_returned|tup_fetched|tup_updated
BEFORE:  pgio    | 10936836909 |   7446768 |  10723783569 | 10715210130 |      988187
AFTER:   pgio    | 11785746100 |   7973262 |  11559676977 | 11550904052 |      988213
DBNAME:  pgio. 4 schemas, 1 threads(each). Run time: 600 seconds. RIOPS >877< CACHE_HITS/s >1414848<

objdump -d /usr/local/pgsql/bin/postgres | awk '/\t(ldxr|ldaxr|stxr|stlxr)/{print $3"\t(load and store exclusives)"}/\t(cas|casp|swp|ldadd|stadd|ldclr|stclr|ldeor|steor|ldset|stset|ldsmax|stsmax|ldsmin|stsmin|ldumax|stumax|ldumin|stumin)/{print $3"\t(large-system extensions)"}' | sort | uniq -c | sort -n
     15 ldaxr   (load and store exclusives)
     31 stlxr   (load and store exclusives)
     31 stxr    (load and store exclusives)
     47 ldxr    (load and store exclusives)

1414848 page read per second from the 4 sessions. This is 353712 LIOPS/thread, with CFLAGS=”-march=armv8 -mno-outline-atomics”. I’ve run 2% less buffer reads when the LWLock is implemented as load store loop.

I show all these to show how you can check if your binaries use all the processor features, and how to measure and understand the difference. But the numbers have no meaning outside of this specific context. Numbers from benchmarks are useless outside of marketing slides. Here, my context is only 4 sessions (because I want my demos to run on free tier) and PostgreSQL has the shared buffer partitioned to 16 latches. My data is equally distributed, there are no page misses, no session reading the same blocks, no update… I have set, on purpose, the situation with the less contention to show that the lightweight lock optimizations are still important here. And you see where the contention will be in case of CPU starvation: non-LSE loop on load store will be a problem and LSE atomics will help, outlined for runtime detection, or not.

Update 27-MAY-2021

I mentioned above that I installed the latest GCC (version 11) and PostgreSQL (14 beta) for this test, but I think it is important to give the correct way to install the latest stable version (here 13.3) with the latest GCC shipped with OL7 (the devtoolset-10 software collection):


VER=13.3 CFLAGS="-mcpu=neoverse-n1" scl enable devtoolset-10 bash <<'SCL'
sudo yum install -y git gcc readline-devel zlib-devel bison bison-devel flex && \
curl https://ftp.postgresql.org/pub/source/v$VER/postgresql-$VER.tar.gz | tar -zxf - && \
cd postgresql-$VER &&  ./configure --enable-debug && make && sudo make install
SCL
/usr/local/pgsql/bin/initdb -D /var/tmp/test_pgdata
/usr/local/pgsql/bin/pg_ctl -D /var/tmp/test_pgdata start -l /dev/stdout | grep GCC
/usr/local/pgsql/bin/pg_ctl -D /var/tmp/test_pgdata stop  

waiting for server to start….2021-05-27 19:20:43.024 GMT [30480] LOG: starting PostgreSQL 14beta1 on aarch64-unknown-linux-gnu, compiled by gcc (GCC) 10.2.1 20200804 (Red Hat 10.2.1-2.0.2), 64-bit

Cet article PostgreSQL on Oracle free tier ARM est apparu en premier sur Blog dbi services.

Patroni on FreeBSD 13?

$
0
0

You can already find several blog posts around PostgreSQL protected by Patroni on our blog. They all have in common, that the setup is on a flavor of Linux. As I got impressed by FreeBSD and played with it during the last weeks (some blog posts here), I wanted to know if Patroni runs well on FreeBSD. Well, here are the results …

As usual you need at least three nodes for automatic fail-over to work, in my case these are:

postgres@freebsd-patroni1:/home/postgres/ [pg133] grep patroni /etc/hosts
192.168.22.71          patroni1-freebsd  patroni1-freebsd.it.dbi-services.com
192.168.22.72          patroni2-freebsd  patroni2-freebsd.it.dbi-services.com
192.168.22.73          patroni3-freebsd  patroni3-freebsd.it.dbi-services.com

On all of these nodes PostgreSQL 13.3 is already installed in the same location:

postgres@freebsd-patroni2:/home/postgres/ [pg133] ls /u01/app/postgres/product/13/db_3/
bin     include lib     share

Except for this sudo configuration:

postgres@freebsd-patroni1:/home/postgres/ [pg133] sudo grep postgres /usr/local/etc/sudoers 
postgres ALL=(ALL) NOPASSWD: ALL

… nothing happened on these systems.

The first bit we usually install and configure is etcd but there are no pre-build binaries to download from the GitHub page. So either we need to compile it from source, or we use the packages provided by FreeBSD:

postgres@freebsd-patroni1:/home/postgres/ [pg133] sudo pkg search etcd
coreos-etcd-2.3.8_4            Highly-available key value store and service discovery
coreos-etcd31-3.1.20_2         Highly-available key value store and service discovery
coreos-etcd32-3.2.32           Highly-available key value store and service discovery
coreos-etcd33-3.3.23           Highly-available key value store and service discovery
coreos-etcd34-3.4.14           Highly-available key value store and service discovery
etcd-1.0.1_3                   Enhanced Tiny CD, a simple ncurses-based CD player
netcdf-4.7.4                   C library for machine-independent, array-oriented data access
netcdf-cxx-4.3.1_1             C++ library for machine-independent, array-oriented data access
netcdf-fortran-4.5.3           Fortran library for machine-independent, array-oriented data access
octave-forge-netcdf-1.0.14_2   Octave-forge package netcdf
p5-NetCDF-1.2.4_7              Perl5 module to read and write netCDF files
pnetcdf-1.8.1_10               Library providing high-performance I/O
py37-netCDF4-1.3.1_8           Python Interface to the NetCDF Library (versions 3 and 4)
py37-netcdf-flattener-1.2.0    Flatten NetCDF files while preserving references
rubygem-ruby-netcdf-0.7.2_3    Ruby interface to the NetCDF scientific IO library
setcdboot-1.0                  Mark a file bootable within a DEC Alpha ISO-966

Let’s use the latest release and install the FreeBSD package (on all nodes):

postgres@freebsd-patroni1:/home/postgres/ [pg133] sudo pkg install coreos-etcd34-3.4.14
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        coreos-etcd34: 3.4.14

Number of packages to be installed: 1

The process will require 35 MiB more space.
9 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching coreos-etcd34-3.4.14.txz: 100%    9 MiB   2.3MB/s    00:04    
Checking integrity... done (0 conflicting)
[1/1] Installing coreos-etcd34-3.4.14...
[1/1] Extracting coreos-etcd34-3.4.14: 100%

Once etcd is installed it needs a configuration file on each host, here they are:

postgres@freebsd-patroni1:/home/postgres/ [pg133] cat /usr/local/etc/etcd.conf 
name: patroni1
data-dir: /u02/pgdata/etcd
initial-advertise-peer-urls: http://192.168.22.71:2380
listen-peer-urls: http://192.168.22.71:2380
listen-client-urls: http://192.168.22.71:2379,http://localhost:2379
advertise-client-urls: http://192.168.22.71:2379
initial-cluster: patroni1=http://192.168.22.71:2380,patroni2=http://192.168.22.72:2380,patroni3=http://192.168.22.73:2380
enable-v2: true

postgres@freebsd-patroni1:/home/postgres/ [pg133] ssh patroni2-freebsd cat /usr/local/etc/etcd.conf
name: patroni2
data-dir: /u02/pgdata/etcd
initial-advertise-peer-urls: http://192.168.22.72:2380
listen-peer-urls: http://192.168.22.72:2380
listen-client-urls: http://192.168.22.72:2379,http://localhost:2379
advertise-client-urls: http://192.168.22.72:2379
initial-cluster: patroni1=http://192.168.22.71:2380,patroni2=http://192.168.22.72:2380,patroni3=http://192.168.22.73:2380
enable-v2: true

postgres@freebsd-patroni1:/home/postgres/ [pg133] ssh patroni3-freebsd cat /usr/local/etc/etcd.conf
name: patroni3
data-dir: /u02/pgdata/etcd
initial-advertise-peer-urls: http://192.168.22.73:2380
listen-peer-urls: http://192.168.22.73:2380
listen-client-urls: http://192.168.22.73:2379,http://localhost:2379
advertise-client-urls: http://192.168.22.73:2379
initial-cluster: patroni1=http://192.168.22.71:2380,patroni2=http://192.168.22.72:2380,patroni3=http://192.168.22.73:2380
enable-v2: true

Testing the configuration can easily be done by starting etcd on all nodes:

postgres@freebsd-patroni2:/home/postgres/ [pg133] etcd --config-file /usr/local/etc/etcd.conf 

Once up and running on all three nodes, verify that all is fine with etcd:

postgres@freebsd-patroni1:/home/postgres/ [pg133] etcdctl endpoint status --endpoints=192.168.22.71:2379,192.168.22.72:2379,192.168.22.73:2379 -w table
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.22.71:2379 | 52009b42a701a943 |  3.4.14 |   25 kB |      true |      false |        13 |         10 |                 10 |        |
| 192.168.22.72:2379 | 26d9ce490122dc7a |  3.4.14 |   25 kB |     false |      false |        13 |         10 |                 10 |        |
| 192.168.22.73:2379 | 6141508d7bd79cb5 |  3.4.14 |   25 kB |     false |      false |        13 |         10 |                 10 |        |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

The next step is to configure automatic start of etcd when the nodes comes up. This is a bit different in FreeBSD than what you know from Linux. A detailed guide on how to do that can be found here. We’ll be using this simple script (you should of course extend that script to provide a stop command, add logging to a logfile …):

postgres@freebsd-patroni1:/home/postgres/ [pg133] cat /etc/rc.d/etcd 
#!/usr/local/bin/bash

# PROVIDE: ETCD
# REQUIRE: DAEMON FILESYSTEMS NETWORKING
# BEFORE:  LOGIN 
# KEYWORD: shutdown

. /etc/rc.subr

name="etcd"
rcvar=etcd_enable
start_cmd="su - postgres -c '/usr/sbin/daemon -c -f /usr/local/bin/etcd --config-file /usr/local/etc/etcd.conf'"

load_rc_config $name
run_rc_command "$1"

To test if it basically works use the “onestart” argument:

ostgres@freebsd-patroni1:/home/postgres/ [pg133] sudo /etc/rc.d/etcd onestart
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-06-03 17:31:49.344284 I | etcdmain: Loading server configuration from "/usr/local/etc/etcd.conf". Other configuration command line flags and environment variables will be ignored if provided.
2021-06-03 17:31:49.344428 I | etcdmain: etcd Version: 3.4.14
2021-06-03 17:31:49.344461 I | etcdmain: Git SHA: Not provided (use ./build instead of go build)
...

Looks good, let’s add it to rc.conf so etcd will startup automatically:

postgres@freebsd-patroni1:/home/postgres/ [pg133] grep etcd /etc/rc.conf
etcd_enable="YES"

Once you’ve done that on all nodes, reboot and check that all is fine:

postgres@freebsd-patroni1:/home/postgres/ [pg133] sudo init 6
postgres@freebsd-patroni1:/home/postgres/ [pg133] ssh patroni2-freebsd init 6
postgres@freebsd-patroni1:/home/postgres/ [pg133] ssh patroni3-freebsd init 6
# wait for reboot
postgres@freebsd-patroni1:/home/postgres/ [pg133] etcdctl endpoint status --endpoints=192.168.22.71:2379,192.168.22.72:2379,192.168.22.73:2379 -w table
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.22.71:2379 | 52009b42a701a943 |  3.4.14 |   33 kB |     false |      false |       146 |         16 |                 16 |        |
| 192.168.22.72:2379 | 26d9ce490122dc7a |  3.4.14 |   33 kB |      true |      false |       146 |         16 |                 16 |        |
| 192.168.22.73:2379 | 6141508d7bd79cb5 |  3.4.14 |   25 kB |     false |      false |       146 |         16 |                 16 |        |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

So far, so good for etcd. The next step is to install Patroni, and this is not different from an installation on Linux (no surprise here, it is based on Python):

postgres@freebsd-patroni1:/home/postgres/ [pg133] sudo pkg install py37-pip
postgres@freebsd-patroni1:/home/postgres/ [pg133] sudo pip install --upgrade setuptools
postgres@freebsd-patroni1:/home/postgres/ [pg133] pip install --user psycopg2-binary
postgres@freebsd-patroni1:/home/postgres/ [pg133] pip install --user python-etcd
postgres@freebsd-patroni1:/home/postgres/ [pg133] pip install --user wheel
postgres@freebsd-patroni1:/home/postgres/ [pg133] pip install --user patroni[etcd]

The configuration of Patroni is of course not different than on Linux, here is an example for the first host:

postgres@freebsd-patroni1:/home/postgres/ [pg133] cat /usr/local/etc/patroni.yml 
scope: postgres_patroni
name: pg-patroni1

restapi:
  listen: 192.168.22.71:8008
  connect_address: 192.168.22.71:8008

etcd:
    hosts: 192.168.22.71:2379,192.168.22.72:2379,192.168.22.73:2379

log:
  dir: /u02/log/patroni/
  file_num: 14
  file_size: 10000000
  
bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout : 10
    maximum_lag_on_failover: 1048576
    postgresql:
      use_pg_rewind: true
      use_slots: true
      parameters:
        wal_keep_segments: 100

  initdb:
  - encoding: UTF8
  - data-checksums

  pg_hba:
  - host replication all 0.0.0.0/0 md5
  - host all all 0.0.0.0/0 md5
  - hostssl all all 0.0.0.0/0 md5
postgresql:
  listen: "*:5432"
  connect_address: 192.168.22.71:5432
  data_dir: /u02/pgdata/13/PG1
  bin_dir: /u01/app/postgres/product/13/db_3/bin
  authentication:
    replication:
      username: postgres
      password: postgres
    superuser:
      username: postgres
      password: postgres
  parameters:
    logging_collector: 'on'
    log_truncate_on_rotation: 'on'
    log_filename: 'postgresql-%a.log'
    log_rotation_age: '1440'
    log_line_prefix: '%m - %l - %p - %h - %u@%d - %x'
    log_directory: '/u02/pgdata/13/PG1/pg_log'
    log_min_messages: 'WARNING'
    log_autovacuum_min_duration: '60s'
    log_min_error_statement: 'NOTICE'
    log_min_duration_statement: '30s'
    log_checkpoints: 'on'
    log_statement: 'ddl'
    log_lock_waits: 'on'
    log_temp_files: '0'
    log_timezone: 'Europe/Zurich'
    log_connections: off
    log_disconnections: off
    log_duration: off
    checkpoint_completion_target: 0.9
    checkpoint_timeout: '20min'
    client_min_messages: 'WARNING'
    wal_level: 'replica'
    hot_standby_feedback: 'on'
    max_wal_senders: '10'
    max_replication_slots: '10'
    shared_preload_libraries: 'pg_stat_statements'
    autovacuum_vacuum_threshold: 50
    archive_mode: 'on'
    archive_command: '/bin/true'
    cluster_name: 'PG1' 

If all is fine, you should be able to bootstrap the first node:

postgres@freebsd-patroni1:/home/postgres/ [pg133] .local/bin/patroni /usr/local/etc/patroni.yml 
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "C.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are enabled.

creating directory /u02/pgdata/13/PG1 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Europe/Vaduz
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /u01/app/postgres/product/13/db_3/bin/pg_ctl -D /u02/pgdata/13/PG1 -l logfile start

localhost:5432 - no response
2021-06-03 18:39:24.699 CEST - 1 - 1495 -  - @ - 0LOG:  redirecting log output to logging collector process
2021-06-03 18:39:24.699 CEST - 2 - 1495 -  - @ - 0HINT:  Future log output will appear in directory "/u02/pgdata/13/PG1/pg_log".
localhost:5432 - accepting connections
localhost:5432 - accepting connections

Do exactly the same on the remaining to nodes and then create the rc script for Patroni (note the dependency on etcd):

postgres@freebsd-patroni1:/home/postgres/ [pg133] sudo cat /etc/rc.d/patroni
#!/usr/local/bin/bash

# PROVIDE: PATRONI
# REQUIRE: ETCD
# BEFORE:  LOGIN 
# KEYWORD: shutdown

. /etc/rc.subr

name="patroni"
rcvar=patroni_enable
start_cmd="su - postgres -c '/usr/sbin/daemon -c -f /home/postgres/.local/bin/patroni /usr/local/etc/patroni.yml'"

load_rc_config $name
run_rc_command "$1"

Enable autostart in rc.conf:

postgres@freebsd-patroni1:/home/postgres/ [pg133] grep patroni /etc/rc.conf
hostname="freebsd-patroni1.it.dbi-services.com"
patroni_enable="YES"

… and reboot all nodes. The result should be this:

postgres@freebsd-patroni1:/home/postgres/ [pg133] .local/bin/patronictl -c /usr/local/etc/patroni.yml list 
+ Cluster: postgres_patroni (6969608203440481745) +----+-----------+
| Member      | Host          | Role    | State   | TL | Lag in MB |
+-------------+---------------+---------+---------+----+-----------+
| pg-patroni1 | 192.168.22.71 | Leader  | running |  5 |           |
| pg-patroni2 | 192.168.22.72 | Replica | running |  5 |         0 |
| pg-patroni3 | 192.168.22.73 | Replica | running |  5 |         0 |
+-------------+---------------+---------+---------+----+-----------+

Works as expected.

Cet article Patroni on FreeBSD 13? est apparu en premier sur Blog dbi services.

Recurring PostgreSQL Installations using RHEL 8 and Clones

$
0
0

This Blog is a follow up uf one of my older Blogs and my Article at heise.de:

Blog at dbi-services.com
Article at heise.de

For RHEL 8 at its clones like AlmaLinux, CentOS, Oracle Linux and Rocky Linux i have written a shell script to automated recurring setups.
This is used for a cloud project using own virtual machines.

The script automates the steps i have described at my article at heise.de.

The script has an help opportunity, -h tell how to use it:

$ [[root@rocky ~]$ sh pg_setup.sh -h
$ [[root@rocky ~]$ PostgreSQL Installation on AlmaLinux 8 / CentOS 8 / Oracle Linxu 8 / RHEL 8 / Rocky Linux 8
$ [[root@rocky ~]$ 
$ [[root@rocky ~]$ Usage:
$ [[root@rocky ~]$  [OPTION]
$ [[root@rocky ~]$ 
$ [[root@rocky ~]$ Options:
$ [[root@rocky ~]$          -v               Major Release of Postgresql (required)
$ [[root@rocky ~]$          -t               If the System is Primary / Single (p) or Secondary (s) in a Cluster (required)
$ [[root@rocky ~]$          -h               prints this help
$ [[root@rocky ~]$ 

So the script supports any version of PostgreSQL supported by the PostgreSQL Community.

Required is an own volume or folder /pgdata or even per PostgreSQL version /pgdata/Major Releae, so for PostgreSQL 13 /pgdata/13/, if the the folder with the release number is not existing, it will be created.

The code of this script:

$ [[root@rocky ~]$ ############################################################################################
$ [[root@rocky ~]$ # Installing PostgreSQL on AlmaLinux 8 / CentOS 8 / Oracle Linux 8 / RHEL 8 /Rocky Linux 8 #
$ [[root@rocky ~]$ # including opening port 5432                                                              #
$ [[root@rocky ~]$ # By Karsten Lenz dbi-services sa 2020.06.16                                               #
$ [[root@rocky ~]$ ############################################################################################
$ [[root@rocky ~]$
$ [[root@rocky ~]$ #!/bin/bash
$ [[root@rocky ~]$
$ [[root@rocky ~]$ echo "PostgreSQL Installation on AlmaLinux 8 / CentOS 8 / Oracle Linxu 8 / RHEL 8 / Rocky Linux 8"
$ [[root@rocky ~]$ echo ""
$ [[root@rocky ~]$
$ [[root@rocky ~]$ function printHelp() {
$ [[root@rocky ~]$   printf "Usage:\n"
$ [[root@rocky ~]$   printf "${progName} [OPTION]\n\n"
$ [[root@rocky ~]$   printf "Options:\n"
$ [[root@rocky ~]$   printf "\t -v \t\tMajor Release of Postgresql (required)\n"
$ [[root@rocky ~]$   printf "\t -t \t\tIf the System is Primary / Single (p) or Secondary (s) in a Cluster (required)\n"
$ [[root@rocky ~]$   printf "\t -h \t\t\t\tprints this help\n"
$ [[root@rocky ~]$ }
$ [[root@rocky ~]$
$ [[root@rocky ~]$ while getopts v:t:h option 2>/dev/null
$ [[root@rocky ~]$ do
$ [[root@rocky ~]$   case "${option}"
$ [[root@rocky ~]$   in
$ [[root@rocky ~]$   v) VERSION=${OPTARG};;
$ [[root@rocky ~]$   t) TYPE=${OPTARG};;
$ [[root@rocky ~]$   h) printHelp; exit 2;;
$ [[root@rocky ~]$   *) printf "Unsupported option or parameter value missing '$*'\n";
$ [[root@rocky ~]$      printf "Run ${progName} -h to print help\n"; exit 1;;
$ [[root@rocky ~]$   esac
$ [[root@rocky ~]$ done
$ [[root@rocky ~]$
$ [[root@rocky ~]$ echo "Disabling PostgreSQL Packages provided by the Distros"
$ [[root@rocky ~]$ dnf -y module disable postgresql
$ [[root@rocky ~]$ 
$ [[root@rocky ~]$ echo "Adding PostgreSQL Repository"
$ [[root@rocky ~]$ dnf -y install https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm
$ [[root@rocky ~]$ 
$ [[root@rocky ~]$ echo "Installing PostgreSQL Packages including barman, repmgr and pg_stat_kcache"
$ [[root@rocky ~]$ dnf -y install postgresql$VERSION postgresql$VERSION-libs postgresql$VERSION-server postgresql$VERSION-contrib pg_stat_kcache$VERSION repmgr$VERSION barman
$ [[root@rocky ~]$ 
$ [[root@rocky ~]$ # creating empty override.conf for postgresql service
$ [[root@rocky ~]$ mkdir /etc/systemd/system/postgresql-$VERSION.service.d
$ [[root@rocky ~]$
$ [[root@rocky ~]$ # define vriable for tee
$ [[root@rocky ~]$ OVERRIDE_CONF=/etc/systemd/system/postgresql-$VERSION.service.d/override.conf
$ [[root@rocky ~]$ touch $OVERRIDE_CONF
$ [[root@rocky ~]$
$ [[root@rocky ~]$ # adding values to override.conf
$ [[root@rocky ~]$ echo "[Service]" | tee -a $OVERRIDE_CONF
$ [[root@rocky ~]$ echo "Environment=PGDATA=/pgdata/$VERSION/data" | tee -a $OVERRIDE_CONF
$ [[root@rocky ~]$
$ [[root@rocky ~]$
$ [[root@rocky ~]$ # different steps for single/primary or standby
$ [[root@rocky ~]$ if [ $TYPE == 'p' ]
$ [[root@rocky ~]$ then
$ [[root@rocky ~]$         echo "Initialyze PostgreSQL $VERSION"
$ [[root@rocky ~]$         /usr/pgsql-$VERSION/bin/postgresql-$VERSION-setup initdb
$ [[root@rocky ~]$ 
$ [[root@rocky ~]$         echo "Enable PostgreSQL Service"
$ [[root@rocky ~]$         systemctl enable postgresql-$VERSION.service
$ [[root@rocky ~]$
$ [[root@rocky ~]$         echo "Start PostgreSQL $VERSION"
$ [[root@rocky ~]$         systemctl start postgresql-$VERSION.service
$ [[root@rocky ~]$
$ [[root@rocky ~]$ elif [ $TYPE == 's' ]
$ [[root@rocky ~]$ then
$ [[root@rocky ~]$         echo "Initialyze PostgreSQL $VERSION"
$ [[root@rocky ~]$         /usr/pgsql-$VERSION/bin/postgresql-$VERSION-setup initdb
$ [[root@rocky ~]$ 
$ [[root@rocky ~]$         echo "Enable PostgreSQL Service"
$ [[root@rocky ~]$         systemctl enable postgresql-$VERSION.service
$ [[root@rocky ~]$ fi
$ [[root@rocky ~]$ 
$ [[root@rocky ~]$ # open port 5432 for PostgreSQL
$ [[root@rocky ~]$ firewall-cmd --zone=public --permanent --add-port 5432/tcp
$ [[root@rocky ~]$ firewall-cmd --reload

OK, lets install PostgreSQL 13 as single server, according to the help information the call is:

$ [[root@rocky ~]$ sh pg_setup.sh -v 13 -t s
PostgreSQL Installation on AlmaLinux 8 / CentOS 8 / Oracle Linxu 8 / RHEL 8 / Rocky Linux 8

Disabling PostgreSQL Packages provided by the Distros
Last metadata expiration check: 0:12:33 ago on Tue 15 Jun 2021 11:33:01 AM CEST.
Dependencies resolved.
==========================================================================================================================================================================================================================================================================================
 Package                                                              Architecture                                                        Version                                                              Repository                                                            Size
==========================================================================================================================================================================================================================================================================================
Disabling modules:
 postgresql

Transaction Summary
==========================================================================================================================================================================================================================================================================================

Complete!
Adding PostgreSQL Repository
Last metadata expiration check: 0:12:34 ago on Tue 15 Jun 2021 11:33:01 AM CEST.
pgdg-redhat-repo-latest.noarch.rpm                                                                                                                                                                                                                        8.6 kB/s |  12 kB     00:01
Dependencies resolved.
==========================================================================================================================================================================================================================================================================================
 Package                                                                   Architecture                                                    Version                                                            Repository                                                             Size
==========================================================================================================================================================================================================================================================================================
Installing:
 pgdg-redhat-repo                                                          noarch                                                          42.0-17                                                            @commandline                                                           12 k

Transaction Summary
==========================================================================================================================================================================================================================================================================================
Install  1 Package

Total size: 12 k
Installed size: 11 k
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                                                  1/1
  Installing       : pgdg-redhat-repo-42.0-17.noarch                                                                                                                                                                                                                                  1/1
  Verifying        : pgdg-redhat-repo-42.0-17.noarch                                                                                                                                                                                                                                  1/1

Installed:
  pgdg-redhat-repo-42.0-17.noarch

Complete!
Installing PostgreSQL Packages including barman, repmgr and pg_stat_kcache
PostgreSQL common RPMs for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                         103  B/s | 195  B     00:01
PostgreSQL common RPMs for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                         1.6 MB/s | 1.7 kB     00:00
Importing GPG key 0x442DF0F8:
 Userid     : "PostgreSQL RPM Building Project "
 Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
PostgreSQL common RPMs for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                         152 kB/s | 480 kB     00:03
PostgreSQL 13 for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                                  122  B/s | 195  B     00:01
PostgreSQL 13 for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                                  1.6 MB/s | 1.7 kB     00:00
Importing GPG key 0x442DF0F8:
 Userid     : "PostgreSQL RPM Building Project "
 Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
PostgreSQL 13 for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                                  130 kB/s | 359 kB     00:02
PostgreSQL 12 for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                                  136  B/s | 195  B     00:01
PostgreSQL 12 for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                                  1.6 MB/s | 1.7 kB     00:00
Importing GPG key 0x442DF0F8:
 Userid     : "PostgreSQL RPM Building Project "
 Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
PostgreSQL 12 for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                                  166 kB/s | 550 kB     00:03
PostgreSQL 11 for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                                  106  B/s | 195  B     00:01
PostgreSQL 11 for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                                  1.6 MB/s | 1.7 kB     00:00
Importing GPG key 0x442DF0F8:
 Userid     : "PostgreSQL RPM Building Project "
 Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
PostgreSQL 11 for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                                  212 kB/s | 747 kB     00:03
PostgreSQL 10 for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                                  103  B/s | 195  B     00:01
PostgreSQL 10 for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                                  1.6 MB/s | 1.7 kB     00:00
Importing GPG key 0x442DF0F8:
 Userid     : "PostgreSQL RPM Building Project "
 Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
PostgreSQL 10 for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                                  143 kB/s | 482 kB     00:03
PostgreSQL 9.6 for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                                 121  B/s | 195  B     00:01
PostgreSQL 9.6 for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                                 1.6 MB/s | 1.7 kB     00:00
Importing GPG key 0x442DF0F8:
 Userid     : "PostgreSQL RPM Building Project "
 Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
PostgreSQL 9.6 for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                                 174 kB/s | 490 kB     00:02
Dependencies resolved.
==========================================================================================================================================================================================================================================================================================
 Package                                                                Architecture                                           Version                                                                                  Repository                                                   Size
==========================================================================================================================================================================================================================================================================================
Installing:
 barman                                                                 noarch                                                 2.12-1.rhel8                                                                             pgdg-common                                                  41 k
 pg_stat_kcache_13                                                      x86_64                                                 2.2.0-1.rhel8                                                                            pgdg13                                                       46 k
 postgresql13                                                           x86_64                                                 13.3-2PGDG.rhel8                                                                         pgdg13                                                      1.5 M
 postgresql13-contrib                                                   x86_64                                                 13.3-2PGDG.rhel8                                                                         pgdg13                                                      638 k
 postgresql13-libs                                                      x86_64                                                 13.3-2PGDG.rhel8                                                                         pgdg13                                                      413 k
 postgresql13-server                                                    x86_64                                                 13.3-2PGDG.rhel8                                                                         pgdg13                                                      5.5 M
 repmgr_13                                                              x86_64                                                 5.2.1-1.rhel8                                                                            pgdg13                                                      293 k
Installing dependencies:
 perl-Carp                                                              noarch                                                 1.42-396.el8                                                                             baseos                                                       29 k
 perl-Data-Dumper                                                       x86_64                                                 2.167-399.el8                                                                            baseos                                                       57 k
 perl-Digest                                                            noarch                                                 1.17-395.el8                                                                             appstream                                                    26 k
 perl-Digest-MD5                                                        x86_64                                                 2.55-396.el8                                                                             appstream                                                    36 k
 perl-Encode                                                            x86_64                                                 4:2.97-3.el8                                                                             baseos                                                      1.5 M
 perl-Errno                                                             x86_64                                                 1.28-419.el8                                                                             baseos                                                       75 k
 perl-Exporter                                                          noarch                                                 5.72-396.el8                                                                             baseos                                                       33 k
 perl-File-Path                                                         noarch                                                 2.15-2.el8                                                                               baseos                                                       37 k
 perl-File-Temp                                                         noarch                                                 0.230.600-1.el8                                                                          baseos                                                       62 k
 perl-Getopt-Long                                                       noarch                                                 1:2.50-4.el8                                                                             baseos                                                       62 k
 perl-HTTP-Tiny                                                         noarch                                                 0.074-1.el8                                                                              baseos                                                       57 k
 perl-IO                                                                x86_64                                                 1.38-419.el8                                                                             baseos                                                      141 k
 perl-MIME-Base64                                                       x86_64                                                 3.15-396.el8                                                                             baseos                                                       30 k
 perl-Net-SSLeay                                                        x86_64                                                 1.88-1.module+el8.4.0+512+d4f0fc54                                                       appstream                                                   378 k
 perl-PathTools                                                         x86_64                                                 3.74-1.el8                                                                               baseos                                                       89 k
 perl-Pod-Escapes                                                       noarch                                                 1:1.07-395.el8                                                                           baseos                                                       19 k
 perl-Pod-Perldoc                                                       noarch                                                 3.28-396.el8                                                                             baseos                                                       85 k
 perl-Pod-Simple                                                        noarch                                                 1:3.35-395.el8                                                                           baseos                                                      212 k
 perl-Pod-Usage                                                         noarch                                                 4:1.69-395.el8                                                                           baseos                                                       33 k
 perl-Scalar-List-Utils                                                 x86_64                                                 3:1.49-2.el8                                                                             baseos                                                       67 k
 perl-Socket                                                            x86_64                                                 4:2.027-3.el8                                                                            baseos                                                       58 k
 perl-Storable                                                          x86_64                                                 1:3.11-3.el8                                                                             baseos                                                       97 k
 perl-Term-ANSIColor                                                    noarch                                                 4.06-396.el8                                                                             baseos                                                       45 k
 perl-Term-Cap                                                          noarch                                                 1.17-395.el8                                                                             baseos                                                       22 k
 perl-Text-ParseWords                                                   noarch                                                 3.30-395.el8                                                                             baseos                                                       17 k
 perl-Text-Tabs+Wrap                                                    noarch                                                 2013.0523-395.el8                                                                        baseos                                                       23 k
 perl-Time-Local                                                        noarch                                                 1:1.280-1.el8                                                                            baseos                                                       32 k
 perl-URI                                                               noarch                                                 1.73-3.el8                                                                               appstream                                                   115 k
 perl-Unicode-Normalize                                                 x86_64                                                 1.25-396.el8                                                                             baseos                                                       81 k
 perl-constant                                                          noarch                                                 1.33-396.el8                                                                             baseos                                                       24 k
 perl-interpreter                                                       x86_64                                                 4:5.26.3-419.el8                                                                         baseos                                                      6.3 M
 perl-libnet                                                            noarch                                                 3.11-3.el8                                                                               appstream                                                   120 k
 perl-libs                                                              x86_64                                                 4:5.26.3-419.el8                                                                         baseos                                                      1.6 M
 perl-macros                                                            x86_64                                                 4:5.26.3-419.el8                                                                         baseos                                                       71 k
 perl-parent                                                            noarch                                                 1:0.237-1.el8                                                                            baseos                                                       19 k
 perl-podlators                                                         noarch                                                 4.11-1.el8                                                                               baseos                                                      117 k
 perl-threads                                                           x86_64                                                 1:2.21-2.el8                                                                             baseos                                                       60 k
 perl-threads-shared                                                    x86_64                                                 1.58-2.el8                                                                               baseos                                                       47 k
 python3-argcomplete                                                    noarch                                                 1.9.3-6.el8                                                                              appstream                                                    59 k
 python3-argh                                                           noarch                                                 0.26.1-8.el8                                                                             appstream                                                    58 k
 python3-barman                                                         noarch                                                 2.12-1.rhel8                                                                             pgdg-common                                                 358 k
 python3-pip                                                            noarch                                                 9.0.3-19.el8                                                                             appstream                                                    19 k
 python3-psycopg2                                                       x86_64                                                 2.8.6-1.rhel8                                                                            pgdg-common                                                 178 k
 python3-setuptools                                                     noarch                                                 39.2.0-6.el8                                                                             baseos                                                      162 k
 python36                                                               x86_64                                                 3.6.8-2.module+el8.3.0+120+426d8baf                                                      appstream                                                    18 k
Installing weak dependencies:
 perl-IO-Socket-IP                                                      noarch                                                 0.39-5.el8                                                                               appstream                                                    46 k
 perl-IO-Socket-SSL                                                     noarch                                                 2.066-4.module+el8.4.0+512+d4f0fc54                                                      appstream                                                   297 k
 perl-Mozilla-CA                                                        noarch                                                 20160104-7.module+el8.4.0+529+e3b3e624                                                   appstream                                                    14 k
Enabling module streams:
 perl                                                                                                                          5.26
 perl-IO-Socket-SSL                                                                                                            2.066
 perl-libwww-perl                                                                                                              6.34
 python36                                                                                                                      3.6

Transaction Summary
==========================================================================================================================================================================================================================================================================================
Install  55 Packages

Total download size: 21 M
Installed size: 73 M
Downloading Packages:
(1/55): perl-Digest-1.17-395.el8.noarch.rpm                                                                                                                                                                                                                17 kB/s |  26 kB     00:01
(2/55): perl-Digest-MD5-2.55-396.el8.x86_64.rpm                                                                                                                                                                                                            24 kB/s |  36 kB     00:01
(3/55): perl-IO-Socket-IP-0.39-5.el8.noarch.rpm                                                                                                                                                                                                            28 kB/s |  46 kB     00:01
(4/55): perl-Mozilla-CA-20160104-7.module+el8.4.0+529+e3b3e624.noarch.rpm                                                                                                                                                                                 105 kB/s |  14 kB     00:00
(5/55): perl-URI-1.73-3.el8.noarch.rpm                                                                                                                                                                                                                    424 kB/s | 115 kB     00:00
(6/55): perl-libnet-3.11-3.el8.noarch.rpm                                                                                                                                                                                                                 851 kB/s | 120 kB     00:00
(7/55): perl-Net-SSLeay-1.88-1.module+el8.4.0+512+d4f0fc54.x86_64.rpm                                                                                                                                                                                     858 kB/s | 378 kB     00:00
(8/55): perl-IO-Socket-SSL-2.066-4.module+el8.4.0+512+d4f0fc54.noarch.rpm                                                                                                                                                                                 508 kB/s | 297 kB     00:00
(9/55): python3-argcomplete-1.9.3-6.el8.noarch.rpm                                                                                                                                                                                                        425 kB/s |  59 kB     00:00
(10/55): python3-argh-0.26.1-8.el8.noarch.rpm                                                                                                                                                                                                             403 kB/s |  58 kB     00:00
(11/55): python3-pip-9.0.3-19.el8.noarch.rpm                                                                                                                                                                                                              128 kB/s |  19 kB     00:00
(12/55): python36-3.6.8-2.module+el8.3.0+120+426d8baf.x86_64.rpm                                                                                                                                                                                          124 kB/s |  18 kB     00:00
(13/55): perl-Data-Dumper-2.167-399.el8.x86_64.rpm                                                                                                                                                                                                        299 kB/s |  57 kB     00:00
(14/55): perl-Carp-1.42-396.el8.noarch.rpm                                                                                                                                                                                                                144 kB/s |  29 kB     00:00
(15/55): perl-Errno-1.28-419.el8.x86_64.rpm                                                                                                                                                                                                               1.0 MB/s |  75 kB     00:00
(16/55): perl-Exporter-5.72-396.el8.noarch.rpm                                                                                                                                                                                                            449 kB/s |  33 kB     00:00
(17/55): perl-File-Path-2.15-2.el8.noarch.rpm                                                                                                                                                                                                             973 kB/s |  37 kB     00:00
(18/55): perl-File-Temp-0.230.600-1.el8.noarch.rpm                                                                                                                                                                                                        820 kB/s |  62 kB     00:00
(19/55): perl-Getopt-Long-2.50-4.el8.noarch.rpm                                                                                                                                                                                                           1.5 MB/s |  62 kB     00:00
(20/55): perl-HTTP-Tiny-0.074-1.el8.noarch.rpm                                                                                                                                                                                                            1.4 MB/s |  57 kB     00:00
(21/55): perl-IO-1.38-419.el8.x86_64.rpm                                                                                                                                                                                                                  2.0 MB/s | 141 kB     00:00
(22/55): perl-MIME-Base64-3.15-396.el8.x86_64.rpm                                                                                                                                                                                                         671 kB/s |  30 kB     00:00
(23/55): perl-PathTools-3.74-1.el8.x86_64.rpm                                                                                                                                                                                                             2.9 MB/s |  89 kB     00:00
(24/55): perl-Encode-2.97-3.el8.x86_64.rpm                                                                                                                                                                                                                4.3 MB/s | 1.5 MB     00:00
(25/55): perl-Pod-Escapes-1.07-395.el8.noarch.rpm                                                                                                                                                                                                         555 kB/s |  19 kB     00:00
(26/55): perl-Pod-Perldoc-3.28-396.el8.noarch.rpm                                                                                                                                                                                                         2.2 MB/s |  85 kB     00:00
(27/55): perl-Pod-Simple-3.35-395.el8.noarch.rpm                                                                                                                                                                                                          5.4 MB/s | 212 kB     00:00
(28/55): perl-Pod-Usage-1.69-395.el8.noarch.rpm                                                                                                                                                                                                           900 kB/s |  33 kB     00:00
(29/55): perl-Scalar-List-Utils-1.49-2.el8.x86_64.rpm                                                                                                                                                                                                     1.8 MB/s |  67 kB     00:00
(30/55): perl-Socket-2.027-3.el8.x86_64.rpm                                                                                                                                                                                                               1.6 MB/s |  58 kB     00:00
(31/55): perl-Term-ANSIColor-4.06-396.el8.noarch.rpm                                                                                                                                                                                                      1.2 MB/s |  45 kB     00:00
(32/55): perl-Term-Cap-1.17-395.el8.noarch.rpm                                                                                                                                                                                                            615 kB/s |  22 kB     00:00
(33/55): perl-Storable-3.11-3.el8.x86_64.rpm                                                                                                                                                                                                              1.3 MB/s |  97 kB     00:00
(34/55): perl-Text-ParseWords-3.30-395.el8.noarch.rpm                                                                                                                                                                                                     463 kB/s |  17 kB     00:00
(35/55): perl-Time-Local-1.280-1.el8.noarch.rpm                                                                                                                                                                                                           863 kB/s |  32 kB     00:00
(36/55): perl-Text-Tabs+Wrap-2013.0523-395.el8.noarch.rpm                                                                                                                                                                                                 583 kB/s |  23 kB     00:00
(37/55): perl-Unicode-Normalize-1.25-396.el8.x86_64.rpm                                                                                                                                                                                                   2.1 MB/s |  81 kB     00:00
(38/55): perl-constant-1.33-396.el8.noarch.rpm                                                                                                                                                                                                            601 kB/s |  24 kB     00:00
(39/55): perl-macros-5.26.3-419.el8.x86_64.rpm                                                                                                                                                                                                            1.8 MB/s |  71 kB     00:00
(40/55): perl-parent-0.237-1.el8.noarch.rpm                                                                                                                                                                                                               556 kB/s |  19 kB     00:00
(41/55): perl-podlators-4.11-1.el8.noarch.rpm                                                                                                                                                                                                             2.9 MB/s | 117 kB     00:00
(42/55): perl-threads-2.21-2.el8.x86_64.rpm                                                                                                                                                                                                               1.5 MB/s |  60 kB     00:00
(43/55): perl-threads-shared-1.58-2.el8.x86_64.rpm                                                                                                                                                                                                        1.2 MB/s |  47 kB     00:00
(44/55): python3-setuptools-39.2.0-6.el8.noarch.rpm                                                                                                                                                                                                       4.0 MB/s | 162 kB     00:00
(45/55): perl-libs-5.26.3-419.el8.x86_64.rpm                                                                                                                                                                                                              4.1 MB/s | 1.6 MB     00:00
(46/55): barman-2.12-1.rhel8.noarch.rpm                                                                                                                                                                                                                    45 kB/s |  41 kB     00:00
(47/55): python3-psycopg2-2.8.6-1.rhel8.x86_64.rpm                                                                                                                                                                                                        426 kB/s | 178 kB     00:00
(48/55): python3-barman-2.12-1.rhel8.noarch.rpm                                                                                                                                                                                                           285 kB/s | 358 kB     00:01
(49/55): pg_stat_kcache_13-2.2.0-1.rhel8.x86_64.rpm                                                                                                                                                                                                       314 kB/s |  46 kB     00:00
(50/55): perl-interpreter-5.26.3-419.el8.x86_64.rpm                                                                                                                                                                                                       3.0 MB/s | 6.3 MB     00:02
(51/55): postgresql13-contrib-13.3-2PGDG.rhel8.x86_64.rpm                                                                                                                                                                                                 1.5 MB/s | 638 kB     00:00
(52/55): postgresql13-13.3-2PGDG.rhel8.x86_64.rpm                                                                                                                                                                                                         2.6 MB/s | 1.5 MB     00:00
(53/55): repmgr_13-5.2.1-1.rhel8.x86_64.rpm                                                                                                                                                                                                               2.0 MB/s | 293 kB     00:00
(54/55): postgresql13-server-13.3-2PGDG.rhel8.x86_64.rpm                                                                                                                                                                                                  9.1 MB/s | 5.5 MB     00:00
(55/55): postgresql13-libs-13.3-2PGDG.rhel8.x86_64.rpm                                                                                                                                                                                                    264 kB/s | 413 kB     00:01
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                                                     3.0 MB/s |  21 MB     00:06
warning: /var/cache/dnf/pgdg-common-38b6c5045045f841/packages/barman-2.12-1.rhel8.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID 442df0f8: NOKEY
PostgreSQL common RPMs for RHEL/CentOS 8 - x86_64                                                                                                                                                                                                         1.6 MB/s | 1.7 kB     00:00
Importing GPG key 0x442DF0F8:
 Userid     : "PostgreSQL RPM Building Project "
 Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                                                  1/1
  Installing       : postgresql13-libs-13.3-2PGDG.rhel8.x86_64                                                                                                                                                                                                                       1/55
  Running scriptlet: postgresql13-libs-13.3-2PGDG.rhel8.x86_64                                                                                                                                                                                                                       1/55
  Installing       : postgresql13-13.3-2PGDG.rhel8.x86_64                                                                                                                                                                                                                            2/55
  Running scriptlet: postgresql13-13.3-2PGDG.rhel8.x86_64                                                                                                                                                                                                                            2/55
  Running scriptlet: postgresql13-server-13.3-2PGDG.rhel8.x86_64                                                                                                                                                                                                                     3/55
  Installing       : postgresql13-server-13.3-2PGDG.rhel8.x86_64                                                                                                                                                                                                                     3/55
  Running scriptlet: postgresql13-server-13.3-2PGDG.rhel8.x86_64                                                                                                                                                                                                                     3/55
  Installing       : python3-setuptools-39.2.0-6.el8.noarch                                                                                                                                                                                                                          4/55
  Installing       : python36-3.6.8-2.module+el8.3.0+120+426d8baf.x86_64                                                                                                                                                                                                             5/55
  Running scriptlet: python36-3.6.8-2.module+el8.3.0+120+426d8baf.x86_64                                                                                                                                                                                                             5/55
  Installing       : python3-pip-9.0.3-19.el8.noarch                                                                                                                                                                                                                                 6/55
  Installing       : python3-psycopg2-2.8.6-1.rhel8.x86_64                                                                                                                                                                                                                           7/55
  Installing       : perl-Digest-1.17-395.el8.noarch                                                                                                                                                                                                                                 8/55
  Installing       : perl-Digest-MD5-2.55-396.el8.x86_64                                                                                                                                                                                                                             9/55
  Installing       : perl-Data-Dumper-2.167-399.el8.x86_64                                                                                                                                                                                                                          10/55
  Installing       : perl-libnet-3.11-3.el8.noarch                                                                                                                                                                                                                                  11/55
  Installing       : perl-Net-SSLeay-1.88-1.module+el8.4.0+512+d4f0fc54.x86_64                                                                                                                                                                                                      12/55
  Installing       : perl-URI-1.73-3.el8.noarch                                                                                                                                                                                                                                     13/55
  Installing       : perl-Pod-Escapes-1:1.07-395.el8.noarch                                                                                                                                                                                                                         14/55
  Installing       : perl-Mozilla-CA-20160104-7.module+el8.4.0+529+e3b3e624.noarch                                                                                                                                                                                                  15/55
  Installing       : perl-IO-Socket-IP-0.39-5.el8.noarch                                                                                                                                                                                                                            16/55
  Installing       : perl-Time-Local-1:1.280-1.el8.noarch                                                                                                                                                                                                                           17/55
  Installing       : perl-IO-Socket-SSL-2.066-4.module+el8.4.0+512+d4f0fc54.noarch                                                                                                                                                                                                  18/55
  Installing       : perl-Term-ANSIColor-4.06-396.el8.noarch                                                                                                                                                                                                                        19/55
  Installing       : perl-Term-Cap-1.17-395.el8.noarch                                                                                                                                                                                                                              20/55
  Installing       : perl-File-Temp-0.230.600-1.el8.noarch                                                                                                                                                                                                                          21/55
  Installing       : perl-Pod-Simple-1:3.35-395.el8.noarch                                                                                                                                                                                                                          22/55
  Installing       : perl-HTTP-Tiny-0.074-1.el8.noarch                                                                                                                                                                                                                              23/55
  Installing       : perl-podlators-4.11-1.el8.noarch                                                                                                                                                                                                                               24/55
  Installing       : perl-Pod-Perldoc-3.28-396.el8.noarch                                                                                                                                                                                                                           25/55
  Installing       : perl-Text-ParseWords-3.30-395.el8.noarch                                                                                                                                                                                                                       26/55
  Installing       : perl-Pod-Usage-4:1.69-395.el8.noarch                                                                                                                                                                                                                           27/55
  Installing       : perl-MIME-Base64-3.15-396.el8.x86_64                                                                                                                                                                                                                           28/55
  Installing       : perl-Storable-1:3.11-3.el8.x86_64                                                                                                                                                                                                                              29/55
  Installing       : perl-Getopt-Long-1:2.50-4.el8.noarch                                                                                                                                                                                                                           30/55
  Installing       : perl-Errno-1.28-419.el8.x86_64                                                                                                                                                                                                                                 31/55
  Installing       : perl-Socket-4:2.027-3.el8.x86_64                                                                                                                                                                                                                               32/55
  Installing       : perl-Encode-4:2.97-3.el8.x86_64                                                                                                                                                                                                                                33/55
  Installing       : perl-Carp-1.42-396.el8.noarch                                                                                                                                                                                                                                  34/55
  Installing       : perl-Exporter-5.72-396.el8.noarch                                                                                                                                                                                                                              35/55
  Installing       : perl-libs-4:5.26.3-419.el8.x86_64                                                                                                                                                                                                                              36/55
  Installing       : perl-Scalar-List-Utils-3:1.49-2.el8.x86_64                                                                                                                                                                                                                     37/55
  Installing       : perl-parent-1:0.237-1.el8.noarch                                                                                                                                                                                                                               38/55
  Installing       : perl-macros-4:5.26.3-419.el8.x86_64                                                                                                                                                                                                                            39/55
  Installing       : perl-Text-Tabs+Wrap-2013.0523-395.el8.noarch                                                                                                                                                                                                                   40/55
  Installing       : perl-Unicode-Normalize-1.25-396.el8.x86_64                                                                                                                                                                                                                     41/55
  Installing       : perl-File-Path-2.15-2.el8.noarch                                                                                                                                                                                                                               42/55
  Installing       : perl-IO-1.38-419.el8.x86_64                                                                                                                                                                                                                                    43/55
  Installing       : perl-PathTools-3.74-1.el8.x86_64                                                                                                                                                                                                                               44/55
  Installing       : perl-constant-1.33-396.el8.noarch                                                                                                                                                                                                                              45/55
  Installing       : perl-threads-1:2.21-2.el8.x86_64                                                                                                                                                                                                                               46/55
  Installing       : perl-threads-shared-1.58-2.el8.x86_64                                                                                                                                                                                                                          47/55
  Installing       : perl-interpreter-4:5.26.3-419.el8.x86_64                                                                                                                                                                                                                       48/55
  Installing       : python3-argh-0.26.1-8.el8.noarch                                                                                                                                                                                                                               49/55
  Installing       : python3-argcomplete-1.9.3-6.el8.noarch                                                                                                                                                                                                                         50/55
  Installing       : python3-barman-2.12-1.rhel8.noarch                                                                                                                                                                                                                             51/55
  Running scriptlet: barman-2.12-1.rhel8.noarch                                                                                                                                                                                                                                     52/55
  Installing       : barman-2.12-1.rhel8.noarch                                                                                                                                                                                                                                     52/55
  Installing       : postgresql13-contrib-13.3-2PGDG.rhel8.x86_64                                                                                                                                                                                                                   53/55
  Installing       : pg_stat_kcache_13-2.2.0-1.rhel8.x86_64                                                                                                                                                                                                                         54/55
  Running scriptlet: pg_stat_kcache_13-2.2.0-1.rhel8.x86_64                                                                                                                                                                                                                         54/55
  Running scriptlet: repmgr_13-5.2.1-1.rhel8.x86_64                                                                                                                                                                                                                                 55/55
  Installing       : repmgr_13-5.2.1-1.rhel8.x86_64                                                                                                                                                                                                                                 55/55
  Running scriptlet: repmgr_13-5.2.1-1.rhel8.x86_64                                                                                                                                                                                                                                 55/55
  Verifying        : perl-Digest-1.17-395.el8.noarch                                                                                                                                                                                                                                 1/55
  Verifying        : perl-Digest-MD5-2.55-396.el8.x86_64                                                                                                                                                                                                                             2/55
  Verifying        : perl-IO-Socket-IP-0.39-5.el8.noarch                                                                                                                                                                                                                             3/55
  Verifying        : perl-IO-Socket-SSL-2.066-4.module+el8.4.0+512+d4f0fc54.noarch                                                                                                                                                                                                   4/55
  Verifying        : perl-Mozilla-CA-20160104-7.module+el8.4.0+529+e3b3e624.noarch                                                                                                                                                                                                   5/55
  Verifying        : perl-Net-SSLeay-1.88-1.module+el8.4.0+512+d4f0fc54.x86_64                                                                                                                                                                                                       6/55
  Verifying        : perl-URI-1.73-3.el8.noarch                                                                                                                                                                                                                                      7/55
  Verifying        : perl-libnet-3.11-3.el8.noarch                                                                                                                                                                                                                                   8/55
  Verifying        : python3-argcomplete-1.9.3-6.el8.noarch                                                                                                                                                                                                                          9/55
  Verifying        : python3-argh-0.26.1-8.el8.noarch                                                                                                                                                                                                                               10/55
  Verifying        : python3-pip-9.0.3-19.el8.noarch                                                                                                                                                                                                                                11/55
  Verifying        : python36-3.6.8-2.module+el8.3.0+120+426d8baf.x86_64                                                                                                                                                                                                            12/55
  Verifying        : perl-Carp-1.42-396.el8.noarch                                                                                                                                                                                                                                  13/55
  Verifying        : perl-Data-Dumper-2.167-399.el8.x86_64                                                                                                                                                                                                                          14/55
  Verifying        : perl-Encode-4:2.97-3.el8.x86_64                                                                                                                                                                                                                                15/55
  Verifying        : perl-Errno-1.28-419.el8.x86_64                                                                                                                                                                                                                                 16/55
  Verifying        : perl-Exporter-5.72-396.el8.noarch                                                                                                                                                                                                                              17/55
  Verifying        : perl-File-Path-2.15-2.el8.noarch                                                                                                                                                                                                                               18/55
  Verifying        : perl-File-Temp-0.230.600-1.el8.noarch                                                                                                                                                                                                                          19/55
  Verifying        : perl-Getopt-Long-1:2.50-4.el8.noarch                                                                                                                                                                                                                           20/55
  Verifying        : perl-HTTP-Tiny-0.074-1.el8.noarch                                                                                                                                                                                                                              21/55
  Verifying        : perl-IO-1.38-419.el8.x86_64                                                                                                                                                                                                                                    22/55
  Verifying        : perl-MIME-Base64-3.15-396.el8.x86_64                                                                                                                                                                                                                           23/55
  Verifying        : perl-PathTools-3.74-1.el8.x86_64                                                                                                                                                                                                                               24/55
  Verifying        : perl-Pod-Escapes-1:1.07-395.el8.noarch                                                                                                                                                                                                                         25/55
  Verifying        : perl-Pod-Perldoc-3.28-396.el8.noarch                                                                                                                                                                                                                           26/55
  Verifying        : perl-Pod-Simple-1:3.35-395.el8.noarch                                                                                                                                                                                                                          27/55
  Verifying        : perl-Pod-Usage-4:1.69-395.el8.noarch                                                                                                                                                                                                                           28/55
  Verifying        : perl-Scalar-List-Utils-3:1.49-2.el8.x86_64                                                                                                                                                                                                                     29/55
  Verifying        : perl-Socket-4:2.027-3.el8.x86_64                                                                                                                                                                                                                               30/55
  Verifying        : perl-Storable-1:3.11-3.el8.x86_64                                                                                                                                                                                                                              31/55
  Verifying        : perl-Term-ANSIColor-4.06-396.el8.noarch                                                                                                                                                                                                                        32/55
  Verifying        : perl-Term-Cap-1.17-395.el8.noarch                                                                                                                                                                                                                              33/55
  Verifying        : perl-Text-ParseWords-3.30-395.el8.noarch                                                                                                                                                                                                                       34/55
  Verifying        : perl-Text-Tabs+Wrap-2013.0523-395.el8.noarch                                                                                                                                                                                                                   35/55
  Verifying        : perl-Time-Local-1:1.280-1.el8.noarch                                                                                                                                                                                                                           36/55
  Verifying        : perl-Unicode-Normalize-1.25-396.el8.x86_64                                                                                                                                                                                                                     37/55
  Verifying        : perl-constant-1.33-396.el8.noarch                                                                                                                                                                                                                              38/55
  Verifying        : perl-interpreter-4:5.26.3-419.el8.x86_64                                                                                                                                                                                                                       39/55
  Verifying        : perl-libs-4:5.26.3-419.el8.x86_64                                                                                                                                                                                                                              40/55
  Verifying        : perl-macros-4:5.26.3-419.el8.x86_64                                                                                                                                                                                                                            41/55
  Verifying        : perl-parent-1:0.237-1.el8.noarch                                                                                                                                                                                                                               42/55
  Verifying        : perl-podlators-4.11-1.el8.noarch                                                                                                                                                                                                                               43/55
  Verifying        : perl-threads-1:2.21-2.el8.x86_64                                                                                                                                                                                                                               44/55
  Verifying        : perl-threads-shared-1.58-2.el8.x86_64                                                                                                                                                                                                                          45/55
  Verifying        : python3-setuptools-39.2.0-6.el8.noarch                                                                                                                                                                                                                         46/55
  Verifying        : barman-2.12-1.rhel8.noarch                                                                                                                                                                                                                                     47/55
  Verifying        : python3-barman-2.12-1.rhel8.noarch                                                                                                                                                                                                                             48/55
  Verifying        : python3-psycopg2-2.8.6-1.rhel8.x86_64                                                                                                                                                                                                                          49/55
  Verifying        : pg_stat_kcache_13-2.2.0-1.rhel8.x86_64                                                                                                                                                                                                                         50/55
  Verifying        : postgresql13-13.3-2PGDG.rhel8.x86_64                                                                                                                                                                                                                           51/55
  Verifying        : postgresql13-contrib-13.3-2PGDG.rhel8.x86_64                                                                                                                                                                                                                   52/55
  Verifying        : postgresql13-libs-13.3-2PGDG.rhel8.x86_64                                                                                                                                                                                                                      53/55
  Verifying        : postgresql13-server-13.3-2PGDG.rhel8.x86_64                                                                                                                                                                                                                    54/55
  Verifying        : repmgr_13-5.2.1-1.rhel8.x86_64                                                                                                                                                                                                                                 55/55

Installed:
  barman-2.12-1.rhel8.noarch                 perl-Carp-1.42-396.el8.noarch                                  perl-Data-Dumper-2.167-399.el8.x86_64                      perl-Digest-1.17-395.el8.noarch                      perl-Digest-MD5-2.55-396.el8.x86_64
  perl-Encode-4:2.97-3.el8.x86_64            perl-Errno-1.28-419.el8.x86_64                                 perl-Exporter-5.72-396.el8.noarch                          perl-File-Path-2.15-2.el8.noarch                     perl-File-Temp-0.230.600-1.el8.noarch
  perl-Getopt-Long-1:2.50-4.el8.noarch       perl-HTTP-Tiny-0.074-1.el8.noarch                              perl-IO-1.38-419.el8.x86_64                                perl-IO-Socket-IP-0.39-5.el8.noarch                  perl-IO-Socket-SSL-2.066-4.module+el8.4.0+512+d4f0fc54.noarch
  perl-MIME-Base64-3.15-396.el8.x86_64       perl-Mozilla-CA-20160104-7.module+el8.4.0+529+e3b3e624.noarch  perl-Net-SSLeay-1.88-1.module+el8.4.0+512+d4f0fc54.x86_64  perl-PathTools-3.74-1.el8.x86_64                     perl-Pod-Escapes-1:1.07-395.el8.noarch
  perl-Pod-Perldoc-3.28-396.el8.noarch       perl-Pod-Simple-1:3.35-395.el8.noarch                          perl-Pod-Usage-4:1.69-395.el8.noarch                       perl-Scalar-List-Utils-3:1.49-2.el8.x86_64           perl-Socket-4:2.027-3.el8.x86_64
  perl-Storable-1:3.11-3.el8.x86_64          perl-Term-ANSIColor-4.06-396.el8.noarch                        perl-Term-Cap-1.17-395.el8.noarch                          perl-Text-ParseWords-3.30-395.el8.noarch             perl-Text-Tabs+Wrap-2013.0523-395.el8.noarch
  perl-Time-Local-1:1.280-1.el8.noarch       perl-URI-1.73-3.el8.noarch                                     perl-Unicode-Normalize-1.25-396.el8.x86_64                 perl-constant-1.33-396.el8.noarch                    perl-interpreter-4:5.26.3-419.el8.x86_64
  perl-libnet-3.11-3.el8.noarch              perl-libs-4:5.26.3-419.el8.x86_64                              perl-macros-4:5.26.3-419.el8.x86_64                        perl-parent-1:0.237-1.el8.noarch                     perl-podlators-4.11-1.el8.noarch
  perl-threads-1:2.21-2.el8.x86_64           perl-threads-shared-1.58-2.el8.x86_64                          pg_stat_kcache_13-2.2.0-1.rhel8.x86_64                     postgresql13-13.3-2PGDG.rhel8.x86_64                 postgresql13-contrib-13.3-2PGDG.rhel8.x86_64
  postgresql13-libs-13.3-2PGDG.rhel8.x86_64  postgresql13-server-13.3-2PGDG.rhel8.x86_64                    python3-argcomplete-1.9.3-6.el8.noarch                     python3-argh-0.26.1-8.el8.noarch                     python3-barman-2.12-1.rhel8.noarch
  python3-pip-9.0.3-19.el8.noarch            python3-psycopg2-2.8.6-1.rhel8.x86_64                          python3-setuptools-39.2.0-6.el8.noarch                     python36-3.6.8-2.module+el8.3.0+120+426d8baf.x86_64  repmgr_13-5.2.1-1.rhel8.x86_64

Complete!
[Service]
Environment=PGDATA=/pgdata/13/data
Initialyze PostgreSQL 13
Initializing database ... OK

Enable PostgreSQL Service
Created symlink /etc/systemd/system/multi-user.target.wants/postgresql-13.service → /usr/lib/systemd/system/postgresql-13.service.
success
success
$ [[root@rocky ~]$

Installation is done, service is enabled to start at boot.
Lets start PostgreSQL 13 now:

$ [[root@rocky ~]$ systemctl start postgresql-13.service

Lets check the status:

$ [[root@rocky ~]$ systemctl status postgresql-13.service
$ [[root@rocky ~]$ ● postgresql-13.service - PostgreSQL 13 database server
$ [[root@rocky ~]$    Loaded: loaded (/usr/lib/systemd/system/postgresql-13.service; enabled; vendor preset: disabled)
$ [[root@rocky ~]$    Drop-In: /etc/systemd/system/postgresql-13.service.d
$ [[root@rocky ~]$             └─override.conf
$ [[root@rocky ~]$    Active: active (running) since Tue 2021-06-15 11:53:46 CEST; 7s ago
$ [[root@rocky ~]$    Docs: https://www.postgresql.org/docs/13/static/
$ [[root@rocky ~]$    Process: 4443 ExecStartPre=/usr/pgsql-13/bin/postgresql-13-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)
$ [[root@rocky ~]$    Main PID: 4449 (postmaster)
$ [[root@rocky ~]$    Tasks: 8 (limit: 50551)
$ [[root@rocky ~]$    Memory: 16.8M
$ [[root@rocky ~]$    CGroup: /system.slice/postgresql-13.service
$ [[root@rocky ~]$            ├─4449 /usr/pgsql-13/bin/postmaster -D /pgdata/13/data
$ [[root@rocky ~]$            ├─4450 postgres: logger
$ [[root@rocky ~]$            ├─4452 postgres: checkpointer
$ [[root@rocky ~]$            ├─4453 postgres: background writer
$ [[root@rocky ~]$            ├─4454 postgres: walwriter
$ [[root@rocky ~]$            ├─4455 postgres: autovacuum launcher
$ [[root@rocky ~]$            ├─4456 postgres: stats collector
$ [[root@rocky ~]$            └─4457 postgres: logical replication launcher
$ [[root@rocky ~]$ 
$ [[root@rocky ~]$ Jun 15 11:53:46 rocky.fritz.box systemd[1]: Starting PostgreSQL 13 database server...
$ [[root@rocky ~]$ Jun 15 11:53:46 rocky.fritz.box postmaster[4449]: 2021-06-15 11:53:46.664 CEST [4449] LOG:  redirecting log output to logging collector process
$ [[root@rocky ~]$ Jun 15 11:53:46 rocky.fritz.box postmaster[4449]: 2021-06-15 11:53:46.664 CEST [4449] HINT:  Future log output will appear in directory "log".
$ [[root@rocky ~]$ Jun 15 11:53:46 rocky.fritz.box systemd[1]: Started PostgreSQL 13 database server.

With this script it is possible to operate several PostgreSQL versions at the same time on the same machine.
The binary folder and $pgdata have the Major Releasenumber of PostgreSQL within the naming.
For PostgreSQL 13 $pgdata is /pgdata/13/data and the binaries are in /usr/pgsql-13/.

With this concept ist is possible to use pg_upgrade in copy mode and having the older version for a certain time in the background available for checks after upgrade.

In my upcoming blogs i will also provide scripst for scale up and down on memory and cpu parameters according to vCPU and Memory setting, pgpass and replication setup using repmgr.

Cet article Recurring PostgreSQL Installations using RHEL 8 and Clones est apparu en premier sur Blog dbi services.

New predefined roles for PostgreSQL 14

$
0
0

Time is moving fast and PostgreSQL 14 is already in beta 2. PostgreSQL 14 will ship with a lot of new features and in this post will look at a smaller one: There are three new predefined roles: pg_read_all_data, pg_write_all_data and pg_database_owner. While it seems to be obvious what the first two roles are about, the third one might sound strange at the beginning. Let’s have a look.

We’ll start with a sample setup, as always:

postgres=# create user u1 with login password 'u1';
CREATE ROLE
postgres=# create user u2 with login password 'u2';
CREATE ROLE
postgres=# create user u3 with login password 'u3';
CREATE ROLE
postgres=# create schema s1;
CREATE SCHEMA
postgres=# create schema s2;
CREATE SCHEMA
postgres=# alter schema s1 owner to u1;
ALTER SCHEMA
postgres=# alter schema s2 owner to u2;
ALTER SCHEMA
postgres=# \c postgres u1
You are now connected to database "postgres" as user "u1".
postgres=> create table s1.t1 ( a int, b text, c date);
CREATE TABLE
postgres=> insert into s1.t1 select i,i::text,now() from generate_series(1,1000000) i;
INSERT 0 1000000
ostgres=> \c postgres u2
You are now connected to database "postgres" as user "u2".
postgres=> create table s2.t1 ( a int, b text, c date);
CREATE TABLE
postgres=> insert into s2.t1 select i,i::text,now() from generate_series(1,1000000) i;
INSERT 0 1000000
postgres=> \c postgres postgres
You are now connected to database "postgres" as user "postgres".

We’ve created three users (u1,u2,u3) and two schemas (s1,s2). The s1 schema is owned by u1 and the s2 schema is owned by u2. Both users (u1,u2) have a table with some data in their schemas. The u3 user currently does not have anything.

When we connect as user u3 we of course do not have any permissions on the schemas we created above:

postgres=# \c postgres u3
You are now connected to database "postgres" as user "u3".
postgres=> select count(*) from s1.t1;
ERROR:  permission denied for schema s1
LINE 1: select count(*) from s1.t1;
                             ^
postgres=> select count(*) from s2.t1;
ERROR:  permission denied for schema s2
LINE 1: select count(*) from s2.t1;
                             ^
postgres=> \c postgres postgres
You are now connected to database "postgres" as user "postgres".

Granting pg_read_all_data to u3 gives as read access to all objects:

postgres=# grant pg_read_all_data to u3;
GRANT ROLE
postgres=# \c postgres u3
You are now connected to database "postgres" as user "u3".
postgres=> select count(*) from s1.t1;
  count  
---------
 1000000
(1 row)

postgres=> select count(*) from s2.t1;
  count  
---------
 1000000
(1 row)

Writing data or creating object is not yet possible:

postgres=> select current_user;
 current_user 
--------------
 u3
(1 row)

postgres=> create table s1.t2(a int);
ERROR:  permission denied for schema s1
LINE 1: create table s1.t2(a int);
                     ^
postgres=> create table s2.t2(a int);
ERROR:  permission denied for schema s2
LINE 1: create table s2.t2(a int);

postgres=> \c postgres postgres
You are now connected to database "postgres" as user "postgres".

This is what pg_write_all_data seems to be about:

postgres=> \c postgres postgres
You are now connected to database "postgres" as user "postgres".
postgres=# grant pg_write_all_data to u3;
GRANT ROLE
postgres=# \c postgres u3
You are now connected to database "postgres" as user "u3".
postgres=> create table s1.t2(a int);
ERROR:  permission denied for schema s1
LINE 1: create table s1.t2(a int);
                     ^
postgres=> create table s2.t2(a int);
ERROR:  permission denied for schema s2
LINE 1: create table s2.t2(a int);

This role does not grant permissions to create new objects but it grant permission to write data into existing relations:

postgres=> select current_user;
 current_user 
--------------
 u3
(1 row)

postgres=> insert into s1.t1 values (-1,'xxx',now() );
INSERT 0 1
postgres=> insert into s2.t1 values (-1,'xxx',now() );
INSERT 0 1

As roles are global objects that works for all databases in a cluster:

postgres=> \c postgres postgres
You are now connected to database "postgres" as user "postgres".
postgres=# create database db1;
CREATE DATABASE
postgres=# \c db1
You are now connected to database "db1" as user "postgres".
db1=# create schema s1;
CREATE SCHEMA
db1=# create table s1.t1(a int);
CREATE TABLE
db1=# insert into s1.t1 values(1);
INSERT 0 1
db1=# \c db1 u3
You are now connected to database "db1" as user "u3".
db1=> select * from s1.t1;
 a 
---
 1
(1 row)

db1=> insert into s1.t1 values(2);
INSERT 0 1

This is really nice if you want to give a specific user or role read and/or write permissions for all objects in all databases.

Finally the third predefined role: pg_database_owner. What is this about? Let’s grant that role to the u3 user:

db1=> \c postgres postgres
You are now connected to database "postgres" as user "postgres".
postgres=# grant pg_database_owner to u3;
ERROR:  role "pg_database_owner" cannot have explicit members

A new predefined role that can not be granted explicitly? Looking at the role definition it can not do much:

postgres=# \x
Expanded display is on.
postgres=# select * from pg_roles where rolname = 'pg_database_owner';
-[ RECORD 1 ]--+------------------
rolname        | pg_database_owner
rolsuper       | f
rolinherit     | t
rolcreaterole  | f
rolcreatedb    | f
rolcanlogin    | f
rolreplication | f
rolconnlimit   | -1
rolpassword    | ********
rolvaliduntil  | 
rolbypassrls   | f
rolconfig      | 
oid            | 6171

What is it there for then? The answer is here.

Cet article New predefined roles for PostgreSQL 14 est apparu en premier sur Blog dbi services.

Creating simple extensions for PostgreSQL

$
0
0

When you are using PostgreSQL you should already know that PostgreSQL comes with a set of extensions by default. It might be that you need to install an additional package if you installed PostgreSQL with a package manager to get those extensions. Usually it is called something with “contrib” in its name. There is also the PostgreSQL Extension Network which list a lot of external extensions that might be useful for you. If you can’t find what you’re looking for, you can still write your own extension. How to do that is the topic of this post: We’ll create a very simple extension for PostgreSQL.

Lets assume you always need a table containing the list of countries with a few properties like this:

postgres=# create table countries ( id int primary key, name text, alpha2 text, alpha3 text );
CREATE TABLE
postgres=# copy countries 
           from program 'curl https://raw.githubusercontent.com/stefangabos/world_countries/master/data/en/countries.csv'
           with (header true, delimiter ',', format csv);
COPY 193

This gives as 193 countries and each country has a few properties:

postgres=# select * from countries limit 5;
 id |    name     | alpha2 | alpha3 
----+-------------+--------+--------
  4 | Afghanistan | af     | afg
  8 | Albania     | al     | alb
 12 | Algeria     | dz     | dza
 20 | Andorra     | ad     | and
 24 | Angola      | ao     | ago
(5 rows)

Packaging this into an extension is quite easy. You need a so called control file that provides some basic information about your extension. In my case it looks like this and I’ve placed it where all the other extensions are as well:

postgres@debian11:/u01/app/postgres/product/DEV/db_1/share/extension/ [pg15] pwd
/u01/app/postgres/product/DEV/db_1/share/extension
postgres@debian11:/u01/app/postgres/product/DEV/db_1/share/extension/ [pg15] cat my_cool_extension.control
# my cool extension control file
comment = 'my cool extension for providing a table with countries'
default_version = '1.0'
relocatable = false
trusted = true

Now we need the table and the data and this goes into a simple SQL file:

postgres@debian11:/home/postgres/ [pg15] pg_dump --column-inserts --table=countries postgres > /u01/app/postgres/product/DEV/db_1/share/extension/my_cool_extension--1.0.sql
postgres@debian11:/home/postgres/ [pg15] psql -c "drop table countries" postgres

If you look at the default extensions they all have the first two lines of the SQL files like e.g. this:

-- complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "CREATE EXTENSION ltree_plpython2u" to load this file. \quit

This prevents that the file can directly be piped into psql and we should do the same. Only “create extension” should be used to create extensions:

postgres@debian11:/home/postgres/ [pgdev] sed -i '1s/^/-- complain if script is sourced in psql, rather than via CREATE EXTENSION/' /u01/app/postgres/product/DEV/db_1/share/extension/my_cool_extension--1.0.sql
postgres@debian11:/home/postgres/ [pgdev] sed -i '2s/^/\\echo Use "CREATE EXTENSION my_cool_extension" to load this file. \\quit\n/' /u01/app/postgres/product/DEV/db_1/share/extension/my_cool_extension--1.0.sql

That’s all you need to do. Having these two files in place the extension is recognized by PostgreSQL:

postgres=# select * from pg_available_extensions where comment like 'my cool%';
       name        | default_version | installed_version |                       comment                        
-------------------+-----------------+-------------------+------------------------------------------------------
 my_cool_extension | 1.0             |                   | my cool extension for providing a table of countries
(1 row)

Now we can install it as usual:

postgres=# create extension my_cool_extension;
CREATE EXTENSION
postgres=# \dx
                                  List of installed extensions
       Name        | Version |   Schema   |                     Description                      
-------------------+---------+------------+------------------------------------------------------
 my_cool_extension | 1.0     | public     | my cool extension for providing a table of countries
 plpgsql           | 1.0     | pg_catalog | PL/pgSQL procedural language
(2 rows)
postgres=# select * from countries limit 5;
 id |    name     | alpha2 | alpha3 
----+-------------+--------+--------
  4 | Afghanistan | af     | afg
  8 | Albania     | al     | alb
 12 | Algeria     | dz     | dza
 20 | Andorra     | ad     | and
 24 | Angola      | ao     | ago
(5 rows)

Over time you might need to update your extension so lets assume in our case that there is a new country. What you need to do is create a new SQL file that does the necessary changes and update the default version in the control file:

postgres@debian11:/u01/app/postgres/product/DEV/db_1/share/extension/ [pgdev] pwd
/u01/app/postgres/product/DEV/db_1/share/extension
postgres@debian11:/u01/app/postgres/product/DEV/db_1/share/extension/ [pgdev] cat my_cool_extension--1.0--1.1.sql
-- complain if script is sourced in psql, rather than via ALTER EXTENSION
\echo Use "ALTER EXTENSION my_cool_extension UPDATE TO '1.1'" to load this file. \quit
-- the list of new countries
insert into countries (id,name,alpha2,alpha3) values (-1,'my new country','aaa','aaa');

postgres@debian11:/u01/app/postgres/product/DEV/db_1/share/extension/ [pgdev] grep default my_cool_extension.control 
default_version = '1.1'

Afterwards update the extension using alter extension:

postgres=# select * from pg_available_extensions where name = 'my_cool_extension';
       name        | default_version | installed_version |                       comment                        
-------------------+-----------------+-------------------+------------------------------------------------------
 my_cool_extension | 1.1             | 1.0               | my cool extension for providing a table of countries
(1 row)

postgres=# alter extension my_cool_extension update;
ALTER EXTENSION
postgres=# select count(*) from countries where id = -1;
 count 
-------
     1
(1 row)

All fine, ready to go.

Cet article Creating simple extensions for PostgreSQL est apparu en premier sur Blog dbi services.


What are these *.ready and *.done files for in PostgreSQL?

$
0
0

When you run PostgreSQL workloads in production you must have a backup and restore implementation. Even for development instances, which are like production for the developers, a well-tested backup and restore procedure sometimes must be in place. Community PostgreSQL comes with pg_basebackup to help you with creating a full consistent backup of your PostgreSQL cluster. If you want to be able to do point in time recovery (PITR) you need to archive the WAL segments which can later be used to roll forward from a base backup. WAL segments go to the pg_wal directory, but if you have closer look into this directory you will see more than just the WAL segments in case you have enabled archiving.

IF you look at a freshly initialized PostgreSQL cluster the content of pg_wal looks like this:

postgres@debian11:/home/postgres/ [pgdev] ls -la $PGDATA/pg_wal
total 16396
drwx------  3 postgres postgres     4096 Jul  4 11:47 .
drwx------ 20 postgres postgres     4096 Jul  4 11:47 ..
-rw-------  1 postgres postgres 16777216 Jul  4 11:52 000000010000000000000001
drwx------  2 postgres postgres     4096 Jul  4 11:47 archive_status

There is one WAL segment and a directory called “archived_status” which is empty right now:

postgres@debian11:/home/postgres/ [pgdev] ls -la $PGDATA/pg_wal/archive_status/
total 8
drwx------ 2 postgres postgres 4096 Jul  4 11:47 .
drwx------ 3 postgres postgres 4096 Jul  4 11:47 ..

To see what’s going on in this directory we need to enable archiving and tell PostgreSQL what do when a WAL segment needs to be archived:

postgres@debian11:/home/postgres/ [pgdev] mkdir /tmp/archived_wal
postgres@debian11:/home/postgres/ [pgdev] psql -c "alter system set archive_mode='on'" postgres
ALTER SYSTEM
postgres@debian11:/home/postgres/ [pgdev] psql -c "alter system set archive_command='test ! -f /tmp/archived_wal/%f && cp %p /tmp/archived_wal/%f'" postgres
ALTER SYSTEM

Of course you should not archive to /tmp and you also should not use cp, as this does not guarantee that the archived segment really is written to disk. For the scope of this post it is fine like that.

Changing the archive command can be done online, but enabling or disabling archiving requires a restart:

postgres@debian11:/home/postgres/ [pgdev] pg_ctl restart

Let’s see what happens if we force the current segment to be archived:

postgres@debian11:/home/postgres/ [pgdev] psql -c "select pg_switch_wal()" postgres
 pg_switch_wal 
---------------
 0/167AF68
(1 row)
postgres@debian11:/home/postgres/ [pgdev] ls -la /tmp/archived_wal/
total 16392
drwxr-xr-x  2 postgres postgres     4096 Jul  4 12:08 .
drwxrwxrwt 10 root     root         4096 Jul  4 12:07 ..
-rw-------  1 postgres postgres 16777216 Jul  4 12:08 000000010000000000000001

As expected we see the archived segment in the directory we specified in the archive command. In addition PostgreSQL generated a “.done” file in the archive_status directory in pg_wal:

postgres@debian11:/home/postgres/ [pgdev] ls -la $PGDATA/pg_wal/archive_status/
total 8
drwx------ 2 postgres postgres 4096 Jul  4 12:08 .
drwx------ 3 postgres postgres 4096 Jul  4 12:08 ..
-rw------- 1 postgres postgres    0 Jul  4 12:08 000000010000000000000001.done

So, whenever a segment was successfully archived you get the corresponding “.done” file. When do you see “.ready” files then? The answer is simple: Exactly when the opposite happens: Archiving of a segment failed for whatever reason:

postgres@debian11:/home/postgres/ [pgdev] psql -c "alter system set archive_command='/bin/false'" postgres
ALTER SYSTEM
postgres@debian11:/home/postgres/ [pgdev] psql -c "select pg_reload_conf()";
 pg_reload_conf 
----------------
 t
(1 row)
postgres@debian11:/home/postgres/ [pgdev] psql -c "select pg_switch_wal()" postgres
 pg_switch_wal 
---------------
 0/2000160
(1 row)
postgres@debian11:/home/postgres/ [pgdev] ls -la $PGDATA/pg_wal/archive_status/
total 8
drwx------ 2 postgres postgres 4096 Jul  4 12:13 .
drwx------ 3 postgres postgres 4096 Jul  4 12:12 ..
-rw------- 1 postgres postgres    0 Jul  4 12:13 000000010000000000000002.ready

This file will be there until archiving succeeds again:

postgres@debian11:/home/postgres/ [pgdev] psql -c "alter system set archive_command='test ! -f /tmp/archived_wal/%f && cp %p /tmp/archived_wal/%f'" postgres
ALTER SYSTEM
postgres@debian11:/home/postgres/ [pgdev] psql -c "select pg_reload_conf()";
 pg_reload_conf 
----------------
 t
(1 row)
postgres@debian11:/home/postgres/ [pgdev] ls -la $PGDATA/pg_wal/archive_status/
total 8
drwx------ 2 postgres postgres 4096 Jul  4 12:15 .
drwx------ 3 postgres postgres 4096 Jul  4 12:12 ..
-rw------- 1 postgres postgres    0 Jul  4 12:13 000000010000000000000002.done

When you see many “.ready” files this can either mean your archive command is currently failing or load on the system is so high, that archiving can not keep up.

Cet article What are these *.ready and *.done files for in PostgreSQL? est apparu en premier sur Blog dbi services.

Nutanix Era with oracle databases : Part 1 – Introduction

$
0
0

I’m currently setting up a lab for Nutanix Era in order to be able to provide some presentation on this subject and see how Nutanix Era interacts with Oracle databases. I therefore thought it would be certainly helpful and a good opportunity to share the feedback I’m getting from this experience. This fist blog intends to provide only a brief introduction on Nutanix Era. I will then write some additional blogs in order to describe :
Part 2 – How to create a VM template for Oracle database implementation
Part 3 – How to provision an oracle database
Part 4 – How to clone an oracle database
Part 5 – How to refresh an oracle database clone
And more part coming on the way…


What is Nutanix?

Nutanix is a private Cloud solution for on-premises. You will see that Nutanix GUI is really look and feel Cloud. It is a hybrid cloud solution as Nutanix can be integrated with AWS, Azure and Google Cloud.

Nutanix is one of the most popular hyperconverged solution (HCI). Its distributed systems architecture will provide unlimited scalability, high performance and high data protection. A hyperconverged solution will combine datacenter hardware into shared resource pool for :

  • Compute
  • Storage
  • Storage network
  • Virtualization

Hyperconverged solution will then facilitate the datacenter management and deliver quick services on on-premise.

Nutanix solution is hardware agnostic. It is compatible with Dell, Lenovo, Cisco and HPE Proliant hardware. Of course Nutanix has their own hardware as well.

The solution is composed of a cluster of a minimum of 3 servers/nodes. Nutanix software AOS is deployed on the cluster.

Native Nutanix hypervisor is called AHV. The solution support also various of other hypervisor like Vmware ESXi, Microsoft Hyper-V and Citrix Hpervisor.

The next schema (referenced from Nutanix documentation) will briefly describe the architecture.



  • The CVM is the controller VM. It will host the core software in order to serve and manage all the I/O operations for the hypervisor and all VMs running on the specific host. The CVM will host as well Prism Management GUI which will be used to manage the resource.
  • The Distributed Storage Fabric (DFS) controls the storage pool and distributes data and metadata across all the nodes in the cluster.
  • The hypervisor is the virtual machine monitor. It will be in charge of creating and running all the VMs. The hypervisor is in charge of virtually sharing the resources (memory and processor) from its physical server/node to all the created VMs.

The solution will allow tunable redundancy. The Replication Factor (RF) will define the number of copies of data at all times accross nodes.

What is Nutanix Era?

On the other hand Nutanix Era is a platform, running on one VM of the Nutanix Cluster, that will help us to easily manage, create, update and keep track of the databases. It will simplify the database management through 1 central interface. With Nutanix Era I can easily :

  • Provision new database
  • Clone existing database
  • Delete database
  • Refresh database
  • Backup database
  • Patch database

It will as well support HA with RAC possiblities, but still do not incorporate Data Guard. You will still need to build your Data Guard solution between primary and standby databases running on separate VMs from the cluster.

Be careful. Management of the database done by Nutanix Era does not include your DBA tasks!

One of the most interesting part is also that from the same central GUI you will be able to run and manage various of databases :

  • SQL Server
  • PostgreSQL
  • SAP HANA
  • MySQL
  • MariaDB
  • Oracle database



Current Nutanix Era version is 2.2.

Nutanix Era is a database as a service (DBaaS). It will solve the long traditional provisioning process involving multiple teams and specialists :

  • DB request
  • Configure server
    • Create Server
    • Allocate storage
    • Setup the network
    • Create the cluster
    • Provision a DB
  • and so on…

How to easily test Nutanix Era?

You can easily test Nutanix Era on your own with MariaDB or PostgreSQL databases using the live lab test drive.

Creation of our Nutanix lab

For our purpose we started a 30 days cluster trial from my Nutanix website.
With the help of Nutanix team we could interface our AWS Cloud with the Nutanix 30 days Cluster in order to setup the cluster and deploy Nutanix Era using a image configuration in Prism GUI.

Documentation

There is one link to know, the most important : Nutanix Bible

Nutanix Era documentation : Nutanix Era User Guide

Cet article Nutanix Era with oracle databases : Part 1 – Introduction est apparu en premier sur Blog dbi services.

What are those *_init files for in PostgreSQL?

$
0
0

Quite some time ago I’ve written about relation files in PostgreSQL and when they get created. In that post we talked about the files containing the actual data, the free space map (fsm) and the visibility map (vm). But this is not all, there is another kind of file you might spot if you take a look into the directory of a database.

The directory of the postgres database of a fresh initialized PostgreSQL cluster looks like this:

postgres@debian11pg:/u02/pgdata/PGDEV/base/13540/ [pgdev] oid2name 
All databases:
    Oid  Database Name  Tablespace
----------------------------------
  13540       postgres  pg_default
  13539      template0  pg_default
      1      template1  pg_default
postgres@debian11pg:/u02/pgdata/PGDEV/base/13540/ [pgdev] ls
112       1259       13366      13377  2337      2603      2607_fsm  2611      2617      2651  2662  2675  2688  2704      2833      2840      3085  3381      3455      3541      3600      3603_vm   3766  4151  4164  5002  6175
113       1259_fsm   13367      1417   2579      2603_fsm  2607_vm   2612      2617_fsm  2652  2663  2678  2689  2753      2834      2840_fsm  3118  3394      3456      3541_fsm  3600_fsm  3604      3767  4152  4165  548   6176
1247      1259_vm    13368      1418   2600      2603_vm   2608      2612_fsm  2617_vm   2653  2664  2679  2690  2753_fsm  2835      2840_vm   3119  3394_fsm  3456_fsm  3541_vm   3600_vm   3605      3997  4153  4166  549   826
1247_fsm  13358      13368_fsm  16387  2600_fsm  2604      2608_fsm  2612_vm   2618      2654  2665  2680  2691  2753_vm   2836      2841      3164  3394_vm   3456_vm   3542      3601      3606      4143  4154  4167  6102  827
1247_vm   13358_fsm  13368_vm   174    2600_vm   2605      2608_vm   2613      2618_fsm  2655  2666  2681  2692  2754      2836_fsm  2995      3256  3395      3466      3574      3601_fsm  3607      4144  4155  4168  6104  828
1249      13358_vm   13371      175    2601      2605_fsm  2609      2615      2618_vm   2656  2667  2682  2693  2755      2836_vm   2996      3257  3429      3467      3575      3601_vm   3608      4145  4156  4169  6106  pg_filenode.map
1249_fsm  13361      13372      2187   2601_fsm  2605_vm   2609_fsm  2615_fsm  2619      2657  2668  2683  2696  2756      2837      3079      3258  3430      3468      3576      3602      3609      4146  4157  4170  6110  pg_internal.init
1249_vm   13362      13373      2224   2601_vm   2606      2609_vm   2615_vm   2619_fsm  2658  2669  2684  2699  2757      2838      3079_fsm  3350  3431      3501      3596      3602_fsm  3712      4147  4158  4171  6111  PG_VERSION
1255      13363      13373_fsm  2228   2602      2606_fsm  2610      2616      2619_vm   2659  2670  2685  2701  2830      2838_fsm  3079_vm   3351  3433      3502      3597      3602_vm   3764      4148  4159  4172  6112
1255_fsm  13363_fsm  13373_vm   2328   2602_fsm  2606_vm   2610_fsm  2616_fsm  2620      2660  2673  2686  2702  2831      2838_vm   3080      3379  3439      3503      3598      3603      3764_fsm  4149  4160  4173  6113
1255_vm   13363_vm   13376      2336   2602_vm   2607      2610_vm   2616_vm   2650      2661  2674  2687  2703  2832      2839      3081      3380  3440      3534      3599      3603_fsm  3764_vm   4150  4163  4174  6117

You can see the files containing actual data (the numbers with any suffix), free space and visibility maps and three other files will not look at in this post (pg_filenode.map,pg_internal.init,PG_VERSION).

Lets assume the cluster is running for the time, the application got deployed and users are already working on it. For whatever reason you check the directory of the postgres database once more and now it looks like this:

postgres@debian11pg:/u02/pgdata/PGDEV/base/13540/ [pgdev] ls
112       1259       13366      13377       2224      2601_vm   2606      2609_vm   2615_vm   2619_fsm  2658  2669  2684  2699      2757      2838      3079_fsm  3350      3431      3501      3596      3602_fsm  3712      4147  4158  4171  6111              PG_VERSION
113       1259_fsm   13367      1417        2228      2602      2606_fsm  2610      2616      2619_vm   2659  2670  2685  2701      2830      2838_fsm  3079_vm   3351      3433      3502      3597      3602_vm   3764      4148  4159  4172  6112
1247      1259_vm    13368      1418        2328      2602_fsm  2606_vm   2610_fsm  2616_fsm  2620      2660  2673  2686  2702      2831      2838_vm   3080      3379      3439      3503      3598      3603      3764_fsm  4149  4160  4173  6113
1247_fsm  13358      13368_fsm  16387       2336      2602_vm   2607      2610_vm   2616_vm   2650      2661  2674  2687  2703      2832      2839      3081      3380      3440      3534      3599      3603_fsm  3764_vm   4150  4163  4174  6117
1247_vm   13358_fsm  13368_vm   16388       2337      2603      2607_fsm  2611      2617      2651      2662  2675  2688  2704      2833      2840      3085      3381      3455      3541      3600      3603_vm   3766      4151  4164  5002  6175
1249      13358_vm   13371      16388_init  2579      2603_fsm  2607_vm   2612      2617_fsm  2652      2663  2678  2689  2753      2834      2840_fsm  3118      3394      3456      3541_fsm  3600_fsm  3604      3767      4152  4165  548   6176
1249_fsm  13361      13372      16391       2600      2603_vm   2608      2612_fsm  2617_vm   2653      2664  2679  2690  2753_fsm  2835      2840_vm   3119      3394_fsm  3456_fsm  3541_vm   3600_vm   3605      3997      4153  4166  549   826
1249_vm   13362      13373      16391_init  2600_fsm  2604      2608_fsm  2612_vm   2618      2654      2665  2680  2691  2753_vm   2836      2841      3164      3394_vm   3456_vm   3542      3601      3606      4143      4154  4167  6102  827
1255      13363      13373_fsm  174         2600_vm   2605      2608_vm   2613      2618_fsm  2655      2666  2681  2692  2754      2836_fsm  2995      3256      3395      3466      3574      3601_fsm  3607      4144      4155  4168  6104  828
1255_fsm  13363_fsm  13373_vm   175         2601      2605_fsm  2609      2615      2618_vm   2656      2667  2682  2693  2755      2836_vm   2996      3257      3429      3467      3575      3601_vm   3608      4145      4156  4169  6106  pg_filenode.map
1255_vm   13363_vm   13376      2187        2601_fsm  2605_vm   2609_fsm  2615_fsm  2619      2657      2668  2683  2696  2756      2837      3079      3258      3430      3468      3576      3602      3609      4146      4157  4170  6110  pg_internal.init

In addition to the files we already know, there are two “_init” files. What are they used for? To answer that lets check the names of those relations using oid2name:

postgres@debian11pg:/u02/pgdata/PGDEV/base/13540/ [pgdev] oid2name -f 16388
From database "postgres":
  Filenode  Table Name
----------------------
     16388          t1
postgres@debian11pg:/u02/pgdata/PGDEV/base/13540/ [pgdev] oid2name -f 16391
From database "postgres":
  Filenode  Table Name
----------------------
     16391          t2

Asking pg_class what type of relation those are, tells as both are ordinary tables:

postgres=# select relkind from pg_class where relname in ('t1','t2');
 relkind 
---------
 r
 r
(2 rows)

But ordinary tables do not have any "*_init" files, do they? They have, as long as they are created as "unlogged" tables:

postgres=# \d t1
            Unlogged table "public.t1"
 Column |  Type   | Collation | Nullable | Default 
--------+---------+-----------+----------+---------
 a      | integer |           |          | 

postgres=# \d t2
            Unlogged table "public.t2"
 Column |  Type   | Collation | Nullable | Default 
--------+---------+-----------+----------+---------
 a      | integer |           |          | 

Each unlogged table automatically gets a so-called initialization fork:

postgres=# create unlogged table t3 ( a int );
CREATE TABLE
postgres=# select pg_relation_filepath('t3');
 pg_relation_filepath 
----------------------
 base/13540/16394
(1 row)

postgres=# \! ls -l $PGDATA/base/13540/16394*
-rw------- 1 postgres postgres 0 Aug 24 13:17 /u02/pgdata/PGDEV/base/13540/16394
-rw------- 1 postgres postgres 0 Aug 24 13:17 /u02/pgdata/PGDEV/base/13540/16394_init

What are those forks used for? What you need to know with unlogged tables is, that they are re-initialized when PostgreSQL crashes. First lets add some data to the unlogged table:

postgres=# insert into t3 select * from generate_series(1,1000);
INSERT 0 1000
postgres=# select count(*) from t3;
 count 
-------
  1000
(1 row)

Now lets crash PostgreSQL:

postgres@debian11pg:/home/postgres/ [pgdev] ps -ef | grep "postgres \-D"
postgres   35443       1  0 13:22 ?        00:00:00 /u01/app/postgres/product/DEV/db_0/bin/postgres -D /u02/pgdata/PGDEV
postgres@debian11pg:/home/postgres/ [pgdev] kill -9 35443

Starting up and checking the content of the tables confirms that it is empty:

postgres@debian11pg:/home/postgres/ [pgdev] pg_ctl start
postgres@debian11pg:/home/postgres/ [pgdev] psql -c "select count(*) from t3" postgres
 count 
-------
     0
(1 row)

What happened is, that PostgreSQL replaced the data file of that unlogged table with it’s initialization fork.

Cet article What are those *_init files for in PostgreSQL? est apparu en premier sur Blog dbi services.

Parallel distinct for PostgreSQL 15

$
0
0

When parallel execution was introduced with PostgreSQL 9.6 it was quite limited and supported only a few operations to run in parallel. Over time more and more operations were enabled to run in parallel. When PostgreSQL 15 will be released next year it will come with parallel distinct. Here is an easy example:

If you do something like this, e.g. in PostgreSQL 13:

postgres=# select version();
                                              version                                              
---------------------------------------------------------------------------------------------------
 PostgreSQL 13.3 on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
(1 row)
postgres=# create table t1 ( a int );
CREATE TABLE
postgres=# insert into t1 select 1 from generate_series(1,100000);
INSERT 0 100000
postgres=# insert into t1 select 2 from generate_series(1,300000);
INSERT 0 300000
postgres=# 
postgres=# insert into t1 select 4 from generate_series(1,400000);
INSERT 0 400000
postgres=# insert into t1 select 5 from generate_series(1,500000);
INSERT 0 500000
postgres=# analyze t1;
ANALYZE

A distinct over the “a” column will always run serially:

postgres=# explain analyze select distinct a from t1;

                                                     QUERY PLAN                                                     
--------------------------------------------------------------------------------------------------------------------
 HashAggregate  (cost=22003.00..22003.04 rows=4 width=4) (actual time=1295.168..1295.172 rows=4 loops=1)
   Group Key: a
   Batches: 1  Memory Usage: 24kB
   ->  Seq Scan on t1  (cost=0.00..18753.00 rows=1300000 width=4) (actual time=0.041..608.124 rows=1300000 loops=1)
 Planning Time: 0.249 ms
 Execution Time: 1295.887 ms
(6 rows)

Doing the same test against the current master branch shows that parallel execution is triggered:

postgres=# select version();
                                               version                                                
------------------------------------------------------------------------------------------------------
 PostgreSQL 15devel on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
(1 row)

postgres=# create table t1 ( a int );
CREATE TABLE
postgres=# insert into t1 select 1 from generate_series(1,100000);
INSERT 0 100000
postgres=# insert into t1 select 2 from generate_series(1,300000);
INSERT 0 300000
postgres=# insert into t1 select 4 from generate_series(1,400000);
INSERT 0 400000
postgres=# insert into t1 select 5 from generate_series(1,500000);
INSERT 0 500000
postgres=# analyze t1;
ANALYZE
postgres=# explain analyze select distinct a from t1;
                                                                 QUERY PLAN                                                                  
---------------------------------------------------------------------------------------------------------------------------------------------
 Unique  (cost=13524.79..13524.83 rows=4 width=4) (actual time=1382.720..1383.346 rows=4 loops=1)
   ->  Sort  (cost=13524.79..13524.81 rows=8 width=4) (actual time=1382.716..1383.332 rows=12 loops=1)
         Sort Key: a
         Sort Method: quicksort  Memory: 25kB
         ->  Gather  (cost=13523.83..13524.67 rows=8 width=4) (actual time=1382.681..1383.307 rows=12 loops=1)
               Workers Planned: 2
               Workers Launched: 2
               ->  HashAggregate  (cost=12523.83..12523.87 rows=4 width=4) (actual time=1340.241..1340.245 rows=4 loops=3)
                     Group Key: a
                     Batches: 1  Memory Usage: 24kB
                     Worker 0:  Batches: 1  Memory Usage: 24kB
                     Worker 1:  Batches: 1  Memory Usage: 24kB
                     ->  Parallel Seq Scan on t1  (cost=0.00..11169.67 rows=541667 width=4) (actual time=0.028..663.671 rows=433333 loops=3)
 Planning Time: 0.338 ms
 Execution Time: 1383.477 ms
(15 rows)

Nice, details here.

Cet article Parallel distinct for PostgreSQL 15 est apparu en premier sur Blog dbi services.

Temporal tables for PostgreSQL 15?

$
0
0

One of the features which is currently missing in PostgreSQL is Temporal Tables. Other database systems have that since a long time and many people want to have in PostgreSQL as well. If you don’t know what it is, here is a short overview. Basically you can ask for a row as it was at a specific point in time. The patch which implements this, is currently in status “Ready for Committer”, which does not guarantee that it actually will be committed, but it should be in shape to test the functionality that comes with it.

Having applied the patch and re-compiled PostgreSQL the feature can be tested. The most easy way to enable the feature for a table is this:

postgres=# create table t1 ( a int primary key generated always as identity
                           , b text ) 
                           with system versioning;
CREATE TABLE
postgres=# \d t1
                                      Table "public.t1"
  Column   |           Type           | Collation | Nullable |            Default            
-----------+--------------------------+-----------+----------+-------------------------------
 a         | integer                  |           | not null | generated always as identity
 b         | text                     |           |          | 
 starttime | timestamp with time zone |           | not null | generated always as row start
 endtime   | timestamp with time zone |           | not null | generated always as row end
Indexes:
    "t1_pkey" PRIMARY KEY, btree (a, endtime)

Two additional columns have been added automatically, starttime and endtime. If you don’t like the column names you can also do it like this:

postgres=# create table t2 ( a int primary key generated always as identity
                           , b text
                           , mystart timestamptz generated always as row start
                           , myend timestamptz generated always as row end );
CREATE TABLE
postgres=# \d t2
                                     Table "public.t2"
 Column  |           Type           | Collation | Nullable |            Default            
---------+--------------------------+-----------+----------+-------------------------------
 a       | integer                  |           | not null | generated always as identity
 b       | text                     |           |          | 
 mystart | timestamp with time zone |           | not null | generated always as row start
 myend   | timestamp with time zone |           | not null | generated always as row end
Indexes:
    "t2_pkey" PRIMARY KEY, btree (a, myend)

Note that the two columns need to be defined as timestamptz:

postgres=# create table t3 ( a int primary key generated always as identity, b text, mystart date generated always as row start, myend date generated always as row end );
ERROR:  data type of row start time must be timestamptz

To see the feature in action lets add some data, fire an update and have a look at the history:

postgres=# insert into t1 (b) select md5(a::text) from generate_series(1,10) a;
INSERT 0 10
postgres=# update t1 set b = random()::text;
UPDATE 10
postgres=# select * from t1 for system_time from '-infinity' to 'infinity' order by a;
 a  |                b                 |           starttime           |            endtime            
----+----------------------------------+-------------------------------+-------------------------------
  1 | c4ca4238a0b923820dcc509a6f75849b | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  1 | 0.42111793538855835              | 2021-08-24 16:16:55.417076+02 | infinity
  2 | c81e728d9d4c2f636f067f89cc14862c | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  2 | 0.5479128803753532               | 2021-08-24 16:16:55.417076+02 | infinity
  3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  3 | 0.5512468293024142               | 2021-08-24 16:16:55.417076+02 | infinity
  4 | a87ff679a2f3e71d9181a67b7542122c | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  4 | 0.4112741522472554               | 2021-08-24 16:16:55.417076+02 | infinity
  5 | e4da3b7fbbce2345d7772b0674a318d5 | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  5 | 0.46017420469036807              | 2021-08-24 16:16:55.417076+02 | infinity
  6 | 1679091c5a880faf6fb5e6087eb1b2dc | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  6 | 0.3495216613664702               | 2021-08-24 16:16:55.417076+02 | infinity
  7 | 8f14e45fceea167a5a36dedd4bea2543 | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  7 | 0.2657576876373895               | 2021-08-24 16:16:55.417076+02 | infinity
  8 | c9f0f895fb98ab9159f51fd0297e236d | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  8 | 0.9808748465536858               | 2021-08-24 16:16:55.417076+02 | infinity
  9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  9 | 0.4533070845652887               | 2021-08-24 16:16:55.417076+02 | infinity
 10 | d3d9446802a44259755d38e6d163e820 | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
 10 | 0.20914767879762408              | 2021-08-24 16:16:55.417076+02 | infinity
(20 rows)

The rows with “endtime = infinity” are the current ones, the ones with an actual endtime are historical versions. Doing the almost same thing again makes it more clear:

postgres=# update t1 set b = 'xxxxx';
UPDATE 10
postgres=# select * from t1 for system_time from '-infinity' to 'infinity' order by a, starttime;
 a  |                b                 |           starttime           |            endtime            
----+----------------------------------+-------------------------------+-------------------------------
  1 | c4ca4238a0b923820dcc509a6f75849b | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  1 | 0.42111793538855835              | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  1 | xxxxx                            | 2021-08-24 16:58:24.799322+02 | infinity
  2 | c81e728d9d4c2f636f067f89cc14862c | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  2 | 0.5479128803753532               | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  2 | xxxxx                            | 2021-08-24 16:58:24.799322+02 | infinity
  3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  3 | 0.5512468293024142               | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  3 | xxxxx                            | 2021-08-24 16:58:24.799322+02 | infinity
  4 | a87ff679a2f3e71d9181a67b7542122c | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  4 | 0.4112741522472554               | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  4 | xxxxx                            | 2021-08-24 16:58:24.799322+02 | infinity
  5 | e4da3b7fbbce2345d7772b0674a318d5 | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  5 | 0.46017420469036807              | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  5 | xxxxx                            | 2021-08-24 16:58:24.799322+02 | infinity
  6 | 1679091c5a880faf6fb5e6087eb1b2dc | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  6 | 0.3495216613664702               | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  6 | xxxxx                            | 2021-08-24 16:58:24.799322+02 | infinity
  7 | 8f14e45fceea167a5a36dedd4bea2543 | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  7 | 0.2657576876373895               | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  7 | xxxxx                            | 2021-08-24 16:58:24.799322+02 | infinity
  8 | c9f0f895fb98ab9159f51fd0297e236d | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  8 | 0.9808748465536858               | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  8 | xxxxx                            | 2021-08-24 16:58:24.799322+02 | infinity
  9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  9 | 0.4533070845652887               | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  9 | xxxxx                            | 2021-08-24 16:58:24.799322+02 | infinity
 10 | d3d9446802a44259755d38e6d163e820 | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
 10 | 0.20914767879762408              | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
 10 | xxxxx                            | 2021-08-24 16:58:24.799322+02 | infinity
(30 rows)

Without specifying any time frame you get the current version of rows, of course:

postgres=# select * from t1 order by a;
 a  |   b   |           starttime           | endtime  
----+-------+-------------------------------+----------
  1 | xxxxx | 2021-08-24 16:58:24.799322+02 | infinity
  2 | xxxxx | 2021-08-24 16:58:24.799322+02 | infinity
  3 | xxxxx | 2021-08-24 16:58:24.799322+02 | infinity
  4 | xxxxx | 2021-08-24 16:58:24.799322+02 | infinity
  5 | xxxxx | 2021-08-24 16:58:24.799322+02 | infinity
  6 | xxxxx | 2021-08-24 16:58:24.799322+02 | infinity
  7 | xxxxx | 2021-08-24 16:58:24.799322+02 | infinity
  8 | xxxxx | 2021-08-24 16:58:24.799322+02 | infinity
  9 | xxxxx | 2021-08-24 16:58:24.799322+02 | infinity
 10 | xxxxx | 2021-08-24 16:58:24.799322+02 | infinity

Asking for a specific point in time works as well:

postgres=# select * from t1 for system_time as of '2021-08-24 16:16:25.648976+02'::timestamptz;
 a  |                b                 |           starttime           |            endtime            
----+----------------------------------+-------------------------------+-------------------------------
  1 | c4ca4238a0b923820dcc509a6f75849b | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  2 | c81e728d9d4c2f636f067f89cc14862c | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  3 | eccbc87e4b5ce2fe28308fd9f2a7baf3 | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  4 | a87ff679a2f3e71d9181a67b7542122c | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  5 | e4da3b7fbbce2345d7772b0674a318d5 | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  6 | 1679091c5a880faf6fb5e6087eb1b2dc | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  7 | 8f14e45fceea167a5a36dedd4bea2543 | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  8 | c9f0f895fb98ab9159f51fd0297e236d | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
  9 | 45c48cce2e2d7fbdea1afc51c7c6ad26 | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02
 10 | d3d9446802a44259755d38e6d163e820 | 2021-08-24 16:16:25.648976+02 | 2021-08-24 16:16:55.417076+02

… or asking like this:

postgres=# select * from t1 for system_time between '2021-08-24 16:16:55.417076+02'::timestamptz and '2021-08-24 16:58:24.799322+02'::timestamptz;
 a  |          b          |           starttime           |            endtime            
----+---------------------+-------------------------------+-------------------------------
  1 | 0.42111793538855835 | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  1 | xxxxx               | 2021-08-24 16:58:24.799322+02 | infinity
  2 | 0.5479128803753532  | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  2 | xxxxx               | 2021-08-24 16:58:24.799322+02 | infinity
  3 | 0.5512468293024142  | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  3 | xxxxx               | 2021-08-24 16:58:24.799322+02 | infinity
  4 | 0.4112741522472554  | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  4 | xxxxx               | 2021-08-24 16:58:24.799322+02 | infinity
  5 | 0.46017420469036807 | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  5 | xxxxx               | 2021-08-24 16:58:24.799322+02 | infinity
  6 | 0.3495216613664702  | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  6 | xxxxx               | 2021-08-24 16:58:24.799322+02 | infinity
  7 | 0.2657576876373895  | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  7 | xxxxx               | 2021-08-24 16:58:24.799322+02 | infinity
  8 | 0.9808748465536858  | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  8 | xxxxx               | 2021-08-24 16:58:24.799322+02 | infinity
  9 | 0.4533070845652887  | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
  9 | xxxxx               | 2021-08-24 16:58:24.799322+02 | infinity
 10 | 0.20914767879762408 | 2021-08-24 16:16:55.417076+02 | 2021-08-24 16:58:24.799322+02
 10 | xxxxx               | 2021-08-24 16:58:24.799322+02 | infinity

Really looks promising. Thanks to all involved.

Cet article Temporal tables for PostgreSQL 15? est apparu en premier sur Blog dbi services.

Viewing all 522 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>