Quantcast
Channel: Archives des PostgreSQL - dbi Blog
Viewing all 526 articles
Browse latest View live

Save the date: Swiss PGDay 2017

$
0
0

The 30th of June is not too far away and if you want to learn more about the “World’s most advanced open source database” you should register soon. The sessions are not yet officially announced but you can expect that to happen next week. What we can say already is that we will be there and that you can expect these sessions from dbi:

  • How to migrate data from MongoDB to Postgres with ToroDB
  • Elking your PostgreSQL database infrastructure
  • PostgreSQL upgrade best practices

Hope to see you in June.
PGDay2017CH

 

Cet article Save the date: Swiss PGDay 2017 est apparu en premier sur Blog dbi services.


PostgreSQL on a Pure Storage All Flash Array – Preparations

$
0
0

Yesterday we got a Pure Storage All Flash Array for testing. As the name implies this is all about Flash storage. What makes Pure Storage different from other vendors is that you don’t buy just a storage box and then pay the usual maintenance costs but you pay for a storage subscription which should keep your storage up to date all the time. The promise is that all the components of the array get replaced by the then current versions over time without forcing you to re-buy. Check the link above for more details on the available subscriptions. This is the first post and describes the setup we did for connecting a PostgreSQL VMWare based machine to the Pure Storage box. The PostgreSQL server will be running as a virtual machine in VMWare ESX and connect over iSCSI to the storage system.

As usual we used CentOS 7 for the PostgreSQL server:

[root@pgpurestorage ~]$ cat /etc/centos-release
CentOS Linux release 7.3.1611 (Core)
[root@pgpurestorage ~]$ uname -a
Linux pgpurestorage.it.dbi-services.com 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

We have 4 vCPUs:

[root@pgpurestorage ~]$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             4
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Model name:            Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
Stepping:              1
CPU MHz:               2399.583
BogoMIPS:              4799.99
Hypervisor vendor:     VMware
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              35840K
NUMA node0 CPU(s):     0-3

… and 8GB of memory:

[root@pgpurestorage ~]$ cat /proc/meminfo | head -5
MemTotal:        7994324 kB
MemFree:         7508232 kB
MemAvailable:    7528048 kB
Buffers:            1812 kB
Cached:           233648 kB

purestorage-setup-vm

Because by default you’ll get the “virtual-guest” tuned profile when you install CentOS in a virtualized environment we created our own and switched to the same:

root@:/home/postgres/ [] tuned-adm active
Current active profile: virtual-guest
root@:/home/postgres/ [] tuned-adm profile dbi-postgres
root@:/home/postgres/ [] tuned-adm active
Current active profile: dbi-postgres
root@:/home/postgres/ [] cat /usr/lib/tuned/dbi-postgres/tuned.conf | egrep -v "^#|^$"
[main]
summary=dbi services tuned profile for PostgreSQL servers
[cpu]
governor=performance
energy_perf_bias=performance
min_perf_pct=100
[disk]
readahead=>4096
[sysctl]
kernel.sched_min_granularity_ns = 10000000
kernel.sched_wakeup_granularity_ns = 15000000
vm.overcommit_memory=2
vm.swappiness=0
vm.dirty_ratio=2
vm.dirty_background_ratio=1
vm.nr_hugepages=1024

To gather statistics we created a cronjob:

root@:/home/postgres/ []  crontab -l
* * * * * /usr/lib64/sa/sa1 -S XALL 60 1

PostgreSQL was installed from source with what was committed to the source tree as of today with the following options:

[postgres@pgpurestorage postgresql]$ PGHOME=/u01/app/postgres/product/10/db_0
[postgres@pgpurestorage postgresql]$ SEGSIZE=2
[postgres@pgpurestorage postgresql]$ BLOCKSIZE=8
[postgres@pgpurestorage postgresql]$ WALSEGSIZE=64
[postgres@pgpurestorage postgresql]$ ./configure --prefix=${PGHOME} \
>             --exec-prefix=${PGHOME} \
>             --bindir=${PGHOME}/bin \
>             --libdir=${PGHOME}/lib \
>             --sysconfdir=${PGHOME}/etc \
>             --includedir=${PGHOME}/include \
>             --datarootdir=${PGHOME}/share \
>             --datadir=${PGHOME}/share \
>             --with-pgport=5432 \
>             --with-perl \
>             --with-python \
>             --with-tcl \
>             --with-openssl \
>             --with-pam \
>             --with-ldap \
>             --with-libxml \
>             --with-libxslt \
>             --with-segsize=${SEGSIZE} \
>             --with-blocksize=${BLOCKSIZE} \
>             --with-wal-segsize=${WALSEGSIZE}  \
>             --with-extra-version=" dbi services build"

For being able to connect to the Pure Storage box you’ll need the iSCSI IQN:

root@:/home/postgres/ [] cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:185a3499ac9

Knowing the IQN (InitiatorName) we can logon to the Pure Storage console for adding our host, creating a volume and attaching the volume to the host:

Thu May 04 11:44:10 2017
Welcome pureuser. This is Purity Version 4.8.8 on FlashArray dbipure01

http://www.purestorage.com/

pureuser@dbipure01> purehost create --iqn iqn.1994-05.com.redhat:185a3499ac9 pgpurestorage
Name           WWN  IQN
pgpurestorage  -    iqn.1994-05.com.redhat:185a3499ac9
pureuser@dbipure01> purevol create --size 500G volpgtest
Name       Size  Source  Created                   Serial
volpgtest  500G  -       2017-05-04 11:46:58 CEST  BA56B4A72DE94A4400011012
pureuser@dbipure01> purehost connect --vol volpgtest pgpurestorage
Name           Vol        LUN
pgpurestorage  volpgtest  1

The Pure Storage system has two controllers (10.10.1.93 and 10.10.1.94) so we should be able to ping them:

root@:/home/postgres/ [] ping 10.10.1.93
PING 10.10.1.93 (10.10.1.93) 56(84) bytes of data.
64 bytes from 10.10.1.93: icmp_seq=1 ttl=63 time=2.53 ms
64 bytes from 10.10.1.93: icmp_seq=2 ttl=63 time=0.816 ms
64 bytes from 10.10.1.93: icmp_seq=3 ttl=63 time=0.831 ms
...
root@:/u02/pgdata/pgpure/ [] ping 10.10.1.94
PING 10.10.1.94 (10.10.1.94) 56(84) bytes of data.
64 bytes from 10.10.1.94: icmp_seq=1 ttl=63 time=0.980 ms
64 bytes from 10.10.1.94: icmp_seq=2 ttl=63 time=0.848 ms
...

Ok for the connectivity so a discover should work as well:

root@:/home/postgres/ [] iscsiadm -m discovery -t st -p 10.10.1.93
10.10.1.93:3260,1 iqn.2010-06.com.purestorage:flasharray.516cdd52f827bd21
10.10.1.94:3260,1 iqn.2010-06.com.purestorage:flasharray.516cdd52f827bd21
root@:/home/postgres/ [] iscsiadm -m node
10.10.1.93:3260,1 iqn.2010-06.com.purestorage:flasharray.516cdd52f827bd21
10.10.1.94:3260,1 iqn.2010-06.com.purestorage:flasharray.516cdd52f827bd21

Fine as well, so login:

root@:/home/postgres/ [] iscsiadm -m node --login
Logging in to [iface: default, target: iqn.2010-06.com.purestorage:flasharray.516cdd52f827bd21, portal: 10.10.1.93,3260] (multiple)
Logging in to [iface: default, target: iqn.2010-06.com.purestorage:flasharray.516cdd52f827bd21, portal: 10.10.1.94,3260] (multiple)
Login to [iface: default, target: iqn.2010-06.com.purestorage:flasharray.516cdd52f827bd21, portal: 10.10.1.93,3260] successful.
Login to [iface: default, target: iqn.2010-06.com.purestorage:flasharray.516cdd52f827bd21, portal: 10.10.1.94,3260] successful.
root@:/home/postgres/ [] iscsiadm -m session -o show
tcp: [13] 10.10.1.93:3260,1 iqn.2010-06.com.purestorage:flasharray.516cdd52f827bd21 (non-flash)
tcp: [14] 10.10.1.94:3260,1 iqn.2010-06.com.purestorage:flasharray.516cdd52f827bd21 (non-flash)

The new device is available (sdb) from now on:

root@:/home/postgres/ [] ls -la /dev/sd*
brw-rw----. 1 root disk 8,  0 May  4 13:23 /dev/sda
brw-rw----. 1 root disk 8,  1 May  4 13:23 /dev/sda1
brw-rw----. 1 root disk 8,  2 May  4 13:23 /dev/sda2
brw-rw----. 1 root disk 8, 16 May  4 13:23 /dev/sdb
brw-rw----. 1 root disk 8, 32 May  4 13:23 /dev/sdc

LVM setup:

root@:/home/postgres/ [] pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created.
root@:/home/postgres/ [] vgcreate vgpure /dev/sdb
  Volume group "vgpure" successfully created
root@:/home/postgres/ [] lvcreate -L 450G -n lvpure vgpure
  Logical volume "lvpure" created.
root@:/home/postgres/ [] mkdir -p /u02/pgdata
root@:/home/postgres/ [] mkfs.xfs /dev/mapper/vgpure-lvpure
meta-data=/dev/mapper/vgpure-lvpure isize=512    agcount=4, agsize=29491200 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=117964800, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=57600, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

root@:/home/postgres/ [] echo "/dev/mapper/vgpure-lvpure  /u02/pgdata  xfs defaults,noatime 0 0" >> /etc/fstab
root@:/home/postgres/ [] mount -a
root@:/home/postgres/ [] df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/cl_pgpurestorage-root   26G  2.0G   25G   8% /
devtmpfs                           3.9G     0  3.9G   0% /dev
tmpfs                              3.9G     0  3.9G   0% /dev/shm
tmpfs                              3.9G  8.5M  3.9G   1% /run
tmpfs                              3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1                         1014M  183M  832M  19% /boot
tmpfs                              781M     0  781M   0% /run/user/1000
/dev/mapper/vgpure-lvpure          450G   33M  450G   1% /u02/pgdata
root@:/home/postgres/ [] chown postgres:postgres /u02/pgdata

Initialized the PostgreSQL cluster:

postgres@pgpurestorage:/home/postgres/ [pg10] initdb -D /u02/pgdata/
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locales
  COLLATE:  en_US.UTF-8
  CTYPE:    en_US.UTF-8
  MESSAGES: en_US.UTF-8
  MONETARY: de_CH.UTF-8
  NUMERIC:  de_CH.UTF-8
  TIME:     en_US.UTF-8
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /u02/pgdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    pg_ctl -D /u02/pgdata/ -l logfile start

What we changed from the default configuration is:

postgres@pgpurestorage:/u02/pgdata/pgpure/ [pgpure] cat postgresql.auto.conf
# Do not edit this file manually!
# It will be overwritten by the ALTER SYSTEM command.
listen_addresses = '*'
logging_collector = 'on'
log_truncate_on_rotation = 'on'
log_filename = 'postgresql-%a.log'
log_rotation_age = '8d'
log_line_prefix = '%m - %l - %p - %h - %u@%d '
log_directory = 'pg_log'
log_min_messages = 'WARNING'
log_autovacuum_min_duration = '360s'
log_min_error_statement = 'error'
log_min_duration_statement = '5min'
log_checkpoints = 'on'
log_statement = 'ddl'
log_lock_waits = 'on'
log_temp_files = '1'
log_timezone = 'Europe/Zurich'
client_min_messages = 'WARNING'
wal_level = 'replica'
hot_standby_feedback = 'on'
max_wal_senders = '10'
cluster_name = 'pgpure'
max_replication_slots = '10'
shared_buffers=2048MB
work_mem=128MB
effective_cache_size=6144MB
maintenance_work_mem=512MB
max_wal_size=10GB

Calculating the minimum required amount of huge pages for the PostgreSQL instance:

postgres@pgpurestorage:/u02/pgdata/pgpure/ [pgpure] head -1 $PGDATA/postmaster.pid
3662
postgres@pgpurestorage:/u02/pgdata/pgpure/ [pgpure] grep ^VmPeak /proc/3662//status
VmPeak:	 2415832 kB
postgres@pgpurestorage:/u02/pgdata/pgpure/ [pgpure] echo "2415832/2048" | bc
1179

Set it slightly higher:

postgres@pgpurestorage:/u02/pgdata/pgpure/ [pgpure] sudo sed -i 's/vm.nr_hugepages=1024/vm.nr_hugepages=1200/g' /usr/lib/tuned/dbi-postgres/tuned.conf
postgres@pgpurestorage:/u02/pgdata/pgpure/ [pgpure] sudo tuned-adm profile dbi-postgres
postgres@pgpurestorage:/u02/pgdata/pgpure/ [pgpure] cat /proc/meminfo | grep Huge
AnonHugePages:      6144 kB
HugePages_Total:    1200
HugePages_Free:     1200
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

To disable transparent huge pages we created a file called “disable-thp.service” (from here):

postgres@pgpurestorage:/u02/pgdata/pgpure/ [pgpure] cat /etc/systemd/system/disable-thp.service
# Disable transparent huge pages
# put this file under:
#   /etc/systemd/system/disable-thp.service
# Then:
#   sudo systemctl daemon-reload
#   sudo systemctl start disable-thp
#   sudo systemctl enable disable-thp
[Unit]
Description=Disable Transparent Huge Pages (THP)

[Service]
Type=simple
ExecStart=/bin/sh -c "echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled && echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag"

[Install]
WantedBy=multi-user.target

Then reload the systemd daemon and start and enable the service:

postgres@pgpurestorage:/u02/pgdata/pgpure/ [pgpure] sudo systemctl daemon-reload
postgres@pgpurestorage:/u02/pgdata/pgpure/ [pgpure] sudo systemctl start disable-thp
postgres@pgpurestorage:/u02/pgdata/pgpure/ [pgpure] sudo systemctl enable disable-thp

To verify:

postgres@pgpurestorage:/u02/pgdata/pgpure/ [pgpure] cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
postgres@pgpurestorage:/u02/pgdata/pgpure/ [pgpure] cat /sys/kernel/mm/transparent_hugepage/defrag
always madvise [never]

For being sure that PostgreSQL really will use the huge pages set huge_pages to ‘on’ as this will prevent PostgreSQL from starting when the required pages can not be allocated:

pgpurestorage/postgres MASTER (postgres@5432) # alter system set huge_pages='on';
ALTER SYSTEM
Time: 2.417 ms

… and then restart the instance. When all is fine PostgreSQL will come up.

Finally to close this setup post here are some screenshots of the Pure Storage Management Web Console. The first one shows the “Storage” tab where you can see that the volume “volpgtest” is mapped to my host “pgpurestorage”.
purestorage-setup-storage-1

The name you give the server is not important. The important information is the mapping of the “Host Port” which you can see here (this is the iSCSI IQN):

purestorage-setup-storage-2

Once your server is connected you can see it in the connection map of the server in the console:
purestorage-connection-map

System health:
purestorage-system-health

Last, but not least, here is the dashboard:
purestorage-dashboard

Not much traffic right now but we’ll be changing that in the next post.

 

Cet article PostgreSQL on a Pure Storage All Flash Array – Preparations est apparu en premier sur Blog dbi services.

Testing new PostgreSQL features before alpha/beta/rc releases

$
0
0

A long time ago I blogged on how you can use the PostgreSQL development snapshots to test new PostgreSQL features before alpha/beta/rc releases are officially released. Another way to do this is to use git to get the latest sources and build PostgreSQL from there. Everything which was committed will be available to test. Btw: A great way to stay up to date is to subscribe to the mailing list just referenced. You’ll get a mail for each commit that happened, maybe one of those is getting your attention?

To start you’ll obviously need git. For distributions using yum this is just a matter of:

postgres@pgbox:/home/postgres/ [pg960final] sudo yum install git

For systems using apt use:

postgres@pgbox:/home/postgres/ [pg960final] sudo apt-get install git

Depending on how you want to configure PostgreSQL you’ll need some development packages as well. For yum based systems this is a good starting point:

postgres@pgbox:/home/postgres/ [pg960final] sudo yum install -y gcc openldap-devel python-devel readline-devel redhat-lsb bison flex perl-ExtUtils-Embed zlib-devel crypto-utils openssl-devel pam-devel libxml2-devel libxslt-devel tcl tcl-devel openssh-clients bzip2 net-tools wget screen ksh unzip

For apt based systems you might want to start with this:

postgres@pgbox:/home/postgres/ [pg960final] sudo apt-get install libldap2-dev libpython-dev libreadline-dev libssl-dev bison flex libghc-zlib-dev libcrypto++-dev libxml2-dev libxslt1-dev tcl tclcl-dev bzip2 wget screen ksh libpam0g-dev libperl-dev make unzip libpam0g-dev tcl-dev python

Not all of those packages are required, they just reflect what we usually install before building PostgreSQL from source. Of course you should adjust this and remove packages that are not required for what you plan to do.

How do you then get the latest PostgreSQL sources? Quite easy, it is documented in the PostgreSQL wiki:

postgres@pgbox:/home/postgres/ [pg960final] mkdir IwantToTest
postgres@pgbox:/home/postgres/ [pg960final] cd IwantToTest/
postgres@pgbox:/home/postgres/IwantToTest/ [pg960final] git clone git://git.postgresql.org/git/postgresql.git

The result should look similar to this:

Cloning into 'postgresql'...
remote: Counting objects: 629074, done.
remote: Compressing objects: 100% (95148/95148), done.
remote: Total 629074 (delta 534080), reused 626282 (delta 531478)
Receiving objects: 100% (629074/629074), 184.31 MiB | 26.40 MiB/s, done.
Resolving deltas: 100% (534080/534080), done.

From now on you have the complete PostgreSQL sources locally available.

postgres@pgbox:/home/postgres/IwantToTest/ [pg960final] cd postgresql/; ls
aclocal.m4  config  configure  configure.in  contrib  COPYRIGHT  doc  GNUmakefile.in  HISTORY  Makefile  README  README.git  src

Ready to test? Yes, but what? One possible way to start is asking git for what was committed recently:

postgres@pgbox:/home/postgres/IwantToTest/postgresql/ [pg960final] git log
commit 0de791ed760614991e7cb8a78fddd6874ea6919d
Author: Peter Eisentraut peter_e@gmx.net
Date:   Wed May 3 21:25:01 2017 -0400

    Fix cursor_to_xml in tableforest false mode

    It only produced  elements but no wrapping table element.

    By contrast, cursor_to_xmlschema produced a schema that is now correct
    but did not previously match the XML data produced by cursor_to_xml.

    In passing, also fix a minor misunderstanding about moving cursors in
    the tests related to this.

    Reported-by: filip@jirsak.org
    Based-on-patch-by: Thomas Munro thomas.munro@enterprisedb.com
...

Usually you can find a link to the discussion in the commit message so can you read through the history of a specific commit. Another way is to read the development documentation or the upcoming release notes once available.

All you need to do then is to build PostgreSQL:

postgres@pgbox:/home/postgres/IwantToTest/postgresql/ [pg960final] ./configure
postgres@pgbox:/home/postgres/IwantToTest/postgresql/ [pg960final] make all
postgres@pgbox:/home/postgres/IwantToTest/postgresql/ [pg960final] sudo make install
postgres@pgbox:/home/postgres/IwantToTest/postgresql/ [pg960final] cd contrib
postgres@pgbox:/home/postgres/IwantToTest/postgresql/contrib/ [pg960final] make all
postgres@pgbox:/home/postgres/IwantToTest/postgresql/contrib/ [pg960final] sudo make install
postgres@pgbox:/home/postgres/IwantToTest/postgresql/contrib/ [pg960final] /usr/local/pgsql/bin/initdb -D /var/tmp/test
postgres@pgbox:/home/postgres/IwantToTest/postgresql/contrib/ [pg960final] /usr/local/pgsql/bin/pg_ctl -D /var/tmp/test start
postgres@pgbox:/home/postgres/IwantToTest/postgresql/contrib/ [pg960final] /usr/local/pgsql/bin/psql postgres
psql (10devel)
Type "help" for help.

pgbox/postgres MASTER (postgres@5432) #

Happy testing …

 

Cet article Testing new PostgreSQL features before alpha/beta/rc releases est apparu en premier sur Blog dbi services.

PostgreSQL on a Pure Storage All Flash Array – Populating PostgreSQL

$
0
0

In the last post we explained on how we did the setup for connecting a PostgreSQL server to the Pure Storage array. In this post we’ll actually give the array something to work with. What we use for the tests is pgbench which is included with PostgreSQL and can be used to load a database and then perform a TPC-B like performance test against the loaded data. Btw: pgbench is a great tool when you want to make the same tests against different configurations of PostgreSQL or the same configurations of PostgreSQL against different physical or virtual hardware configurations.

To begin with lets create a dedicated database for loading the data into:

postgres@pgpurestorage:/home/postgres/ [pgpure] psql -c "create database purestorage" postgres
CREATE DATABASE
Time: 552.693 ms

We’ll use pgbench in initialization (“-i”) mode with a scale factor (“-s”) of 10000 for populating the database. The will create the pgbench_accounts, pgbench_branches, pgbench_history, and pgbench_tellers and will result in 1000000000 rows in total for the pgbench_accounts table.

postgres@pgpurestorage:/home/postgres/ [pgpure] pgbench -i -s 10000 purestorage

The total run-time (including the vacuum at the end) was around 2,5 hours (this is including the index creation and the vacuum at the end) and this is the size of the database:

pgpurestorage/postgres MASTER (postgres@5432) # \l+
                                                                      List of databases
┌─────────────┬──────────┬──────────┬─────────────┬─────────────┬───────────────────────┬─────────┬────────────┬────────────────────────
│    Name     │  Owner   │ Encoding │   Collate   │    Ctype    │   Access privileges   │  Size   │ Tablespace │                Descript
├─────────────┼──────────┼──────────┼─────────────┼─────────────┼───────────────────────┼─────────┼────────────┼────────────────────────
│ postgres    │ postgres │ UTF8     │ en_US.UTF-8 │ en_US.UTF-8 │                       │ 7629 kB │ pg_default │ default administrative
│ purestorage │ postgres │ UTF8     │ en_US.UTF-8 │ en_US.UTF-8 │                       │ 146 GB  │ pg_default │
│ template0   │ postgres │ UTF8     │ en_US.UTF-8 │ en_US.UTF-8 │ =c/postgres          ↵│ 7497 kB │ pg_default │ unmodifiable empty data
│             │          │          │             │             │ postgres=CTc/postgres │         │            │
│ template1   │ postgres │ UTF8     │ en_US.UTF-8 │ en_US.UTF-8 │ =c/postgres          ↵│ 7497 kB │ pg_default │ default template for ne
│             │          │          │             │             │ postgres=CTc/postgres │         │            │
└─────────────┴──────────┴──────────┴─────────────┴─────────────┴───────────────────────┴─────────┴────────────┴────────────────────────
(4 rows)

The biggest table is the pgbench_accounts table:

pgpurestorage/purestorage MASTER (postgres@5432) # \d+
                           List of relations
┌────────┬──────────────────┬───────┬──────────┬─────────┬─────────────┐
│ Schema │       Name       │ Type  │  Owner   │  Size   │ Description │
├────────┼──────────────────┼───────┼──────────┼─────────┼─────────────┤
│ public │ pgbench_accounts │ table │ postgres │ 125 GB  │             │
│ public │ pgbench_branches │ table │ postgres │ 392 kB  │             │
│ public │ pgbench_history  │ table │ postgres │ 0 bytes │             │
│ public │ pgbench_tellers  │ table │ postgres │ 4360 kB │             │
└────────┴──────────────────┴───────┴──────────┴─────────┴─────────────┘
(4 rows)

.. which contains 1000000000 rows:

pgpurestorage/purestorage MASTER (postgres@5432) # select count(*) from pgbench_accounts;
┌────────────┐
│   count    │
├────────────┤
│ 1000000000 │
└────────────┘
(1 row)

Time: 219538.051 ms (03:39.538)

Here are the screenshots from the Pure Storage management console for the duration of the run (which started some minutes before 20:00). Orange are write operations, blue is read:

purestorage-pg-load-1
purestorage-pg-load-2

As you can see we are fare away from saturating the storage. So, lets do the same load again but this time with two sessions into two databases in parallel:

# session one
postgres@pgpurestorage:/home/postgres/ [pgpure] psql -c "drop database if exists purestorage" postgres
DROP DATABASE
Time: 983.297 ms
postgres@pgpurestorage:/home/postgres/ [pgpure] psql -c "create database purestorage" postgres
CREATE DATABASE
Time: 514.140 ms
postgres@pgpurestorage:/home/postgres/ [pgpure] pgbench -i -s 10000 purestorage

# session two
postgres@pgpurestorage:/home/postgres/ [pgpure] psql -c "drop database if exists purestorage2" postgres
DROP DATABASE
Time: 0.110 ms
postgres@pgpurestorage:/home/postgres/ [pgpure] psql -c "create database purestorage2" postgres
CREATE DATABASE
Time: 274.576 ms
postgres@pgpurestorage:/home/postgres/ [pgpure] pgbench -i -s 10000 purestorage2

This screenshots from the Pure Storage console:

purestorage-pg-load-2-1
purestorage-pg-load-2-2

It took more than double as long to load the 2000000000 rows but the limit is not on the storage. The average bandwidth went up from around 65 mb/sec to around 90 mb/sec. What is interesting to see is that we need only around 45GB real storage:

purestorage-pg-load-2-reduction

This means we have a almost 9 times compression/de-duplication on the storage layer. Remember that each database has a size of 146GB ( ( 146 * 2 = 292 ) / 9 = 32, plus the WAL files ):

                                                                     List of databases
┌──────────────┬──────────┬──────────┬─────────────┬─────────────┬───────────────────────┬─────────┬────────────┬──────────────────────────────────────
│     Name     │  Owner   │ Encoding │   Collate   │    Ctype    │   Access privileges   │  Size   │ Tablespace │                Description
├──────────────┼──────────┼──────────┼─────────────┼─────────────┼───────────────────────┼─────────┼────────────┼──────────────────────────────────────
│ postgres     │ postgres │ UTF8     │ en_US.UTF-8 │ en_US.UTF-8 │                       │ 7629 kB │ pg_default │ default administrative connection dat
│ purestorage  │ postgres │ UTF8     │ en_US.UTF-8 │ en_US.UTF-8 │                       │ 146 GB  │ pg_default │
│ purestorage2 │ postgres │ UTF8     │ en_US.UTF-8 │ en_US.UTF-8 │                       │ 146 GB  │ pg_default │
│ template0    │ postgres │ UTF8     │ en_US.UTF-8 │ en_US.UTF-8 │ =c/postgres          ↵│ 7497 kB │ pg_default │ unmodifiable empty database
│              │          │          │             │             │ postgres=CTc/postgres │         │            │
│ template1    │ postgres │ UTF8     │ en_US.UTF-8 │ en_US.UTF-8 │ =c/postgres          ↵│ 7497 kB │ pg_default │ default template for new databases
│              │          │          │             │             │ postgres=CTc/postgres │         │            │
└──────────────┴──────────┴──────────┴─────────────┴─────────────┴───────────────────────┴─────────┴────────────┴──────────────────────────────────────

What happens then when the file system gets full on the host but there still is storage available in the back-end? Lets copy one of the test databases: That should almost fill the file system:

pgpurestorage/postgres MASTER (postgres@5432) # \! df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/cl_pgpurestorage-root   26G  2.2G   24G   9% /
devtmpfs                           3.9G     0  3.9G   0% /dev
tmpfs                              3.9G   16K  3.9G   1% /dev/shm
tmpfs                              3.9G   41M  3.8G   2% /run
tmpfs                              3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1                         1014M  183M  832M  19% /boot
tmpfs                              781M     0  781M   0% /run/user/1000
/dev/mapper/vgpure-lvpure          450G  302G  149G  68% /u02/pgdata
pgpurestorage/postgres MASTER (postgres@5432) # create database test template = purestorage;

CREATE DATABASE
Time: 3312062.975 ms (55:12.063)
pgpurestorage/postgres MASTER (postgres@5432) #
pgpurestorage/postgres MASTER (postgres@5432) # \! df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/cl_pgpurestorage-root   26G  2.2G   24G   9% /
devtmpfs                           3.9G     0  3.9G   0% /dev
tmpfs                              3.9G   16K  3.9G   1% /dev/shm
tmpfs                              3.9G   41M  3.8G   2% /run
tmpfs                              3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1                         1014M  183M  832M  19% /boot
tmpfs                              781M     0  781M   0% /run/user/1000
/dev/mapper/vgpure-lvpure          450G  449G  1.7G 100% /u02/pgdata
pgpurestorage/postgres MASTER (postgres@5432) #

Almost full, so lets create another database:

pgpurestorage/postgres MASTER (postgres@5432) # create database test2 template = purestorage;
ERROR:  could not write to file "base/16562/16534": No space left on device
Time: 34449.626 ms (00:34.450)

So, behavior is what has been expected.

In the next post we’ll do some pgbench standard benchmarks against the data set although the numbers will probably not be very good as we are running in a ESX test environment and this seems to be the bottle neck in our case. We’ll have to analyze the sar statistics for that, but no time yet.

What I can say about the storage system so far: I like the simplicity of the management console. There is not much more you can do than creating hosts, volumes and connections between them, that’s it. Reduction is happening automatically and you have the option to replicate to another storage system. Snapshots are available as well, but not yet tested.

 

Cet article PostgreSQL on a Pure Storage All Flash Array – Populating PostgreSQL est apparu en premier sur Blog dbi services.

Replicating from Oracle to PostgreSQL with EDB Replication Server – 1 – A simple replication setup

$
0
0

When you want to offload reporting stuff from Oracle to PostgreSQL there are several solutions you can go for. There most widely known probably are Oracle GoldenGate and SharePlex. EnterpriseDB has its own solution for this which is called EDB Replication Server. As soon as you have one subscription from EnterpriseDB this tool is included and can be used to offload from Oracle to PostgreSQL. Lets do a simple replication setup and see how it works.

For this little demo I have to boxes: One running Oracle (12.1.0.2) and the EDB Replication Server publisher and the other one running PostgreSQL 9.6.2 and the EDB Replication server subscriber and the console.

The installation is quite simple: Download the software, unzip and execute as root:

[root@pgxdb ~]$ cd /var/tmp/
[root@pgxdb tmp]$ ls -la xdbreplicationserver-6.1.0-1-linux-x64.run 
-rw-r--r--. 1 root root 26934145 May 19 09:01 xdbreplicationserver-6.1.0-1-linux-x64.run
[root@pgxdb tmp]$ chmod +x xdbreplicationserver-6.1.0-1-linux-x64.run 
[root@pgxdb tmp]$ ./xdbreplicationserver-6.1.0-1-linux-x64.run 

These are the screenshots for the PostgreSQL box:
xdb_1
xdb_2
xdb_3
xdb_4
xdb_5
xdb_6
xdb_7
xdb_8
xdb_9
xdb_10
xdb_11
xdb_12
xdb_13

Once the installation completed you should be able to see what was installed:

[root@pgxdb tmp]$ su - postgres
Last failed login: Sun May 15 06:35:25 CEST 2016 from 192.168.22.1 on ssh:notty
There were 6 failed login attempts since the last successful login.
-bash-4.2$ ls -la
total 7928
drwxr-xr-x. 7 root root    4096 May 19 09:15 .
drwxr-xr-x. 3 root root      46 May 19 09:15 ..
drwxrwxr-x. 2 root root    4096 May 19 09:15 bin
drwxrwxr-x. 3 root root      78 May 19 09:15 etc
drwxrwxr-x. 4 root root      43 May 19 09:15 installer
drwxrwxr-x. 5 root root    4096 May 19 09:15 lib
drwxrwxr-x. 4 root root      44 May 19 09:15 scripts
-rwx------. 1 root root 7967605 May 19 09:15 uninstall-xdbreplicationserver
-rw-------. 1 root root   26455 May 19 09:15 uninstall-xdbreplicationserver.dat
-r--r--r--. 1 root root   64033 Feb 27 12:31 xdb_3rd_party_licenses.txt
-r--r--r--. 1 root root   15216 Feb 27 12:30 xdb_license.txt

Two new processes are running, which are basically the subscriber:

-bash-4.2$ ps -ef | grep postgres
postgres 13019     1  0 09:15 ?        00:00:00 /bin/bash -c cd /opt/PostgreSQL/EnterpriseDB-xDBReplicationServer/bin; ./runSubServer.sh  >> /var/log/edb/xdbsubserver/edb-xdbsubserver.log 2>&1 &
postgres 13044 13019  0 09:15 ?        00:00:00 /usr/bin/java -XX:-UsePerfData -XX:ErrorFile=/var/log/xdb-6.1/subserver_pid_%p.log -Djava.awt.headless=true -jar /opt/PostgreSQL/EnterpriseDB-xDBReplicationServer/bin/edb-repserver.jar subserver 9052

For the Oracle box I did exactly the same except for this:

xdb_14
xdb_15

Same picture, two new processes which are the publisher:

oracle   29542     1  0 09:48 ?        00:00:00 /bin/bash -c cd /opt/PostgreSQL/EnterpriseDB-xDBReplicationServer/bin; ./runPubServer.sh  >> /var/log/edb/xdbpubserver/edb-xdbpubserver.log 2>&1 &
oracle   29567 29542  0 09:48 ?        00:00:00 /usr/bin/java -XX:-UsePerfData -Xms256m -Xmx1536m -XX:ErrorFile=/var/log/xdb-6.1/pubserver_pid_%p.log -Djava.library.path=/opt/PostgreSQL/EnterpriseDB-xDBReplicationServer/bin -Djava.awt.headless=true -jar /opt/PostgreSQL/EnterpriseDB-xDBReplicationServer/bin/edb-repserver.jar pubserver 9051

Lets start the console to bring up the graphical user interface:

-bash-4.2$ bin/runRepConsole.sh

Register a new subscription server (PostgreSQL box):

xdb_16
xdb_17

Register a new publication server (Oracle box):

xdb_18
xdb_19

Before proceeding from here you should download the Oracle JDBC driver and put it to both boxes:

-bash-4.2$ pwd
/opt/PostgreSQL/EnterpriseDB-xDBReplicationServer/lib/jdbc
-bash-4.2$ ls -l ojdbc*.jar
-rw-r--r--. 1 root root 3698857 May 19 10:45 ojdbc7.jar

Remember to restart the EDB Replication Server services:

sudo systemctl restart edb-xdbpubserver.service
sudo systemctl restart edb-xdbsubserver.service

After that add the Oracle database (I am using the scott/tiger sample schema here):
xdb_20
xdb_21
xdb_22

Ok, easy to fix:

grant connect,resource,create trigger to scott;
Grant succeeded.

Once again:
xdb_23

The next thing is to create the publication itself:
xdb_24
xdb_25

xdb_26

Ok, lets fix it:

alter table scott.bonus add constraint bonus_pk primary key ( ename,job);
Table altered.

xdb_27

Ok, lets fix it:

alter table scott.salgrade add constraint salgrade_pk primary key (grade,losal,hisal);
Table altered.

xdb_28
xdb_29

Done.

Lets create the subscription, but before doing that make sure that you put the details of the controller database (The Oracle one in my case) to the EDB Replication Server configuration file:

postgres@pgxdb:/opt/PostgreSQL/EnterpriseDB-xDBReplicationServer/ [xdbrepl] cat /etc/edb-repl.conf
admin_password=VbjDPk5zwu/naXI2fpEbBQ\=\=
user=scott
port=1521
password=H4rzJCNHH/Q\=
type=oracle
admin_user=admin
host=192.168.22.242
database=servicename\=DB4.it.dbi-services.com

Proceed with creating the subscription database:
xdb_30
xdb_31
xdb_32
xdb_33
xdb_34

All the table are there now in the PostgreSQL instance but empty:

pgxdb/postgres MASTER (postgres@5432) # \d SCOTT.*
Table "scott.bonus"
┌────────┬───────────────────────┬───────────┐
│ Column │ Type │ Modifiers │
├────────┼───────────────────────┼───────────┤
│ ename │ character varying(10) │ not null │
│ job │ character varying(9) │ not null │
│ sal │ numeric │ │
│ comm │ numeric │ │
└────────┴───────────────────────┴───────────┘
Indexes:
"bonus_pk" PRIMARY KEY, btree (ename, job)

Index "scott.bonus_pk"
┌────────┬───────────────────────┬────────────┐
│ Column │ Type │ Definition │
├────────┼───────────────────────┼────────────┤
│ ename │ character varying(10) │ ename │
│ job │ character varying(9) │ job │
└────────┴───────────────────────┴────────────┘
primary key, btree, for table "scott.bonus"

Table "scott.dept"
┌────────┬───────────────────────┬───────────┐
│ Column │ Type │ Modifiers │
├────────┼───────────────────────┼───────────┤
│ deptno │ numeric(2,0) │ not null │
│ dname │ character varying(14) │ │
│ loc │ character varying(13) │ │
└────────┴───────────────────────┴───────────┘
Indexes:
"pk_dept" PRIMARY KEY, btree (deptno)

Table "scott.emp"
┌──────────┬───────────────────────┬───────────┐
│ Column │ Type │ Modifiers │
├──────────┼───────────────────────┼───────────┤
│ empno │ numeric(4,0) │ not null │
│ ename │ character varying(10) │ │
│ job │ character varying(9) │ │
│ mgr │ numeric(4,0) │ │
│ hiredate │ date │ │
│ sal │ numeric(7,2) │ │
│ comm │ numeric(7,2) │ │
│ deptno │ numeric(2,0) │ │
└──────────┴───────────────────────┴───────────┘
Indexes:
"pk_emp" PRIMARY KEY, btree (empno)

Index "scott.pk_dept"
┌────────┬──────────────┬────────────┐
│ Column │ Type │ Definition │
├────────┼──────────────┼────────────┤
│ deptno │ numeric(2,0) │ deptno │
└────────┴──────────────┴────────────┘
primary key, btree, for table "scott.dept"

Index "scott.pk_emp"
┌────────┬──────────────┬────────────┐
│ Column │ Type │ Definition │
├────────┼──────────────┼────────────┤
│ empno │ numeric(4,0) │ empno │
└────────┴──────────────┴────────────┘
primary key, btree, for table "scott.emp"

Table "scott.salgrade"
┌────────┬─────────┬───────────┐
│ Column │ Type │ Modifiers │
├────────┼─────────┼───────────┤
│ grade │ numeric │ not null │
│ losal │ numeric │ not null │
│ hisal │ numeric │ not null │
└────────┴─────────┴───────────┘
Indexes:
"salgrade_pk" PRIMARY KEY, btree (grade, losal, hisal)

Index "scott.salgrade_pk"
┌────────┬─────────┬────────────┐
│ Column │ Type │ Definition │
├────────┼─────────┼────────────┤
│ grade │ numeric │ grade │
│ losal │ numeric │ losal │
│ hisal │ numeric │ hisal │
└────────┴─────────┴────────────┘
primary key, btree, for table "scott.salgrade"

pgxdb/postgres MASTER (postgres@5432) # select count(*) from SCOTT.salgrade;
┌───────┐
│ count │
├───────┤
│ 0 │
└───────┘
(1 row)

Time: 0.443 ms
pgxdb/postgres MASTER (postgres@5432) # select count(*) from SCOTT.depts;
ERROR: relation "scott.depts" does not exist
LINE 1: select count(*) from SCOTT.depts;
^
Time: 0.554 ms
pgxdb/postgres MASTER (postgres@5432) # select count(*) from SCOTT.dept;
┌───────┐
│ count │
├───────┤
│ 0 │
└───────┘
(1 row)

Time: 0.298 ms

To do the initial synchronization do a Snapshot:

xdb_35
xdb_36
xdb_37

All the data is there:

pgxdb/postgres MASTER (postgres@5432) # select * from SCOTT.dept;
┌────────┬────────────┬──────────┐
│ deptno │ dname │ loc │
├────────┼────────────┼──────────┤
│ 10 │ ACCOUNTING │ NEW YORK │
│ 20 │ RESEARCH │ DALLAS │
│ 30 │ SALES │ CHICAGO │
│ 40 │ OPERATIONS │ BOSTON │
└────────┴────────────┴──────────┘
(4 rows)

Time: 0.435 ms
pgxdb/postgres MASTER (postgres@5432) # select * from SCOTT.emp;
┌───────┬────────┬───────────┬──────┬────────────┬─────────┬─────────┬────────┐
│ empno │ ename │ job │ mgr │ hiredate │ sal │ comm │ deptno │
├───────┼────────┼───────────┼──────┼────────────┼─────────┼─────────┼────────┤
│ 7369 │ SMITH │ CLERK │ 7902 │ 1980-12-17 │ 800.00 │ NULL │ 20 │
│ 7499 │ ALLEN │ SALESMAN │ 7698 │ 1981-02-20 │ 1600.00 │ 300.00 │ 30 │
│ 7521 │ WARD │ SALESMAN │ 7698 │ 1981-02-22 │ 1250.00 │ 500.00 │ 30 │
│ 7566 │ JONES │ MANAGER │ 7839 │ 1981-04-02 │ 2975.00 │ NULL │ 20 │
│ 7654 │ MARTIN │ SALESMAN │ 7698 │ 1981-09-28 │ 1250.00 │ 1400.00 │ 30 │
│ 7698 │ BLAKE │ MANAGER │ 7839 │ 1981-05-01 │ 2850.00 │ NULL │ 30 │
│ 7782 │ CLARK │ MANAGER │ 7839 │ 1981-06-09 │ 2450.00 │ NULL │ 10 │
│ 7788 │ SCOTT │ ANALYST │ 7566 │ 1987-04-19 │ 3000.00 │ NULL │ 20 │
│ 7839 │ KING │ PRESIDENT │ NULL │ 1981-11-17 │ 5000.00 │ NULL │ 10 │
│ 7844 │ TURNER │ SALESMAN │ 7698 │ 1981-09-08 │ 1500.00 │ 0.00 │ 30 │
│ 7876 │ ADAMS │ CLERK │ 7788 │ 1987-05-23 │ 1100.00 │ NULL │ 20 │
│ 7900 │ JAMES │ CLERK │ 7698 │ 1981-12-03 │ 950.00 │ NULL │ 30 │
│ 7902 │ FORD │ ANALYST │ 7566 │ 1981-12-03 │ 3000.00 │ NULL │ 20 │
│ 7934 │ MILLER │ CLERK │ 7782 │ 1982-01-23 │ 1300.00 │ NULL │ 10 │
└───────┴────────┴───────────┴──────┴────────────┴─────────┴─────────┴────────┘
(14 rows)

To synchronize changes:
xdb_38

xdb_39

To schedule synchronization:

xdb_40
xdb_41
xdb_42
xdb_43

Test it, in Oracle:

insert into scott.dept values ( 50,'My_Dept','Home');
1 row created.
commit;
Commit complete.

PostgreSQL:

pgxdb/postgres MASTER (postgres@5432) # select * from SCOTT.dept;
┌────────┬────────────┬──────────┐
│ deptno │ dname │ loc │
├────────┼────────────┼──────────┤
│ 10 │ ACCOUNTING │ NEW YORK │
│ 20 │ RESEARCH │ DALLAS │
│ 30 │ SALES │ CHICAGO │
│ 40 │ OPERATIONS │ BOSTON │
│ 50 │ My_Dept │ Home │
└────────┴────────────┴──────────┘
(5 rows)

Quite easy to setup and to do an initial simple replication. In a future post we’ll look at more specific use cases. Hope this helps to get you started.

 

Cet article Replicating from Oracle to PostgreSQL with EDB Replication Server – 1 – A simple replication setup est apparu en premier sur Blog dbi services.

History of Upgrading 9 Tb PostgreSQL database

$
0
0

In this blog I am going share a history of PostgreSQL migration and upgrade from 9.2 to 9.6. Let me first explain the context
We have a PostgreSQL environment with following characteristics. Note that real database name, server name are changed for security reason
Host: CentOS release 6.4
PostgreSQL version: 9.2
Database size : 9Tb


postgres=# select version();
version
---------------------------------------------------------------------------------------------------
------------
PostgreSQL 9.2.4 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-
52), 64-bit
(1 row)
postgres=#


postgres=# select pg_size_pretty(pg_database_size('zoulou'));
pg_size_pretty
----------------
9937 GB
(1 row)

The problem is that we were just on limit about space in the actual server and we plan to move the database in a new server and after to upgrade to 9.6 version.

The questions we had to answer were following:

Do we keep the same Linux version?
How do we transfer data to the new server? pgdump, , pg_basebackup, using cp to copy of datafiles…
What is the fastest way to upgrade 9T of data?

Finally we decide

To use Debian instead of Centos. Why, because the sysadmin prefers Debian
We can summarize the two environments by following picture.

schema

To transfer data to the new server server2 we decide to use rsync. The reason was that the export and the import will take a long time. So while database is running we launch following rsync command on server1.
rsync -av -P --bwlimit=30720 /opt/PostgreSQL/9.2/data/* server2:/u02/pgdata/zoulou/

The bandwidth was reduced because people are complaining about network. The rsync with database running takes 3 days to finish, yes 3 days. And few days after we ask a downtime to stop the cluster to rsync the delta. Indeed as you may know it’s necessary to have a coherent copy to be able to start the cluster. Note that the copy of the delta lasts 2 hours.

Now it’s time to install the 2 versions of PostgreSQL (9.2.4 and 9.6.2) on the new server. We will show here just the main steps.

With apt-get we install the required packages

apt-get install libldap2-dev libpython-dev libreadline-dev libssl-dev bison flex libghc-zlib-dev libcrypto++-dev libxml2-dev libxslt1-dev tcl tclcl-dev bzip2 wget screen ksh libpam0g-dev libperl-dev make unzip libpam0g-dev tcl-dev python

One first important thing is to use the same options than the source for PostgreSQL installation. We use the pg_config command in the source server to retrieve these options. The installation of 9.2 is described below (9.6.2 install is the same)

PGHOME=/u01/app/postgres/product/92/db_4
SEGSIZE=1
BLOCKSIZE=8
WALSEGSIZE=16


./configure --prefix=${PGHOME} \
--exec-prefix=${PGHOME} \
--bindir=${PGHOME}/bin \
--libdir=${PGHOME}/lib \
--sysconfdir=${PGHOME}/etc \
--includedir=${PGHOME}/include \
--datarootdir=${PGHOME}/share \
--datadir=${PGHOME}/share \
--with-pgport=5432 \
--with-perl \
--with-python \
--with-tcl \
--with-openssl \
--with-pam \
--with-ldap \
--with-libxml \
--with-libxslt \
--with-segsize=${SEGSIZE} \
--with-blocksize=${BLOCKSIZE} \
--with-wal-segsize=${WALSEGSIZE}


make world
make install
cd contrib/
make install
cd ../doc/
make install

Now that data are copied and PostgreSQL softwares installed, we can start on server2 the 9.2 cluster.

/u01/app/postgres/product/92/bin/pg_ctl -D /u02/pgdata/zoulou/

Note that the first attempt failed due to local variables. On the source the system is using en_US.UTF8 and on the new server the system is using ch_FR.UTF-8. So we change local values using this command
# dpkg-reconfigure locales

Once the cluster 9.2 started without errors we can now think about the upgrade. With a database size of 9T to upgrade, we decide to use the –link option. Indeed this option uses link instead of copying files to new cluster. This will definitively speedup the upgrade process. While speeding up the upgrade, with this method, if the upgrade fails you cannot restart the old cluster on the same server.

The first step for the upgrade is to initialize a new 9.6.2 cluster on the new server server2

/u01/app/postgres/product/96/db_2/bin/initdb --pgdata=/u02/pgdata/zoulou962/ --xlogdir=/u03/ZOULOU962/pg_xlog/ --pwprompt --auth=md5
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
Enter new superuser password:
Enter it again:
fixing permissions on existing directory /u02/pgdata/zoulou962 ... ok
fixing permissions on existing directory /u03/zoulou962/pg_xlog ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
/u01/app/postgres/product/96/db_2/bin/pg_ctl -D /u02/pgdata/zoulou962/ -l logfile start

We copy configuration files of the 9.2 cluster to the 9.6 cluster


mv postgresql.conf postgresql.conf_origin
mv pg_hba.conf pg_hba.conf_origin
cp ../zoulou962/postgresql.conf .
cp ../zoulou962/pg_hba.conf .

After we stop the 9.2 cluster on the new server

which pg_ctl
/u01/app/postgres/product/92/db_4/bin/pg_ctl
postgres@apsicpap01:~$ pg_ctl stop
waiting for server to shut down.... done
server stopped

Once the 9.2 cluster stopped we run the pg_upgrade command with the –c option.
The option –c checks clusters only, don’t change any data.


u01/app/postgres/product/96/db_2/bin/pg_upgrade -d /u02/pgdata/zoulou/ -D /u02/pgdata/zoulou962/ -b /u01/app/postgres/product/92/db_4/bin/ -B /u01/app/postgres/product/96/db_2/bin/ -c
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for reg* system OID user data types ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for roles starting with 'pg_' ok
Checking for invalid "line" user columns ok
Checking for presence of required libraries ok
Checking database user is the install user ok
Checking for prepared transactions ok
*Clusters are compatible*

And now we are ready to upgrade with the –link option. I was surprised how fast the upgrade was. Yes we upgrade 9T of database in less than 3 minutes. Incredible this –link option.

/u01/app/postgres/product/96/db_2/bin/pg_upgrade -d /u02/pgdata/zoulou/ -D /u02/pgdata/zoulou962/ -b /u01/app/postgres/product/92/db_4/bin/ -B /u01/app/postgres/product/96/db_2/bin/ --link
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for reg* system OID user data types ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for roles starting with 'pg_' ok
Checking for invalid "line" user columns ok
Creating dump of global objects ok
Creating dump of database schemas
.. ok
Checking for presence of required libraries ok
Checking database user is the install user ok
Checking for prepared transactions ok
..
If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.
..
Performing Upgrade
------------------
Analyzing all rows in the new cluster ok
Freezing all rows on the new cluster ok
Deleting files from new pg_clog ok
Copying old pg_clog to new server ok
Setting next transaction ID and epoch for new cluster ok
Deleting files from new pg_multixact/offsets ok
Setting oldest multixact ID on new cluster ok
Resetting WAL archives ok
Setting frozenxid and minmxid counters in new cluster ok
Restoring global objects in the new cluster ok
Restoring database schemas in the new cluster
.. ok
Setting minmxid counter in new cluster ok
Adding ".old" suffix to old global/pg_control ok
..
If you want to start the old cluster, you will need to remove
the ".old" suffix from /u02/pgdata/zoulou/global/pg_control.old.
Because "link" mode was used, the old cluster cannot be safely
started once the new cluster has been started.
..
Linking user relation files
.. ok
Setting next OID for new cluster ok
Sync data directory to disk ok
Creating script to analyze new cluster ok
Creating script to delete old cluster ok
..
Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade so,
once you start the new server, consider running:
./analyze_new_cluster.sh
..
Running this script will delete the old cluster's data files:
./delete_old_cluster.sh

We delete the old cluster
./delete_old_cluster.sh
And we start the new 9.6 cluster

pg_ctl start
2017-05-11 09:11:42 CEST LOG: redirecting log output to logging collector process
2017-05-11 09:11:42 CEST HINT: Future log output will appear in directory "/u99/zoulou962/pg_log".

We generate statistics

./analyze_new_cluster.sh
This script will generate minimal optimizer statistics rapidly
so your system is usable, and then gather statistics twice more
with increasing accuracy. When it is done, your system will
have the default level of optimizer statistics.
If you have used ALTER TABLE to modify the statistics target for
any tables, you might want to remove them and restore them after
running this script because they will delay fast statistics generation.
If you would like default statistics as quickly as possible, cancel
this script and run:
"/u01/app/postgres/product/96/db_2/bin/vacuumdb" --all --analyze-only
vacuumdb: processing database "zoulou": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "postgres": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "template1": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "zoulou": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "postgres": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "template1": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "zoulou": Generating default (full) optimizer statistics
vacuumdb: processing database "postgres": Generating default (full) optimizer statistics
vacuumdb: processing database "template1": Generating default (full) optimizer statistics
Done

And the last step is to verify size and db version

zoulou=# select version();
version
------------------------------------------------------------------------------------------
PostgreSQL 9.6.2 on x86_64-pc-linux-gnu, compiled by gcc (Debian 4.9.2-10) 4.9.2, 64-bit
(1 row)

zoulou=# select pg_size_pretty(pg_database_size('zoulou'));
pg_size_pretty
----------------
9937 GB
(1 row)
zoulou=#

Here is the end of the history. Hope that this can help for future migration and upgrade.

 

Cet article History of Upgrading 9 Tb PostgreSQL database est apparu en premier sur Blog dbi services.

A first look at EDB Postgres Enterprise Manager 7 beta

$
0
0

In case you missed it: EnterpriseDB has released the beta of Postgres Enterprise Manager 7 beta. When installation is as easy as for the current version that should just be a matter of clicking next, lets see.

Because the installer will create the systemd services installation should be done as root:

[root@edbpem tmp]$ ls -l
total 289076
-rw-r--r--. 1 root root 296009946 Jun  1 18:58 pem_server-7.0.0-beta1-1-linux-x64.run
[root@edbpem tmp]$ chmod +x pem_server-7.0.0-beta1-1-linux-x64.run 
[root@edbpem tmp]$ ./pem_server-7.0.0-beta1-1-linux-x64.run 

From now on all is graphical and straight forward:

pem7_1
pem7_2
pem7_3
pem7_4
pem7_5
pem7_6
pem7_7
pem7_8
pem7_9
pem7_10
pem7_11
pem7_12
pem7_13
pem7_14
pem7_15
pem7_16
pem7_17
pem7_18
pem7_19
pem7_20
pem7_21
pem7_22
pem7_23
pem7_24
pem7_25
pem7_26
pem7_27
pem7_28
pem7_29
pem7_30
pem7_31
pem7_32

What you get from a process perspective is this:

[root@edbpem tmp]$ ps -ef | grep pem
postgres 13462     1  0 19:17 ?        00:00:00 /u01/app/postgres/product/96/db_2/bin/postgres -D /u02/pgdata/pem
root     13869     1  0 19:18 ?        00:00:00 /u01/app/postgres/product/pem7/agent/bin/pemagent -c /u01/app/postgres/product/pem7/agent/etc/agent.cfg
root     13870 13869  0 19:18 ?        00:00:00 /u01/app/postgres/product/pem7/agent/bin/pemworker -c /u01/app/postgres/product/pem7/agent/etc/agent.cfg --pid 13869
postgres 13883 13462  1 19:18 ?        00:00:02 postgres: agent1 pem 127.0.0.1(53232) idle
postgres 13885 13462  0 19:18 ?        00:00:00 postgres: agent1 pem 127.0.0.1(53234) idle
postgres 13886 13462  0 19:18 ?        00:00:00 postgres: agent1 pem 127.0.0.1(53236) idle
postgres 13887 13462  0 19:18 ?        00:00:00 postgres: agent1 pem 127.0.0.1(53238) idle
postgres 13888 13462  0 19:18 ?        00:00:00 postgres: agent1 pem 127.0.0.1(53240) idle
pem      13938 13937  0 19:18 ?        00:00:00 EDBPEM                                                              -k start -f /u01/app/postgres/product/EnterpriseDB-ApacheHTTPD/apache/conf/httpd.conf
root     14301 11016  0 19:20 pts/0    00:00:00 grep --color=auto pem

Two new systemd services have been created so PEM should startup and shutdown automatically when the server reboots:

[root@edbpem tmp]# systemctl list-unit-files | egrep "pem|postgres"
pemagent.service                              enabled 
postgresql-9.6.service                        enabled 

Lets connect to PEM: https://192.168.22.248:8443/pem

pem7_33

If you have an EDB subscription now it is the time to enter the product key:

pem7_34

What you immediately can see is that the look and feel changed to that of pgadmin4 (A fat client for PEM as in the current version is not available any more):

pem7_35
pem7_36
pem7_37

In a next post we’ll add a PostgreSQL instance and start to monitor it.

 

Cet article A first look at EDB Postgres Enterprise Manager 7 beta est apparu en premier sur Blog dbi services.

A first look at EDB Postgres Enterprise Manager 7 beta – Connecting a PostgreSQL instance

$
0
0

In the last post we did a click/click/click setup of the PEM server. What we want to do now is to attach a PostgreSQL instance to the PEM server for being able to monitor and administer it. For that we need to install the PEM agent on a host where we have a PostgreSQL instance running (192.168.22.249 in my case, which runs PostgreSQL 10 Beta1). Lets go …

As usual, when you want to have the systemd services generated automatically you should run the installation as root:

[root@pemclient postgres]# ls
pem_agent-7.0.0-beta1-1-linux-x64.run
[root@pemclient postgres]# chmod +x pem_agent-7.0.0-beta1-1-linux-x64.run 
[root@pemclient postgres]# ./pem_agent-7.0.0-beta1-1-linux-x64.run 

The installation itself is not a big deal, just follow the screens:

pem_agent1
pem_agent2
pem_agent3
pem_agent4
pem_agent5
pem_agent6
pem_agent7
pem_agent8
pem_agent9

Once done we have a new systemd service:

[root@pemclient postgres]# systemctl list-unit-files | grep pem
pemagent.service                              enabled 

… and the processes that make up the PEM agent:

[root@pemclient postgres]# ps -ef | grep pem
root      3454     1  0 16:40 ?        00:00:00 /u01/app/postgres/product/pem7/agent/agent/bin/pemagent -c /u01/app/postgres/product/pem7/agent/agent/etc/agent.cfg
root      3455  3454  0 16:40 ?        00:00:00 /u01/app/postgres/product/pem7/agent/agent/bin/pemworker -c /u01/app/postgres/product/pem7/agent/agent/etc/agent.cfg --pid 3454
root      3515  2741  0 16:43 pts/0    00:00:00 grep --color=auto pem

Heading back to the PEM web interface the new agent is visible immediately:
pem_agent10

So, lets add the instance:
pem_agent11
pem_agent12
pem_agent17
pem_agent14
pem_agent15

Of course, we need to allow connections to our PostgreSQL instance from the PEM server. Adding this to the pg_hba.conf and reloading the instance fixes the issue:

host    all             all             192.168.22.248/32       md5

Once done:
pem_agent16

… and the instance is there.

In the next post we’ll setup some monitoring for our newly added PostgreSQL instance.

 

Cet article A first look at EDB Postgres Enterprise Manager 7 beta – Connecting a PostgreSQL instance est apparu en premier sur Blog dbi services.


Summer, autumn and winter: a lot of conferences ahead

$
0
0

It is hot in Europe, it is summer, enjoy, but technology moves fast so you have the chance to already prepare for the next conferences. The IT Tage 2017 will happen the 11th to 14th of December this year and we are happy to be there again.
Selection_001
This event covers a wide range of topics and we will be there again covering and talking about:

In addition you will have the chance to join Jan from EnterpriseDB speaking about the differences between Oracle and PostgreSQL. If you want to hear more about that, you might want to join the PostgreSQL Conference Europe this year.

Selection_002

Maybe Jan and me get the chance to talk there as well, we submitted an interesting session, stay tuned.

Chances are high that you may find us at the ukoug, too:

Selection_004

… and we’ll be at the #DOAG2017 for sure this year again.
Selection_003

Hope to see you soon… Cheers, discussions ahead …

 

Cet article Summer, autumn and winter: a lot of conferences ahead est apparu en premier sur Blog dbi services.

Can I do it with PostgreSQL? – 15 – invisible indexes

$
0
0

It has been quite a while since the last post in this series. Today we’ll look at what you know from Oracle as: Invisible indexes. In case you wonder what they might be useful for: Imagine you want to test if an index would benefit one or more queries without affecting the production workload. In other words: Wouldn’t it be cool to create an index but somehow tell the optimizer not to use it for the ongoing queries? This is what invisible indexes are about: Create an index which you believe should improve performance for one or more queries but at the same step make sure that it is not taken into account when the query plan is generated and then executed. The bad news is: This is not possible in PostgreSQL core. The good news is: There is an extension which does exactly this.

The extension is called hypopg and is available via github. The readme states that it works on all PostgreSQL versions starting with 9.2, so lets try it with PostgreSQL 10 Beta1.

postgres@pgbox:/home/postgres/ [PG10B] psql -X postgres
psql (10beta1 dbi services build)
Type "help" for help.

postgres=# select version();
                                                            version                                                            
-------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 10beta1 dbi services build on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11), 64-bit
(1 row)

postgres=# 

Getting the extension downloaded, compiled and installed is straight forward:

postgres@pgbox:/home/postgres/ [PG10B] wget https://github.com/dalibo/hypopg/archive/master.zip
postgres@pgbox:/home/postgres/ [PG10B] unzip master.zip 
postgres@pgbox:/home/postgres/ [PG10B] cd hypopg-master/
postgres@pgbox:/home/postgres/hypopg-master/ [PG10B] make install
postgres@pgbox:/home/postgres/hypopg-master/ [PG10B] psql -X -c "create extension hypopg" postgres
CREATE EXTENSION
postgres@pgbox:/home/postgres/hypopg-master/ [PG10B] psql -X -c "\dx" postgres
                     List of installed extensions
  Name   | Version  |   Schema   |             Description             
---------+----------+------------+-------------------------------------
 hypopg  | 1.1.0dev | public     | Hypothetical indexes for PostgreSQL
 plpgsql | 1.0      | pg_catalog | PL/pgSQL procedural language
(2 rows)

Here we go, all fine until now and we should be ready to use it. Obviously we need a table and some data to test with:

postgres@pgbox:/home/postgres/ [PG10B] psql -X postgres
psql (10beta1 dbi services build)
Type "help" for help.

postgres=# \! cat a.sql
drop table if exists t1;
create table t1 ( a int );
with generator as 
 ( select a.*
     from generate_series ( 1, 5000000 ) a
    order by random()
 )
insert into t1 ( a ) 
     select a
       from generator;
postgres=# \i a.sql
DROP TABLE
CREATE TABLE
INSERT 0 5000000
postgres=# analyze t1;
ANALYZE
postgres=# select * from pg_size_pretty ( pg_total_relation_size ('t1') );
 pg_size_pretty 
----------------
 173 MB
(1 row)

So now we have a table containing some data. The only choice PostgreSQL has to fetch one or more rows is to use a sequential scan (which is a full table scan in Oracle):

postgres=# explain select * from t1 where a = 5;
                             QUERY PLAN                              
---------------------------------------------------------------------
 Gather  (cost=1000.00..49165.77 rows=1 width=4)
   Workers Planned: 2
   ->  Parallel Seq Scan on t1  (cost=0.00..48165.67 rows=1 width=4)
         Filter: (a = 5)
(4 rows)

Although PostgreSQL already knows that only one row needs to be returned (rows=1) it still needs to read the whole table. Lets look at how that looks like when we really execute the query by using “explain (analyze)”:

postgres=# explain (analyze) select * from t1 where a = 5;
                                                    QUERY PLAN                                                     
-------------------------------------------------------------------------------------------------------------------
 Gather  (cost=1000.00..49165.77 rows=1 width=4) (actual time=133.292..133.839 rows=1 loops=1)
   Workers Planned: 2
   Workers Launched: 2
   ->  Parallel Seq Scan on t1  (cost=0.00..48165.67 rows=1 width=4) (actual time=110.446..124.888 rows=0 loops=3)
         Filter: (a = 5)
         Rows Removed by Filter: 1666666
 Planning time: 0.055 ms
 Execution time: 135.465 ms
(8 rows)

What kicked in here is parallel query which is available since PostgreSQL 9.6 but this is not really important for the scope of this post. Coming back to the invisible or hypothetical indexes: Having the extension installed we can now do something like this:

postgres=# SELECT * FROM hypopg_create_index('CREATE INDEX ON t1 (a)');
 indexrelid |     indexname     
------------+-------------------
      16399 | btree_t1_a
(1 row)

postgres=# select * from pg_size_pretty ( pg_total_relation_size ('t1') );
 pg_size_pretty 
----------------
 173 MB
(1 row)

What this did is to create a hypothetical index but without consuming any space (pg_total_relation_size counts the indexes as well), so it is pretty fast. What happens to our query now?

postgres=# explain select * from t1 where a = 5;
                                   QUERY PLAN                                    
---------------------------------------------------------------------------------
 Index Only Scan using btree_t1_a on t1  (cost=0.06..8.07 rows=1 width=4)
   Index Cond: (a = 5)
(2 rows)

Quite cool, the index is really getting used and we did not consume any resources for the index itself. Could be a good index to implement. What you need to know is, that this does not work for “explain analyze” as this really executes the query (and we do not really have an index on disk):

postgres=# explain (analyze) select * from t1 where a = 5;
                                                    QUERY PLAN                                                     
-------------------------------------------------------------------------------------------------------------------
 Gather  (cost=1000.00..49165.77 rows=1 width=4) (actual time=76.247..130.235 rows=1 loops=1)
   Workers Planned: 2
   Workers Launched: 2
   ->  Parallel Seq Scan on t1  (cost=0.00..48165.67 rows=1 width=4) (actual time=106.861..124.252 rows=0 loops=3)
         Filter: (a = 5)
         Rows Removed by Filter: 1666666
 Planning time: 0.043 ms
 Execution time: 131.866 ms
(8 rows)

If you want to list all the hypothetical indexes you can do this as well:

 indexrelid |     indexname     | nspname | relname | amname 
------------+-------------------+---------+---------+--------
      16399 | btree_t1_a | public  | t1      | btree
(1 row)

Of course you can drop them when not anymore required:

postgres=# select * from  hypopg_drop_index(16399);
 hypopg_drop_index 
-------------------
 t
(1 row)

postgres=# SELECT * FROM hypopg_list_indexes();
 indexrelid | indexname | nspname | relname | amname 
------------+-----------+---------+---------+--------
(0 rows)

Hope this helps …

 

Cet article Can I do it with PostgreSQL? – 15 – invisible indexes est apparu en premier sur Blog dbi services.

Did you ever wonder what PostgreSQL is executing in the background when you use the psql shortcuts?

$
0
0

When you work with PostgreSQL you are probably using psql and when you use psql you are probably using one or more of the shortcuts psql is providing. These shortcuts provide a quick and convenient way to get meta data out of the PostgreSQL catalog which safes you from a lot of typing and typos. The only issue with that is that it hides the statements which are executed to get the meta data so you don’t know were the information is actually coming from. Of course you can check either the information_schema or the PostgreSQL system catalog and then write your own queries for what you are looking for. But, hey, there is a much easier way.

Lets start by creating a dummy table and an index:

postgres=# create table dummy ( a int primary key, b varchar(50), c timestamp with time zone );
CREATE TABLE
postgres=# create index i_dummy on dummy ( c );
CREATE INDEX
postgres=# 

The fastest way to get the definition of the table is:

postgres=# \d dummy
             Table "public.dummy"
 Column |           Type           | Modifiers 
--------+--------------------------+-----------
 a      | integer                  | not null
 b      | character varying(50)    | 
 c      | timestamp with time zone | 
Indexes:
    "dummy_pkey" PRIMARY KEY, btree (a)
    "i_dummy" btree (c)

As you can see you do not only get the definition of the table itself but also information about the primary key and the index. But where does this information come from? As information about the index and the primary key is displayed as well the information must be coming from more than one catalog table, but which? Quite easy when you check the options of psql:

postgres@pgbox:/home/postgres/ [PG962] psql --help | grep "hidden"
  -E, --echo-hidden        display queries that internal commands generate

When you fire up psql with this option all the internal statements will be displayed when you use a short cut:

postgres@pgbox:/home/postgres/ [PG962] psql -E postgres
postgres=# \d dummy
********* QUERY **********
SELECT c.oid,
  n.nspname,
  c.relname
FROM pg_catalog.pg_class c
     LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
WHERE c.relname ~ '^(dummy)$'
  AND pg_catalog.pg_table_is_visible(c.oid)
ORDER BY 2, 3;
**************************

********* QUERY **********
SELECT c.relchecks, c.relkind, c.relhasindex, c.relhasrules, c.relhastriggers, c.relrowsecurity, c.relforcerowsecurity, c.relhasoids, '', c.reltablespace, CASE WHEN c.reloftype = 0 THEN '' ELSE c.reloftype::pg_catalog.regtype::pg_catalog.text END, c.relpersistence, c.relreplident
FROM pg_catalog.pg_class c
 LEFT JOIN pg_catalog.pg_class tc ON (c.reltoastrelid = tc.oid)
WHERE c.oid = '16679';
**************************

********* QUERY **********
SELECT a.attname,
  pg_catalog.format_type(a.atttypid, a.atttypmod),
  (SELECT substring(pg_catalog.pg_get_expr(d.adbin, d.adrelid) for 128)
   FROM pg_catalog.pg_attrdef d
   WHERE d.adrelid = a.attrelid AND d.adnum = a.attnum AND a.atthasdef),
  a.attnotnull, a.attnum,
  (SELECT c.collname FROM pg_catalog.pg_collation c, pg_catalog.pg_type t
   WHERE c.oid = a.attcollation AND t.oid = a.atttypid AND a.attcollation  t.typcollation) AS attcollation,
  NULL AS indexdef,
  NULL AS attfdwoptions
FROM pg_catalog.pg_attribute a
WHERE a.attrelid = '16679' AND a.attnum > 0 AND NOT a.attisdropped
ORDER BY a.attnum;
**************************

********* QUERY **********
SELECT c2.relname, i.indisprimary, i.indisunique, i.indisclustered, i.indisvalid, pg_catalog.pg_get_indexdef(i.indexrelid, 0, true),
  pg_catalog.pg_get_constraintdef(con.oid, true), contype, condeferrable, condeferred, i.indisreplident, c2.reltablespace
FROM pg_catalog.pg_class c, pg_catalog.pg_class c2, pg_catalog.pg_index i
  LEFT JOIN pg_catalog.pg_constraint con ON (conrelid = i.indrelid AND conindid = i.indexrelid AND contype IN ('p','u','x'))
WHERE c.oid = '16679' AND c.oid = i.indrelid AND i.indexrelid = c2.oid
ORDER BY i.indisprimary DESC, i.indisunique DESC, c2.relname;
**************************

********* QUERY **********
SELECT pol.polname,
CASE WHEN pol.polroles = '{0}' THEN NULL ELSE array_to_string(array(select rolname from pg_roles where oid = any (pol.polroles) order by 1),',') END,
pg_catalog.pg_get_expr(pol.polqual, pol.polrelid),
pg_catalog.pg_get_expr(pol.polwithcheck, pol.polrelid),
CASE pol.polcmd 
WHEN 'r' THEN 'SELECT'
WHEN 'a' THEN 'INSERT'
WHEN 'w' THEN 'UPDATE'
WHEN 'd' THEN 'DELETE'
WHEN '*' THEN 'ALL'
END AS cmd
FROM pg_catalog.pg_policy pol
WHERE pol.polrelid = '16679' ORDER BY 1;
**************************

********* QUERY **********
SELECT c.oid::pg_catalog.regclass FROM pg_catalog.pg_class c, pg_catalog.pg_inherits i WHERE c.oid=i.inhparent AND i.inhrelid = '16679' ORDER BY inhseqno;
**************************

********* QUERY **********
SELECT c.oid::pg_catalog.regclass FROM pg_catalog.pg_class c, pg_catalog.pg_inherits i WHERE c.oid=i.inhrelid AND i.inhparent = '16679' ORDER BY c.oid::pg_catalog.regclass::pg_catalog.text;
**************************

             Table "public.dummy"
 Column |           Type           | Modifiers 
--------+--------------------------+-----------
 a      | integer                  | not null
 b      | character varying(50)    | 
 c      | timestamp with time zone | 
Indexes:
    "dummy_pkey" PRIMARY KEY, btree (a)
    "i_dummy" btree (c)

Here you go. Quite a lot of stuff is happening in the background and you can exactly see what it is. This is a great way to get known to the catalog. When you are already inside psql and want to switch the display of the hidden stuff to on you can do that as well:

postgres=# \set ECHO_HIDDEN on
postgres=# \d dummy
********* QUERY **********
SELECT c.oid,
  n.nspname,
  c.relname
FROM pg_catalog.pg_class c
     LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
WHERE c.relname ~ '^(dummy)$'
  AND pg_catalog.pg_table_is_visible(c.oid)
ORDER BY 2, 3;
**************************

...

When you want to make this permanent add it to your .psqlrc (scroll down to the “Files” section). Have fun …

 

Cet article Did you ever wonder what PostgreSQL is executing in the background when you use the psql shortcuts? est apparu en premier sur Blog dbi services.

What are typed tables in PostgreSQL?

$
0
0

While reading the PostgreSQL documentation for “CREATE TABLE” I came across this:

“OF type_name: Creates a typed table, which takes its structure from the specified composite type (name optionally schema-qualified). A typed table is tied to its type; for example the table will be dropped if the type is dropped (with DROP TYPE … CASCADE).”

Sounds interesting, lets have a look.

Obviously we’ll need a composite type to make use of the feature described above:

postgres=# create type ctyp1 as ( a int, b varchar(10), c date );
CREATE TYPE
postgres=# \d ctyp1
       Composite type "public.ctyp1"
 Column |         Type          | Modifiers 
--------+-----------------------+-----------
 a      | integer               | 
 b      | character varying(10) | 
 c      | date                  | 

Using the “CREATE TABLE” statement we can now create a table which is based on that type:

postgres=# create table t1 of ctyp1;
CREATE TABLE
postgres=# \d t1
             Table "public.t1"
 Column |         Type          | Modifiers 
--------+-----------------------+-----------
 a      | integer               | 
 b      | character varying(10) | 
 c      | date                  | 
Typed table of type: ctyp1

But is that useful? First I thought no, but there is at least one use case for which this is a great help. Image you need another table with the same structure, maybe in another schema. All you need to do is:

postgres=# create table t2 of ctyp1;
CREATE TABLE
postgres=# \d t2
             Table "public.t2"
 Column |         Type          | Modifiers 
--------+-----------------------+-----------
 a      | integer               | 
 b      | character varying(10) | 
 c      | date                  | 
Typed table of type: ctyp1

Not much of a help you might think as you can just create another table the usual way by specifying all the columns once more, but wait: How would you manage changes to the table structure then? Having both tables based on the same type gives you this:

postgres=# alter type ctyp1 add attribute d numeric cascade;
ALTER TYPE
postgres=# \d ctyp1
       Composite type "public.ctyp1"
 Column |         Type          | Modifiers 
--------+-----------------------+-----------
 a      | integer               | 
 b      | character varying(10) | 
 c      | date                  | 
 d      | numeric               | 

The magic is in the keyword “cascade”. What happened is that both our tables now look like this:

postgres=# \d t1
             Table "public.t1"
 Column |         Type          | Modifiers 
--------+-----------------------+-----------
 a      | integer               | 
 b      | character varying(10) | 
 c      | date                  | 
 d      | numeric               | 
Typed table of type: ctyp1

postgres=# \d t2
             Table "public.t2"
 Column |         Type          | Modifiers 
--------+-----------------------+-----------
 a      | integer               | 
 b      | character varying(10) | 
 c      | date                  | 
 d      | numeric               | 
Typed table of type: ctyp1

With one command we changed the structure of all the tables which are based on the composite type. Btw: When you skip the “cascade” keyword you’ll get an error:

postgres=# alter type ctyp1 add attribute e numeric;
ERROR:  cannot alter type "ctyp1" because it is the type of a typed table
HINT:  Use ALTER ... CASCADE to alter the typed tables too.

Can be quite useful…Good to know that this is possible.

 

Cet article What are typed tables in PostgreSQL? est apparu en premier sur Blog dbi services.

Can I do it with PostgreSQL? – 16 – DDL triggers

$
0
0

A question I received recently from a customer: Oracle gives you the possibility to create DDL triggers. DDL triggers fire (as the name implies) when DDL events occur. Can we do that in PostgreSQL? Yes, this feature is called event triggers. There is a nice list in the documentation which lists for which DDLs an event trigger can fire: Event Trigger Firing Matrix. Lets go …

As usual lets create a dummy table we can work with:

postgres=# create table t1 ( a int );
CREATE TABLE
postgres=# \d t1
      Table "public.t1"
 Column |  Type   | Modifiers 
--------+---------+-----------
 a      | integer | 

What we want to do is to prevent any modifications of the table structure, how do we do this? Obviously we need a way to catch the alter statement against our table and then raise an exception displaying some text. What we need to do is to create a function which returns the pseudo type “event_trigger”:

CREATE OR REPLACE FUNCTION no_ddl() RETURNS event_trigger AS $$
DECLARE r RECORD;
BEGIN
    FOR r IN SELECT * FROM pg_event_trigger_ddl_commands() LOOP
        RAISE EXCEPTION 'You are not allowed to change %', r.object_identity;
    END LOOP;
END;
$$
LANGUAGE plpgsql;

What this function is doing is to iterate over the result set of pg_event_trigger_ddl_commands and then raises an exception. In addition to that we need the event trigger that calls the function:

CREATE EVENT TRIGGER no_ddl_allowed
  ON ddl_command_end WHEN TAG IN ('ALTER TABLE')
  EXECUTE PROCEDURE no_ddl();

Lets see if it works:

postgres=# alter table t1 add column g text;
ERROR:  You are not allowed to change public.t1
CONTEXT:  PL/pgSQL function no_ddl() line 5 at RAISE

Cool, but there is an issue with the current implementation:

postgres=# create table t2 ( a int );
CREATE TABLE
postgres=# alter table t2 add column b text;
ERROR:  You are not allowed to change public.t2
CONTEXT:  PL/pgSQL function no_ddl() line 5 at RAISE

What we effectively did is to deny all alter statements for all objects in that database. This is probably not what you want. A better approach is this:

CREATE OR REPLACE FUNCTION no_ddl() RETURNS event_trigger AS $$
DECLARE r RECORD;
BEGIN
    FOR r IN SELECT * FROM pg_event_trigger_ddl_commands() LOOP
      IF ( r.objid::regclass::text = 't1' )
      THEN
            RAISE EXCEPTION 'You are not allowed to change %', r.object_identity;
      END IF;
    END LOOP;
END;
$$
LANGUAGE plpgsql;

This way we are only raising the exception when the table “t1″ is involved and do nothing for all other tables:

postgres=# alter table t2 add column b text;
ALTER TABLE
postgres=# alter table t1 add column b text;
ERROR:  You are not allowed to change public.t1

Hope this helps …

 

Cet article Can I do it with PostgreSQL? – 16 – DDL triggers est apparu en premier sur Blog dbi services.

Can I do it with PostgreSQL? – 17 – Identifying a blocking session

$
0
0

One single blocking session in a database can completely halt your application so identifying which session is blocking other sessions is a task you must be able to perform quickly. In Oracle you can query v$session for getting that information (blocking_session, final_blocking_session). Can you do the same in PostgreSQL? Yes, you definitely can, lets go.

As usual we’ll start by creating a test table:

postgres@pgbox:/home/postgres/ [PG10B] psql -X postgres
psql (10beta2 dbi services build)
Type "help" for help.

postgres=# create table t1 ( a int );
CREATE TABLE
postgres=# 

One way to force other sessions to wait is to start a new transaction, modify the table:

postgres=# begin;
BEGIN
postgres=# alter table t1 add column t2 text;
ALTER TABLE
postgres=#  

… and then try to insert data into the same table from another session:

postgres@pgbox:/home/postgres/ [PG10B] psql -X postgres
psql (10beta2 dbi services build)
Type "help" for help.

postgres=# insert into t1 (a) values (1);

The insert statement will hang/wait because the modification of the table is still ongoing (the transaction did neither commit nor rollback, remember that DDLs in PostgreSQL are transactional). Now that we have a blocking session how can we identify the session?

What “v$session” is in Oracle, pg_stat_activity is in PostgreSQL (Note: I am using PostgreSQL 10Beta2 here):

postgres=# \d pg_stat_activity 
                      View "pg_catalog.pg_stat_activity"
      Column      |           Type           | Collation | Nullable | Default 
------------------+--------------------------+-----------+----------+---------
 datid            | oid                      |           |          | 
 datname          | name                     |           |          | 
 pid              | integer                  |           |          | 
 usesysid         | oid                      |           |          | 
 usename          | name                     |           |          | 
 application_name | text                     |           |          | 
 client_addr      | inet                     |           |          | 
 client_hostname  | text                     |           |          | 
 client_port      | integer                  |           |          | 
 backend_start    | timestamp with time zone |           |          | 
 xact_start       | timestamp with time zone |           |          | 
 query_start      | timestamp with time zone |           |          | 
 state_change     | timestamp with time zone |           |          | 
 wait_event_type  | text                     |           |          | 
 wait_event       | text                     |           |          | 
 state            | text                     |           |          | 
 backend_xid      | xid                      |           |          | 
 backend_xmin     | xid                      |           |          | 
 query            | text                     |           |          | 
 backend_type     | text         

There is no column which identifies a blocking session but there are other interesting columns:

postgres=# select datname,pid,usename,wait_event_type,wait_event,state,query from pg_stat_activity where backend_type = 'client backend' and pid != pg_backend_pid();
 datname  | pid  | usename  | wait_event_type | wait_event |        state        |               query                
----------+------+----------+-----------------+------------+---------------------+------------------------------------
 postgres | 2572 | postgres | Client          | ClientRead | idle in transaction | alter table t1 add column t2 text;
 postgres | 2992 | postgres | Lock            | relation   | active              | insert into t1 (a) values (1);
(2 rows)

This shows only client connections (excluding all the backend connections) and does not show the current session. In this case it is easy to identify the session which is blocking because we only have two sessions. When you have hundreds of sessions it becomes more tricky to identify the session which is blocking by looking at pg_stat_activity.

When you want to know which locks are currently being held/granted in PostgreSQL you can query pg_locks:

postgres=# \d pg_locks
                   View "pg_catalog.pg_locks"
       Column       |   Type   | Collation | Nullable | Default 
--------------------+----------+-----------+----------+---------
 locktype           | text     |           |          | 
 database           | oid      |           |          | 
 relation           | oid      |           |          | 
 page               | integer  |           |          | 
 tuple              | smallint |           |          | 
 virtualxid         | text     |           |          | 
 transactionid      | xid      |           |          | 
 classid            | oid      |           |          | 
 objid              | oid      |           |          | 
 objsubid           | smallint |           |          | 
 virtualtransaction | text     |           |          | 
 pid                | integer  |           |          | 
 mode               | text     |           |          | 
 granted            | boolean  |           |          | 
 fastpath           | boolean  |           |          | 

What can we see here:

postgres=# select locktype,database,relation,pid,mode,granted from pg_locks where pid != pg_backend_pid();
   locktype    | database | relation | pid  |        mode         | granted 
---------------+----------+----------+------+---------------------+---------
 virtualxid    |          |          | 2992 | ExclusiveLock       | t
 virtualxid    |          |          | 2572 | ExclusiveLock       | t
 relation      |    13212 |    24576 | 2992 | RowExclusiveLock    | f
 relation      |    13212 |    24581 | 2572 | AccessExclusiveLock | t
 transactionid |          |          | 2572 | ExclusiveLock       | t
 relation      |    13212 |    24579 | 2572 | ShareLock           | t
 relation      |    13212 |    24576 | 2572 | AccessExclusiveLock | t
(7 rows)

There is one lock for session 2992 which is not granted and that is the session which currently is trying to insert a row in the table (see above). We can get more information by joining pg_locks with pg_database and pg_class taking the pids from above:

select b.locktype,d.datname,c.relname,b.pid,b.mode 
  from pg_locks b 
     , pg_database d
     , pg_class c
 where b.pid in (2572,2992)
   and b.database = d.oid
   and b.relation = c.oid;

 locktype | datname  | relname | pid  |        mode         
----------+----------+---------+------+---------------------
 relation | postgres | t1      | 2992 | RowExclusiveLock
 relation | postgres | t1      | 2572 | AccessExclusiveLock
(2 rows)

Does that help us beside that we now know that both sessions want to do some stuff against the t1 table? Not really. So how can we then identify a blocking session? Easy, use the pg_blocking_pids system information function passing in the session which is blocked:

postgres=# select pg_blocking_pids(2992);
 pg_blocking_pids 
------------------
 {2572}
(1 row)

This gives you a list of sessions which are blocking. Can we kill it? Yes, of course, PostgreSQL comes with a rich set of system administration functions:

postgres=# select pg_terminate_backend(2572);
 pg_terminate_backend 
----------------------
 t

… and the insert succeeds. Hope this helps …

PS: There is a great page on the PostgreSQL Wiki about locks.

 

Cet article Can I do it with PostgreSQL? – 17 – Identifying a blocking session est apparu en premier sur Blog dbi services.

Setting up default parameters for roles in PostgreSQL

$
0
0

As you might know you can set various parameters on the session level in PostgreSQL by using the “set” command:

postgres=# \h set
Command:     SET
Description: change a run-time parameter
Syntax:
SET [ SESSION | LOCAL ] configuration_parameter { TO | = } { value | 'value' | DEFAULT }
SET [ SESSION | LOCAL ] TIME ZONE { timezone | LOCAL | DEFAULT }

This allows a session to adjust parameters at runtime and can be a handy way for on the fly configuration when you need special settings. Wouldn’t it be great if we could have a default set of parameters for a role or user? Maybe there is one user who needs a special setting for work_mem and another one who needs a special setting for search_path. Instead of setting that each time after the connect in the session you can also do that on the server side.

Lets create to users, user a and user b:

postgres=# create user a login password 'a';
CREATE ROLE
postgres=# create user b login password 'b';
CREATE ROLE
postgres=# \du
                                   List of roles
 Role name |                         Attributes                         | Member of 
-----------+------------------------------------------------------------+-----------
 a         |                                                            | {}
 b         |                                                            | {}
 postgres  | Superuser, Create role, Create DB, Replication, Bypass RLS | {}

When we want a special setting for work_mem every time user a creates a new connection and a special setting for search_path every time user b creates a connection we can do it like this:

postgres=# alter user a set work_mem = '1MB';
ALTER ROLE
postgres=# alter user b set search_path='b';
ALTER ROLE
postgres=# 

When user a connects from now on:

postgres=# \c postgres a
You are now connected to database "postgres" as user "a".
postgres=> show work_mem;
 work_mem 
----------
 1MB
(1 row)

When user b connects from now on:

postgres=> \c postgres b
You are now connected to database "postgres" as user "b".
postgres=> show search_path ;
 search_path 
-------------
 b
(1 row)

Notice that this does not prevent a user from overriding the setting:

postgres=> select current_user;
 current_user 
--------------
 b
(1 row)

postgres=> set search_path=c;
SET
postgres=> show search_path ;
 search_path 
-------------
 c
(1 row)

… this is more meant as setting defaults that differ from the main server configuration where it makes sense. And how can you know then which settings are configured for a specific role? Easy, there is pg_roles:

postgres=> select rolname,rolconfig from pg_roles where rolname in ('a','b');
 rolname |    rolconfig    
---------+-----------------
 a       | {work_mem=1MB}
 b       | {search_path=b}
(2 rows)

Good to know…

 

Cet article Setting up default parameters for roles in PostgreSQL est apparu en premier sur Blog dbi services.


Re-assigning all objects from on role to another in PostgreSQL

$
0
0

From time to time it might be required to move objects (tables, indexes, whatever) from one user to another user in a database system. You could do that by dumping all the objects with pg_dump and then load it to the target user. But there are other solutions which are faster than that. Lets go.

Obviously we need two users when we want to move objects from one user to another (actually it is not really moving the objects but more about changing the ownership):

postgres=# create role a login password 'a';
CREATE ROLE
postgres=# create role b login password 'b';
CREATE ROLE
postgres=# create schema a authorization a;
CREATE SCHEMA
postgres=# create schema b authorization b;
CREATE SCHEMA
postgres=# alter role a set search_path=a;
ALTER ROLE
postgres=# alter role b set search_path=b;
ALTER ROLE
postgres=# 

Lets create some objects in schema “a” owned by user “a”:

postgres=# \c postgres a
You are now connected to database "postgres" as user "a".
postgres=> \! cat a.sql
create table a ( a int );
create table b ( a int );
create table c ( a int );
create table d ( a int );
create index i1 on a (a);
create index i2 on b (a);
create index i3 on c (a);
create index i4 on d (a);

postgres=> \i a.sql
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX

By joining pg_class and pg_roles we can verify who is actually the owner of the objects:

postgres=> select t.relname, d.rolname 
             from pg_class t, pg_roles d 
            where t.relowner = d.oid and d.rolname = 'a';
 relname | rolname 
---------+---------
 a       | a
 b       | a
 c       | a
 d       | a
 i1      | a
 i2      | a
 i3      | a
 i4      | a
(8 rows)

The easiest way to make these objects owned by another user (call it “c”) would be:

postgres=# alter user a rename to c;
NOTICE:  MD5 password cleared because of role rename
ALTER ROLE
postgres=# select t.relname, d.rolname from pg_class t, pg_roles d where t.relowner = d.oid and d.rolname = 'c';
 relname | rolname 
---------+---------
 a       | c
 b       | c
 c       | c
 d       | c
 i1      | c
 i2      | c
 i3      | c
 i4      | c
(8 rows)

Not a good idea though as the schema still is named “a” and this at least will create some confusion with the naming. Of course we could rename the schema as well:

postgres=# alter schema a rename to c;
ALTER SCHEMA
postgres=# \c postgres c
You are now connected to database "postgres" as user "c".
postgres=> select count(*) from a;
2017-07-28 15:51:25.499 CEST [3415] ERROR:  relation "a" does not exist at character 22
2017-07-28 15:51:25.499 CEST [3415] STATEMENT:  select count(*) from a;
ERROR:  relation "a" does not exist
LINE 1: select count(*) from a;

… but now we have another mess. Because the search_path is still set to “a” we can not see the objects by default but we will need to use the fully qualified name:

postgres=> select count(*) from c.a;
 count 
-------
     0
(1 row)

Finally we would need to adjust the search_path to get back the previous behavior:

postgres=> set search_path=c;
SET
postgres=> select count(*) from a;
 count 
-------
     0
(1 row)

A lot of steps to follow. Easier is:

postgres=# reassign owned by c to b;
REASSIGN OWNED
postgres=# alter user b set search_path=c,b;
ALTER ROLE
postgres=# \c postgres b
You are now connected to database "postgres" as user "b".
postgres=> select count(*) from a;
 count 
-------
     0
(1 row)
postgres=> select t.relname, d.rolname from pg_class t, pg_roles d where t.relowner = d.oid and d.rolname = 'b';
 relname | rolname 
---------+---------
 d       | b
 i4      | b
 c       | b
 i3      | b
 b       | b
 i2      | b
 a       | b
 i1      | b
(8 rows)

Cool :) There is also a command to drop all objects of a user:

postgres=> drop owned by b;
DROP OWNED
postgres=> select t.relname, d.rolname from pg_class t, pg_roles d where t.relowner = d.oid and d.rolname = 'b';
 relname | rolname 
---------+---------
(0 rows)

Nice …

 

Cet article Re-assigning all objects from on role to another in PostgreSQL est apparu en premier sur Blog dbi services.

Postgres unique constraint

$
0
0

I’ll start a series on Postgres vs. Oracle access paths because I know Oracle and I learn Postgres. While preparing it, I came upon some surprises because I’m so used to Oracle that I take some behavior as granted for any SQL databases. I recently posted a tweet about one of them, comparing latest Postgres version to earliest Oracle version I have on my laptop.
The goal of the tweet was exactly what I said above: show my surprise, using Oracle 7 as a reference because this is the version where I started to learn SQL. And there’s no judgment behind this surprise: I can’t compare a software I use for more than 20 years with one I’m just learning. I have a big admiration for the Oracle design and architecture choices. But I’ve also a big admiration for what the Postgres community is doing.

In my tweet I’ve updated a primary key. I think I’ve never designed in real life a primary key that has to be updated later. For each table we need a key that is immutable to identify rows for referential integrity constraints, or for replication. The value must be known from the first insert (which means the columns are declared not null) and the value is never updated. It makes sense to use a primary key for that as it is unique and not null.

Actually, a better case would be a simple unique constraint where we just exchange two rows. A real-life example is a list of items, having probably a surrogate key as the primary key, and a unique key including an item number. When the user wants to move up one item, we just run an update on two rows, exchanging their numbers. The unique constraint just ensures that we have only distinct values so that a select … order by will always return the values in the same order.

All similar cases have the same consequence: when you process row by row the update, the uniqueness may be violated. But at the end of the statement, the constraint is still valid.

Here is the initial example with updating all rows:


create table demo as select generate_series n from generate_series(1,2);
SELECT 2
alter table demo add constraint demo_pk primary key(n);
ALTER TABLE
select * from demo;
n
---
1
2
(2 rows)
 
begin transaction;
BEGIN
update demo set n=n-1;
UPDATE 2
select * from demo;
n
---
0
1
(2 rows)

This works. I’ve inserted the rows in ascending order of n. Decreasing the value doesn’t violate the uniqueness at any time because it reads rows from the beginning to the end.

However, when we increase the value, we have a duplicate value until we process the next row. And by default, Postgres fails:

update demo set n=n+1;
ERROR: duplicate key value violates unique constraint "demo_pk"
DETAIL: Key (n)=(1) already exists.

The bad thing is that the behavior of the application depends on the physical order of the rows and the order where they are processed. This violates the Codd rule about physical independence. In addition to that, the SQL statements should behave as processing the set of rows rather than low-level row-by-row processing.

But there is also a very good thing: because the constraint is validated row by row, you know which value violates the constraint (here: “DETAIL: Key (n)=(1) already exists” ).

So my statement failed and this in Postgres seems to fail the whole transaction:

commit;
ROLLBACK

My second surprise is that the failure of one statement cancels the whole transaction. I see no error at commit, but it simply tells me that it has done a rollback instead of the commit.

deferrable

So, I compared with Oracle where this statement is always successful, because temporary violations that are resolved later, within the same statement, do not violate the constraint. I compared it with the oldest version I have on my laptop (Oracle 7.3) to show that it is something I’ve never seen as a new feature because I started with Oracle 7. And this kind of thing is the reason why I like SQL. Doing the same with a procedural language requires an intermediate update to be sure that there is no duplicate at any time.

The Postgres community is very responsive, especially when we may think that something works better in Oracle than Postgres (which was not the case here and which was not the goal of my tweet anyway – but tweets are short and may not express the tone properly).

Quickly a solutions were proposed: deferred constraint (example in this blog post).

I know deferred constraints in Oracle. They are similar in Postgres and here is the solution proposed:


alter table demo drop constraint demo_pk;
ALTER TABLE
alter table demo add constraint demo_pk primary key(n) deferrable initially deferred;
ALTER TABLE
begin transaction;
BEGIN
update demo set n=n-1;
UPDATE 2
select * from demo;
n
---
0
1
(2 rows)
 
update demo set n=n+1;
UPDATE 2

That seems good. Because the constraint validation is deferred, the update is successful.

However, this is not what I want. I want the previous statement to succeed, but I want the following statement to fail:

insert into demo values(1);
INSERT 0 1

Because constraint is deferred, this statement is successful and it is only at commit that it fails:

commit;
ERROR: duplicate key value violates unique constraint "demo_pk"
DETAIL: Key (n)=(1) already exists.

Why do I think this is not the good solution? First, because I want the statement to fail as soon as possible. And in addition to that, I want the commit to be fast. Doing expensive things at commit should be avoided, if possible. It is the point where all work is supposed to be done and you just want to save it (make it durable and visible to others).

deferrable initially immediate

Actually, the solution is to declare the constraint as deferrable, but not deferred.

alter table demo drop constraint demo_pk;
ALTER TABLE
alter table demo add constraint demo_pk primary key(n) deferrable initially immediate;
ALTER TABLE

This says that it is deferrable, but not deferred (except if you decide to set the constraint deferred for your transaction). That way it accepts temporary constraint violation if they are resolved at the end of the statement.

Now, my update statement is sucessful:

begin transaction;
BEGIN
update demo set n=n-1;
UPDATE 2
select * from demo;
n
---
0
1
(2 rows)
update demo set n=n+1;
UPDATE 2

Any other statement that violates the constraint fails immediately:

insert into demo values(1);
ERROR: duplicate key value violates unique constraint "demo_pk"
DETAIL: Key (n)=(1) already exists.
commit;
ROLLBACK

Documentation

The nice thing is that this is documented! I didn’t find it immediately because it is in the ‘Compatibility’ part of the ‘create table’ documentation. I’m not yet used to the Postgres documentation. I stopped at the ‘DEFERRED’ definition which mentions: A constraint that is not deferrable will be checked immediately after every command

But later Compatibility adds something more specific to the unique constraint:

Non-deferred Uniqueness Constraints
When a UNIQUE or PRIMARY KEY constraint is not deferrable, PostgreSQL checks for uniqueness immediately whenever a row is inserted or modified. The SQL standard says that uniqueness should be enforced only at the end of the statement; this makes a difference when, for example, a single command updates multiple key values. To obtain standard-compliant behavior, declare the constraint as DEFERRABLE but not deferred (i.e., INITIALLY IMMEDIATE). Be aware that this can be significantly slower than immediate uniqueness checking.

That’s another good point. Postgres documentation is clear and gives the right solution. We just have to read it to the end.

A side note for my French speaking readers here to mention that the Postgres documentation has been translated into French by Guillaume Lelarge, who also translated Markus Winand book and website. Translation is as good as the original in both cases.

Performance

The documentation mentions ‘significantly slower’. Here is a test on 100000 rows with non deferable constraint:

create table demo as select generate_series n from generate_series(1,100000);
SELECT 100000
alter table demo add constraint demo_pk primary key(n);
ALTER TABLE
vacuum demo;
VACUUM
select * from pgstatindex('demo_pk');
version | tree_level | index_size | root_block_no | internal_pages | leaf_pages | empty_pages | deleted_pages | avg_leaf_density | leaf_fragmentation
---------+------------+------------+---------------+----------------+------------+-------------+---------------+------------------+--------------------
2 | 1 | 2260992 | 3 | 1 | 274 | 0 | 0 | 89.83 | 0
(1 row)

Here is the update n=n-1 where all rows are updated but none violates the constraint at any time:

explain (analyze,verbose,costs,buffers)update demo set n=n-1;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------
Update on public.demo (cost=0.00..1693.00 rows=100000 width=10) (actual time=425.699..425.699 rows=0 loops=1)
Buffers: shared hit=578646 read=1202 dirtied=1267
-> Seq Scan on public.demo (cost=0.00..1693.00 rows=100000 width=10) (actual time=0.013..16.186 rows=100000 loops=1)
Output: (n - 1), ctid
Buffers: shared hit=443

This update has read 578646+1202=579848 buffers.

Now creating the deferrable constraint:

alter table demo drop constraint demo_pk;
ALTER TABLE
alter table demo add constraint demo_pk primary key(n) deferrable initially immediate;
ALTER TABLE
vacuum demo;
VACUUM
select * from pgstatindex('demo_pk');
version | tree_level | index_size | root_block_no | internal_pages | leaf_pages | empty_pages | deleted_pages | avg_leaf_density | leaf_fragmentation
---------+------------+------------+---------------+----------------+------------+-------------+---------------+------------------+--------------------
2 | 1 | 2260992 | 3 | 1 | 274 | 0 | 0 | 89.83 | 0

And do the n=n+1 update:

explain (analyze,verbose,costs,buffers)update demo set n=n+1;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------
Update on public.demo (cost=0.00..2135.00 rows=100000 width=10) (actual time=481.868..481.868 rows=0 loops=1)
Buffers: shared hit=679405 read=760 dirtied=825
-> Seq Scan on public.demo (cost=0.00..2135.00 rows=100000 width=10) (actual time=0.268..16.329 rows=100000 loops=1)
Output: (n + 1), ctid
Buffers: shared hit=885
Planning time: 0.237 ms
Trigger PK_ConstraintTrigger_75314 for constraint demo_pk: time=174.976 calls=99999
Execution time: 663.799 ms

This read more buffers and we can see that an internal trigger (PK_ConstraintTrigger_75314) has been run to re-check the unique constraint at the end of the statement. But only 17% more here for this special case where all rows are updated.

However, a more realistic test case exchanging only two values is much cheaper:


explain (analyze,verbose,costs,buffers) update demo set n=case when n=2 then 2000 when n=2000 then 2 end where n in (2,2000);
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------
Update on public.demo (cost=8.85..16.60 rows=2 width=10) (actual time=0.079..0.079 rows=0 loops=1)
Buffers: shared hit=23
-> Bitmap Heap Scan on public.demo (cost=8.85..16.60 rows=2 width=10) (actual time=0.016..0.055 rows=2 loops=1)
Output: CASE WHEN (n = 2) THEN 2000 WHEN (n = 2000) THEN 2 ELSE NULL::integer END, ctid
Recheck Cond: (demo.n = ANY ('{2,2000}'::integer[]))
Heap Blocks: exact=3
Buffers: shared hit=9
-> Bitmap Index Scan on demo_pk (cost=0.00..8.85 rows=2 width=0) (actual time=0.009..0.009 rows=4 loops=1)
Index Cond: (demo.n = ANY ('{2,2000}'::integer[]))
Buffers: shared hit=6
Planning time: 0.137 ms
Trigger PK_ConstraintTrigger_75322 for constraint demo_pk: time=0.005 calls=1
Execution time: 0.120 ms

In my opinion, the overhead here is totally acceptable, especially given the fact that this re-check displays exactly which value violates the constraint in case there is a duplicate.

But I’m going too fast here. I’ve not even started my blog series about access paths where I’ll explain the cost of the execution plans, starting from the most simple: Seq Scan. Follow my blog or twitter to get informed. There will be nothing about ‘which is better, Oracle or Postgres?’. But I’m convinced that knowing the difference helps to understand how it works, and to design an application that has the correct behavior if ported from one to the other.

 

Cet article Postgres unique constraint est apparu en premier sur Blog dbi services.

PostgreSQL on Cygwin

$
0
0

I run my laptop with Windows 10 for office programs, and VirtualBox machines with Linux for the big stuff (Oracle databases). I have also Cygwin installed on Windows for GNU programs. I wanted to quickly install PosgreSQL and rather than installing it in a Linux VM, or as a Windows program, I installed the Cygwin version of it. Here is how.

Cygwin

Cygwin is easy to install: just run the setup-x86_64.exe from https://www.cygwin.com/ and choose the packages you want to install. Here is what is related to PostgreSQL:
CapturePGCY0001

Note that if you want to install postgres extensions you may need pg_config and you need to install the libpd-devel in addition to postgresql-devel. And gcc and make. Those are not displayed in the screenshot above but you may get something like the following, if you don’t have them, when installing an extension:
pg_config: Command not found

Of course, PostgreSQL is Open Source and you can also compile it yourself.

Cygserver

Cygwin can run daemons through a Windows service (Cygserver) and you need to set it up if not already done. For this step, you will need to run the Cygwin Terminal as Administrator.
fpa@dell-fpa ~
$ /usr/bin/cygserver-config
Overwrite existing /etc/cygserver.conf file? (yes/no) yes
Generating /etc/cygserver.conf file
 
Warning: The following function requires administrator privileges!
 
Do you want to install cygserver as service?
(Say "no" if it's already installed as service) (yes/no) yes
 
The service has been installed under LocalSystem account.
To start it, call `net start cygserver' or `cygrunsrv -S cygserver'.
 
Further configuration options are available by editing the configuration
file /etc/cygserver.conf. Please read the inline information in that
file carefully. The best option for the start is to just leave it alone.
 
Basic Cygserver configuration finished. Have fun!

You start this service as any Windows service:

fpa@dell-fpa ~
$ net start cygserver
The CYGWIN cygserver service is starting.
The CYGWIN cygserver service was started successfully.

You can check from that the service is running:

fpa@dell-fpa ~
$ cygstart services.msc

CapturePGCY0002

PostgreSQL database cluster

Here is the creation of the PostgreSQL database cluster.
fpa@dell-fpa ~
$ /usr/sbin/initdb -D /usr/share/postgresql/data
The files belonging to this database system will be owned by user "fpa".
This user must also own the server process.
 
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
 
Data page checksums are disabled.
 
creating directory /usr/share/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 30
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
 
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
 
Success. You can now start the database server using:
 
/usr/sbin/pg_ctl -D /usr/share/postgresql/data -l log.txt start

Start PostgreSQL database server

I add my network onto the /usr/share/postgresql/data/postgresql.conf

# IPv4 local connections:
host all all 127.0.0.1/32 trust
host all all 192.168.78.0/24 trust

I define the interface and port where the server listen in /usr/share/postgresql/data/postgresql.conf

listen_addresses = 'localhost,192.168.78.1' # what IP address(es) to listen on;
# comma-separated list of addresses;
# defaults to 'localhost'; use '*' for all
# (change requires restart)
port = 5432 # (change requires restart)
max_connections = 30 # (change requires restart)

Now ready to start the PostgreSQL server:
fpa@dell-fpa ~
$ /usr/sbin/pg_ctl -D /usr/share/postgresql/data -l log.txt start
server starting

Username

My Windows username is ‘FPA’ and so is the Cygwin user which started the database server and I check that I can connect to the maintenance database with this user:

fpa@dell-fpa ~
$ psql -U fpa postgres
psql (9.6.2)
Type "help" for help.
 
postgres=# \du
 
List of roles
Role name | Attributes | Member of
-----------+------------------------------------------------------------+-----------
fpa | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
 
postgres=# quit

PgAdmin

As I am on Windows, I install the graphical console PgAdmin and setup the connection to this database:
CapturePGCY0003

SQL Developer

As an Oracle fan, I prefer to connect with SQL Developer. Just download the JDBC driver for PostgreSQL: https://jdbc.postgresql.org/download.html

In SQL Developer you can declare this .jar from Tools -> Preferences -> Third Party JDBC Drivers

CapturePGCY0004

And create the connection with the new ‘PostgreSQL’ tab:

CapturePGCY0005
Then with ‘Choose Database’ you can fill the dropbox and choose the database you want to connect to.

As I have no database with the same name as the username, I have to mention the database name at the end of the hostname, suffixed with ‘?’ to get the proper JDBC url. And what you put in the dropbox will be ignored. I don’t really know the reason, but this is how I got the correct url.

CapturePGCY0006

Extensions

You can install extensions. For example, I’ve installed pg_hint_plan to be able to hint the access path and join methods.

wget https://osdn.net/dl/pghintplan/pg_hint_plan96-1.2.1.tar.gz
tar -zxvf pg_hint_plan96-1.2.1.tar.gz
cd pg_hint_plan96-1.2.1
make
make install

And I’m now able to load it:

$ psql
psql (9.6.2)
Type "help" for help.
 
fpa=# load 'pg_hint_plan';
LOAD

But Why?

You may wonder why I don’t install it directly on Linux. My laptop is on Windows and, of course, I have a lot of VirtualBox VMs. But this doesn’t require to start a VM.
You may wonder why I don’t install the Windows version? I want to investigate the linux behaviour. And I may want to trace the postgres processes. For example, cygwin has a strace.exe which shows similar output as strace on Linux. Here is the I/O calls from a full table scan (Seq Scan):
CaptureStraceCygwinPostgres
I can see that postgres sequential reads are done through one lseek() and sequential 8k read().

This was simple. Just get the pid of the session process:

fpa=# select pg_backend_pid();
pg_backend_pid
----------------
11960

and strace it:

$ strace -p 11960

I’ve done that in about one hour: download, install, setup and write this blog post. Without any virtual machine, you can have a Linux Postgres database server running on Windows.

 

Cet article PostgreSQL on Cygwin est apparu en premier sur Blog dbi services.

Postgres vs. Oracle access paths – intro

$
0
0

This is the start of a series on PostgreSQL execution plans, access path, join methods, hints and execution statistics. The approach will compare Postgres and Oracle. It is not a comparison to see which one is better, but rather to see what is similar and where the approaches diverge. I have a long experience of reading Oracle execution plans and no experience at all on Postgres. This is my way to learn and share what I learn. You will probably be interested if you are in the same situation: an Oracle DBA wanting to learn about Postgres. But you may also be an experienced Postgres DBA who wants to see a different point of view from a different ‘culture’.

I’ll probably use the Oracle terms more often as I’m more familiar with them: blocks for pages, optimizer for query planner, rows for tuples, tables for relations…

Please, don’t hesitate to comment on the blog posts or through twitter (@FranckPachot) if you find some mistakes in my Postgres interpretation. I tend to verify any assumption in the same way I do it with Oracle: the documented behavior and the test result should match. My test should be fully reproducible (using Postgres 9.6.2 here with all defaults). But as I said above, I’ve not the same experience as I have on Oracle when interpreting execution statistics.

Postgres

I’m using the latest versions here. Postgres 9.2.6 (as the one I installed here)
I’ve installed pg_hint_plan to be able to control the execution plan with hints. This is mandatory when doing some research. In order to understand an optimizer (query planner) choice, we need to see the estimated cost for different possibilities. Most of my tests will be done with: EXPLAIN (ANALYZE,VERBOSE,COSTS,BUFFERS)

fpa=# explain (analyze,verbose,costs,buffers) select 1;
 
QUERY PLAN
------------------------------------------------------------------------------------
Result (cost=0.00..0.01 rows=1 width=4) (actual time=0.001..0.001 rows=1 loops=1)
Output: 1
Planning time: 0.060 ms
Execution time: 0.036 ms
(4 rows)

I my go further with unix tools (like strace to see the system calls)

Oracle

I’m using Oracle 12.2 here and the tests are done by running the statement after setting ALTER SESSION SET STATISTICS_LEVEL=ALL and displaying the execution plan with DBMS_XPLAN:
select * from dbms_xplan.display_cursor(format=>'+cost allstats last -plan_hash +projection');
Note that if you are in lower Oracle versions, you need to call dbms_xplan through the table() function:
select * from table(dbms_xplan.display_cursor(format=>'+cost allstats last -plan_hash +projection'));
Example:

SQL> set arraysize 5000 linesize 150 trimspool on pagesize 1000 feedback off termout off
SQL> alter session set statistics_level=all;
SQL> select 1 from dual;
SQL> set termout on
SQL> select * from dbms_xplan.display_cursor(format=>'+cost allstats last -plan_hash +projection');
PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------------------------------------------------------
SQL_ID 520mkxqpf15q8, child number 0
-------------------------------------
select 1 from dual
--------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | Cost (%CPU)| A-Rows | A-Time |
--------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 2 (100)| 1 |00:00:00.01 |
| 1 | FAST DUAL | | 1 | 1 | 2 (0)| 1 |00:00:00.01 |
--------------------------------------------------------------------------------------

I’ll probably never compare the execution time, as this depends on the system and makes no sense on artificial small examples. But I’ll try to compare all other statistics: estimated cost, the actual number of pages/blocks read, etc.

Table of content

I’ll update (or rather insert /*+ append */) the links to the series posts as soon as they are published.

  1. The first one will be: Postgres vs. Oracle access paths I – SeqScan
 

Cet article Postgres vs. Oracle access paths – intro est apparu en premier sur Blog dbi services.

Postgres vs. Oracle access paths I – Seq Scan

$
0
0

Here is the first test I’ve done for my Postgres vs. Oracle access paths series and the first query did a sequential scan. It illustrates the first constant you find in the documentation for the query planner:
seq_page_cost (floating point)
Sets the planner’s estimate of the cost of a disk page fetch that is part of a series of sequential fetches. The default is 1.0.

Table creation

I start by creating a very simple table with 10000 rows and 3 columns. The first column(n) is indexed:

create table demo1 as select generate_series n , 1 a , lpad('x',1000,'x') x from generate_series(1,10000);
SELECT 10000
create unique index demo1_n on demo1(n);
CREATE INDEX
 
analyze verbose demo1;
INFO: analyzing "public.demo1"
INFO: "demo1": scanned 1429 of 1429 pages, containing 10000 live rows and 0 dead rows; 10000 rows in sample, 10000 estimated total rows
ANALYZE
select relkind,relname,reltuples,relpages from pg_class where relname='demo1';
relkind | relname | reltuples | relpages
---------+---------+-----------+----------
r | demo1 | 10000 | 1429
 
select relkind,relname,reltuples,relpages from pg_class where relname='demo1_n';
relkind | relname | reltuples | relpages
---------+---------+-----------+----------
i | demo1_n | 10000 | 30

I checked the table and index statistics that will be used by the optimizer: 10000 rows, all indexed, 1429 table blocks and 30 index blocks. Note that blocks are called pages, but that’s the same idea: the minimal size read and written to disk. They are also called buffers as they are read into a buffer and cached in the buffer cache.

Here is how I create a similar table in Oracle:

create table demo1 as select rownum n , 1 a , lpad('x',1000,'x') x from xmltable('1 to 10000');
Table created.
create unique index demo1_n on demo1(n);
Index created.
exec dbms_stats.gather_table_stats(user,'demo1');
PL/SQL procedure successfully completed.
 
select table_name,num_rows,blocks from user_tables where table_name='DEMO1';
 
TABLE_NAME NUM_ROWS BLOCKS
---------- ---------- ----------
DEMO1 10000 1461
 
select index_name,num_rows,leaf_blocks,blevel from user_indexes where table_name='DEMO1';
 
INDEX_NAME NUM_ROWS LEAF_BLOCKS BLEVEL
---------- ---------- ----------- ----------
DEMO1_N 10000 20 1

The same rows are stored in 1421 table blocks and the index entries in 20 blocks. Both use 8k blocks, but different storage layout and different defaults. This is about 7 rows per table blocks, for rows that are approximately larger than 1k and about 500 index entries per index block to store the number for column N plus the pointer to table row (a few bytes called TID in Postgres or ROWID for Oracle). I’ll not get into the details of the number here. More about the row storage:

My goal is to detail the execution plans and the execution statistics.

Postgres Seq Scan

I start with a very simple query on my table: SELECT SUM(N) from DEMO1;


explain (analyze,verbose,costs,buffers) select sum(n) from demo1 ;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------
Aggregate (cost=1554.00..1554.01 rows=1 width=8) (actual time=4.616..4.616 rows=1 loops=1)
Output: sum(n)
Buffers: shared hit=1429
-> Seq Scan on public.demo1 (cost=0.00..1529.00 rows=10000 width=4) (actual time=0.011..3.614 rows=10000 loops=1)
Output: n, a, x
Buffers: shared hit=1429
Planning time: 0.468 ms
Execution time: 4.661 ms

This query does a sequential scan (Seq Scan), which is the equivalent of Oracle Full Table Scan: read all rows from the table. You might tell me that it would be cheaper to scan the index because the index I’ve created holds all required columns. We will see that in the next post. Here, after having created the table as I did above, the query planner prefers to scan the table.

Here are the maths: my table has 1429 pages and each page access during a sequential scan has cost=1 as defined by:

show seq_page_cost;
seq_page_cost
---------------
1

Here, I see a cost estimated from 0 to 1529 for the Seq Scan operation.
The first number, 0.00 is the initialization cost estimating the work done before returning any rows. A Seq Scan has nothing to do before, and reading the first block can already return rows.
The second number is the cost to return all rows. We have seen that the scan itself costs 1429 but the rows (tuples) must be read and processed. This is evaluated using the following constant:

show cpu_tuple_cost;
cpu_tuple_cost
----------------
0.01

For 10000 rows, the cost to process them is 0.01*10000=100 which is an additional cost over the Seq Scan 1429 to get it to 1529. This explains cost=0.00..1529.00

Then there is a SUM operation applied to 10000 rows and there is a single parameter for the CPU cost of operators and functions:

show cpu_operator_cost;
cpu_operator_cost
-------------------
0.0025

Capturepgoraseqscan001
The sum (Aggregate) operation adds 0.0025*10000=25 to the cost and then the cost is 1554. You can see this cost in the minimal cost for the query, the first number in cost=1554.00..1554.01, which is the cost before retrieving any rows. This makes sense because before retrieving the first row we need to read (Seq Scan) and process (Aggregate) all rows, which is exactly what the cost of 1554 is.

Then there is an additional cost when we retrieve all rows. It is only one row here because it is a sum without group by, and this adds the default cpu_tuple_cost=0.01 to the initial cost: 1554.01

In summary, The total cost of the query is cost=1554.00..1554.01 and we have seen that it depends on:
– number of pages in the table
– number of rows from the result of the scan (we have no where clause here)
– number of rows summed and retrieved
– the planner parameters seq_page_cost, cpu_tuple_cost, and cpu_operator_cost

Oracle Full Table Scan

When I run the same query on Oracle, the optimizer chooses an index fast full scan rather than a table full scan because all rows and columns are in the index that I’ve created:

  • all rows because the SUM(N) do not need to get rows where N is not null (which are not stored in the index)
  • all columns because I need nothing else than the values for N

We will see that in the next post, for the moment, in order to compare with Postgres, I forced a full table scan with the FULL() hint.

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------------------------------------------------------
SQL_ID bhsjquhh6y08q, child number 0
-------------------------------------
select /*+ full(demo1) */ sum(n) from demo1
---------------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | Cost (%CPU)| A-Rows | A-Time | Buffers |
---------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 397 (100)| 1 |00:00:00.01 | 1449 |
| 1 | SORT AGGREGATE | | 1 | 1 | | 1 |00:00:00.01 | 1449 |
| 2 | TABLE ACCESS FULL| DEMO1 | 1 | 10000 | 397 (0)| 10000 |00:00:00.01 | 1449 |
---------------------------------------------------------------------------------------------------
Column Projection Information (identified by operation id):
-----------------------------------------------------------
1 - (#keys=0) SUM("N")[22] 2 - (rowset=256) "N"[NUMBER,22]

We have seen that Postgres cost=1 is for sequential scans (similar to what we call multiblock reads in Oracle) and random reads (single block reads) have by default cost=4 according to random_page_cost.

Oracle cost unit is based on single block reads and this is why the cost here (397) is lower than the number of blocks (1461). Different units. Postgres counts cost=1 for reads and counts a higher cost when a seek is involved. Oracle counts cost=1 for single block reads (including seek) and lower cost for larger I/O size.
Capturepgoraseqscan002
With the default system statistics, where latency is estimated 10 milliseconds and transfer is estimated to 4KB/ms. The single block read time is estimated to 12 milliseconds (10 + 8192/4096).
Again with the default system statistics where optimizer estimates 8 blocks per multiblock read, the multiblock read time is estimated to 26 milliseconds (10 + 8*8192/4096) which is on average 26/8=3.25 millisecond per block. This means that the ratio of single vs. multi block read is very similar for Oracle (3.25/12=0.27833333) and Postgres (seq_page_cost /random_page_cost=1/4=0.25) with default parameters.

Our table is stored in 1461 blocks and the full table scan involves reading all of them plus some segment header blocks. 1461*0.27833333=396

There is also the costing of CPU (the equivalent to cpu_tuple_cost) which is included here but I’ll not go into the details which are more complex than in Postgres and depends on your processor frequency. The goal of those posts is about Postgres. For Oracle, all this is explained in Jonathan Lewis and Chris Antognini books.

But basically, the idea is the same: Postgres Seq Scan and Oracle Full table Scan read the contiguous table blocks sequentially and the cost mainly depends on the size of the table (number of blocks) and the estimated time for sequential I/O (where bandwidth counts more than latency).

Buffers

In my tests, I’ve not only explained the query, but I executed it to get execution statistics. This is done with EXPLAIN ANALYZE in Postgres and DBMS_XPLAN.DISPLAY_CURSOR in Oracle. The statistics include the number of blocks read at each plan operation, with the BUFFERS option in Postgres and with STATISTICS_LEVEL=ALL in Oracle.


explain (analyze,buffers) select sum(n) from demo1 ;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------
Aggregate (cost=1554.00..1554.01 rows=1 width=8) (actual time=3.622..3.622 rows=1 loops=1)
Buffers: shared hit=1429
-> Seq Scan on demo1 (cost=0.00..1529.00 rows=10000 width=4) (actual time=0.008..1.724 rows=10000 loops=1)
Buffers: shared hit=1429
Planning time: 0.468 ms
Execution time: 4.661 ms

‘Buffers’ displays the number of blocks that have been read by the Seq Scan and is exactly the number of pages in my table. ‘shared hit’ means that they come from the buffer cache.

Let’s run the same when the cache is empty:

explain (analyze,verbose,costs,buffers) select sum(n) from demo1 ;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------
Aggregate (cost=1554.00..1554.01 rows=1 width=8) (actual time=13.837..13.837 rows=1 loops=1)
Output: sum(n)
Buffers: shared read=1429
-> Seq Scan on public.demo1 (cost=0.00..1529.00 rows=10000 width=4) (actual time=0.042..12.506 rows=10000 loops=1)
Output: n, a, x
Buffers: shared read=1429
Planning time: 3.754 ms
Execution time: 13.906 ms

The buffers are now ‘shared read’ instead of ‘shared hit’. In Postgres, the number of logical reads, as we know them in Oracle, is the sum of hits and reads. In Oracle, all blocks are counted as logical reads, which includes the smaller set of physical reads.

IO calls

Here is more about the reads when the block is not in the buffer cache. On Linux, we can trace the system calls to see how those sequential I/Os are implemented.

I get the ‘relfilenode':

postgres=# select relname,relnamespace,reltype,relowner,relfilenode,relpages,reltuples from pg_class where relname='demo1';
relname | relnamespace | reltype | relowner | relfilenode | relpages | reltuples
---------+--------------+---------+----------+-------------+----------+-----------
demo1 | 2200 | 42429 | 10 | 42427 | 1429 | 10000

I get the pid of my session process:

select pg_backend_pid();
-[ RECORD 1 ]--+------
pg_backend_pid | 30732

I can trace system calls:

strace -p 30732

And look at the trace concerning my file (identified with its ‘relfilenode’):

30732 open("base/12924/42427", O_RDWR) = 33
30732 lseek(33, 0, SEEK_END) = 11706368
30732 open("base/12924/42427_vm", O_RDWR) = 43
30732 lseek(33, 0, SEEK_END) = 11706368
30732 lseek(33, 0, SEEK_END) = 11706368
30732 lseek(33, 0, SEEK_SET) = 0
30732 read(33, "\4004\220\3 \4 \360\233\30\10\340\227\30\10"..., 8192) = 8192
30732 read(33, "\4004\220\3 \4 \360\233\30\10\340\227\30\10"..., 8192) = 8192
30732 read(33, "\4004\220\3 \4 \360\233\30\10\340\227\30\10"..., 8192) = 8192
... 1429 read(33) in total

We see two open() calls with the relfilenode of my table in the file name: one for the table and one for the visibility map
The file descriptor for the table file is 33 and I’ve grepped only the related calls.
The lseek(33,0,SEEK_END) goes to the end of the file (11706368 bytes, which is 11706368/8192=1429 pages.
The lseek(33,0,SEEK_SET) goes to the beginning of the file.
Subsequent read() calls read the whole file, reading page per page (8192 bytes), in sequential order.

This is how sequential reads are implemented in Postgres: one lseek() and sequential read() calls. The I/O size is always the same (8k here). The benefit of sequential scan is not larger I/O calls but simply the absence of seek() in between. The optimization is left to the underlying layers filesystem and read-ahead.

This is very different from Oracle. Not going into the details, here are the kind of system calls you see during the full table scan:

open("/u01/oradata/CDB1A/PDB/users01.dbf", O_RDWR|O_DSYNC) = 9
fcntl(9, F_SETFD, FD_CLOEXEC) = 0
fcntl(9, F_DUPFD, 256) = 258
...
pread(258, "\6\242\2\5\3\276\25%\2\4\24\270\1\313!\1x\25%"..., 1032192, 10502144) = 1032192
pread(258, "\6\242\202\5\3\300\25%\2\4\16\247\1\313!\1x\25%"..., 1032192, 11550720) = 1032192
pread(258, "\6\242\2\6\3\302\25%\2\4x\226\1\313!\1x\25%"..., 417792, 12599296) = 417792

Those are also sequential reads of contiguous blocks but done with larger I/O size (126 blocks here). So in addition to the absence of seek() calls, it is optimized to do less I/O calls, not relying on the underlying optimization at OS level.

Oracle can also trace the system calls with wait events, which gives more information about the database calls:

WAIT #140315986764280: nam='db file scattered read' ela= 584 file#=12 block#=1282 blocks=126 obj#=74187 tim=91786554974
WAIT #140315986764280: nam='db file scattered read' ela= 485 file#=12 block#=1410 blocks=126 obj#=74187 tim=91786555877
WAIT #140315986764280: nam='db file scattered read' ela= 181 file#=12 block#=1538 blocks=51 obj#=74187 tim=91786556380

The name ‘scattered’ is misleading. ‘db file scattered read’ are actually multiblock reads: read more than one block in one I/O call. Oracle does not rely on the Operating System read-ahead and this is why we can (and should) use direct I/O and Async I/O if the database buffer cache is correctly sized.

Output and Projection

I’ve run the EXPLAIN with the VERBOSE option which shows the ‘Output’ for each operation, and I’ve done the equivalent in Oracle by adding the ‘+projection’ format in DBMS_XPLAN.

In the Oracle execution plan, we see the columns remaining in the result of each operation, after the projection:

Column Projection Information (identified by operation id):
-----------------------------------------------------------
1 - (#keys=0) SUM("N")[22] 2 - (rowset=256) "N"[NUMBER,22]

The operation 2, the Full Table Scan, reads all rows with all columns, but selects only the one we need: N

In the Postgres equivalent, it seems that the Output mentions the columns available before the projection because we see all table columns here:

explain verbose select sum(n) from demo1 ;
QUERY PLAN
-------------------------------------------------------------------------
Aggregate (cost=1554.00..1554.01 rows=1 width=8)
Output: sum(n)
-> Seq Scan on public.demo1 (cost=0.00..1529.00 rows=10000 width=4)
Output: n, a, x

I prefer to see the columns after the projection and I use it a lot in Oracle to know which columns are needed from the table. A great optimization can be done when we have a covering index where all selected columns are present so that we don’t have to go to the table. But we will see that in the next post about Index Only Scan.

 

Cet article Postgres vs. Oracle access paths I – Seq Scan est apparu en premier sur Blog dbi services.

Viewing all 526 articles
Browse latest View live


Latest Images

<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>