The PostgreSQL psql utility provides some really nice features. One of these features is SQL interpolation which allows us to do interesting things, e.g. reading files and analyze the results directly in the database. This post will show how to use this by reading and analyzing sar files of a linux server.

Usually linux hosts store sar statistics in the /var/log/sa directory:

ls /var/log/sa
 sa02 sa03 sa04 sa05 sa06 sa07 sa08 sa09 sa10 sa11 sa12 sa13 sa14 sa15 sa16 sa18 sa19 sa21 sa25 sa28 sa29 sa30

This files can later be analyzed by providing the file name as an argument to sar:

sar -f /var/log/sa/sa07  | head -8
Linux 3.8.13-68.1.2.el7uek.x86_64 (oel7.lcsys.ch) 	05/07/2015 	_x86_64_	(1 CPU)

09:49:30 AM       LINUX RESTART

09:50:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
10:00:01 AM     all      0.03      0.00      0.11      0.13      0.00     99.73
10:10:01 AM     all      0.48      0.00      0.38      1.93      0.00     97.21
10:20:01 AM     all      0.05      0.00      0.15      0.08      0.00     99.73

For the scope of this post let’s say there is a requirement to store the iowait statistics for a long time so that the values may be analyzed at any point in time. The perfect place to store data that needs to be analyzed/queried is the database
Now let’s come back to SQL interpolation. psql allows us to store the result of an OS command into a variable, e.g:

postgres=# \set blubb `ls -la`

The variable then can be queried with standard sql:

postgres=# select :'blubb';
                                          ?column?                                           
---------------------------------------------------------------------------------------------
 total 2521056                                                                              +
 drwx------.  5 postgres postgres       4096 Jun 15 10:54 .                                 +
 drwxr-xr-x.  4 root     root             37 May 28 11:10 ..                                +
 -rw-rw-r--.  1 postgres postgres        218 May  7 16:37 1mio_copy_index_after_hstore.sql  +
 -rw-rw-r--.  1 postgres postgres        217 May  7 16:36 1mio_copy_index_before_hstore.sql +
 -rw-rw-r--.  1 postgres postgres        149 May  7 16:07 1mio_copy_no_indexes_hstore.sql   +
 -rw-rw-r--.  1 postgres postgres        144 May  8 16:20 1mio_copy_no_indexes_jsonb.sql    +

… …

If the result of any command can be stored into a variable in psql we are able to do something like this:

postgres=# \set iowait `sar -f /var/log/sa/sa07`
postgres=# select :'iowait';
                                            ?column?                                             
-------------------------------------------------------------------------------------------------
 Linux 3.8.13-68.1.2.el7uek.x86_64 (oel7.lcsys.ch)       05/07/2015      _x86_64_        (1 CPU)+
                                                                                                +
 09:49:30 AM       LINUX RESTART                                                                +
                                                                                                +
 09:50:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle                +
 10:00:01 AM     all      0.03      0.00      0.11      0.13      0.00     99.73                +
 10:10:01 AM     all      0.48      0.00      0.38      1.93      0.00     97.21                +

As we are only interested in the iowait statistics let’s get rid of everything we don’t need:

postgres=# \set iowait `sar -f /var/log/sa/sa07 | egrep -v "^$|Average|Linux|LINUX|CPU" | awk -F " " '{print $7}'`
postgres=# select :'iowait';
 ?column? 
----------
 0.13    +
 1.93    +
 0.08    +
 0.15    +
 0.06    +

Looks much better. But if we insert this into a table the result would be one varchar column which is not very good for data analysis. Let’s do some more transformation and delete all new lines and replace all spaces with commas:

postgres=# \set iowait `sar -f /var/log/sa/sa07 | egrep -v "^$|Average|Linux|LINUX|CPU" | awk -F " " '{print $7}' | sed ':a;N;$!ba;s/\n/,/g'`
postgres=# select :'iowait';
                                                                                                        ?c
Column?                                                                                                    
    
----------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------
----
 0.13,1.93,0.08,0.15,0.06,0.79,6.15,0.01,0.00,7.20,5.89,0.46,0.05,0.01,0.00,0.00,0.00,0.00,0.00,0.02,0.00,
0.08,0.11,0.04,0.02,0.00,0.00,0.00,0.00,1.05,0.05,0.12,0.00,0.00,0.07,0.67,0.04,0.87,1.14,0.10,0.12,0.03,0
.00
(1 row)

We now have all the values on one line separated by comma. A perfect starting point for an array:

postgres=# select string_to_array(:'iowait',',');
                                                                                                     string_to_array                                                       
                                               
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------
 {0.13,1.93,0.08,0.15,0.06,0.79,6.15,0.01,0.00,7.20,5.89,0.46,0.05,0.01,0.00,0.00,0.00,0.00,0.00,0.02,0.00,0.08,0.11,0.04,0.02,0.00,0.00,0.00,0.00,1.05,0.05,0.12,0.00,0.00
,0.07,0.67,0.04,0.87,1.14,0.10,0.12,0.03,0.00}
(1 row)

Now we have an array of strings. Still not the best thing for analyzing numbers, so let’s do another cast:

postgres=# select string_to_array(:'iowait',',')::numeric[];
                                                                                                     string_to_array                                                       
                                               
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------
 {0.13,1.93,0.08,0.15,0.06,0.79,6.15,0.01,0.00,7.20,5.89,0.46,0.05,0.01,0.00,0.00,0.00,0.00,0.00,0.02,0.00,0.08,0.11,0.04,0.02,0.00,0.00,0.00,0.00,1.05,0.05,0.12,0.00,0.00
,0.07,0.67,0.04,0.87,1.14,0.10,0.12,0.03,0.00}
(1 row)

This is something we can work with. No we need a table to store this:

postgres=# create table iowaits ( day date, iowaits numeric[] );
CREATE TABLE
postgres=# insert into iowaits (day,iowaits) 
                  values ( current_date, string_to_array(:'iowait',',')::numeric[] );
INSERT 0 1

Perfect. Let’s do some analyzing:

postgres=# with bb as ( select unnest(iowaits) vals, day   from iowaits where day = current_date ) select max(vals) from bb;
 max  
------
 7.20
(1 row)

Sure, for a real analysis the data model needs some more investigation. For showing how easy it is to work with files or results of OS commands in PostgreSQL this should be enough

Cet article SQL Interpolation with psql est apparu en premier sur Blog dbi services.

In PostgreSQL ( as well as in other rdbms ) you can define columns as arrays. What I wondered is: What is more efficient when it comes to space: Creating several columns or just creating once column as array? The result, at least for me, is rather surprising.

First, let’s create a table with columns of type int:

postgres=# create table single_values ( a int, b int, c int, d int, e int, f int, g int, h int, i int, j int );
CREATE TABLE
postgres=# \d single_values
 Table "public.single_values"
 Column |  Type   | Modifiers 
--------+---------+-----------
 a      | integer | 
 b      | integer | 
 c      | integer | 
 d      | integer | 
 e      | integer | 
 f      | integer | 
 g      | integer | 
 h      | integer | 
 i      | integer | 
 j      | integer | 

Next create a table with just one column of type array[int]:

postgres=# create table arr_values ( a int[] );
CREATE TABLE

Ready for populating the first table;

truncate table single_values;
\timing on
Timing is on.
DO $$DECLARE
  ln int := 1000;
BEGIN
  for i in 0..1e6 loop
    insert into single_values (a,b,c,d,e,f,g,h,i,j)
           values (ln,ln,ln,ln,ln,ln,ln,ln,ln,ln);
  end loop;
END$$;
DO
Time: 4225.313 ms
postgres=# 

And the table with the array column:

truncate table arr_values;
 \timing on
 Timing is on.
 DO $$DECLARE
 larr int[] := '{1000,1000,1000,1000,1000,1000,1000,1000,1000,1000}';
 BEGIN
 for i in 0..1e6 loop
 insert into arr_values (a)
 values (larr);
 end loop;
 END$$;

postgres=# \i populate_arr_values.sql
 TRUNCATE TABLE
 DO
 Time: 4033.523 ms
 postgres=#

Now, what about the size?

postgres=# select pg_size_pretty(pg_relation_size('single_values'));
 pg_size_pretty 
----------------
 65 MB
(1 row)

Time: 0.227 ms
postgres=# select pg_size_pretty(pg_relation_size('arr_values'));
 pg_size_pretty 
----------------
 89 MB
(1 row)

Time: 0.737 ms

I did expect the table with the array to be smaller but it is the other way around. The table with the 10 columns is smaller. In the next post I’ll do the same for oracle.

Cet article What is more efficient: arrays or single column values? est apparu en premier sur Blog dbi services.

At the 26th of June I had the chance to attend the second Swiss Postgres Conference at the HSR Rapperswil. It was packed with interesting sessions.

Magnus Hagander, a well known community member, started by giving an overview of PostgreSQL Europe. Then I joined Hans Schönig for learning how to write aggregate functions in PostgreSQL. Honestly I did not understand all of it but at least I am now aware of what you can do with that: Almost everything I’ll try to write a blog about it in some days.
Markus Winand followed with “novercal indexes” (I hope the translation matches). The main cause of bad performance, according to Markus, is query/index mismatch. He pointed out that tuning in a lot of cases seems to be the responsibility of the DBA, but not the developer. So the DBA needs to tune statements a developer has written, which is time consuming and risky. Here is the link to his website: http://use-the-index-luke.com.
The next session I attended was “Postgres-XL vs Bi-Directional Replication” held by Markus Wanner. He started by explaining the basics (synchronous, asynchronous, consistency, multi-master). Then he gave an overview of several implementations: Postgres-XL, Postgres-XC, BDR.


After the lunch Marc Ballmer talked about “Securing your PostreSQL application – Measures to mitigate the effects of a database exploit, e.g. when you application has bugs”. This talk was all about security: Data stealing, data manipulation, data destruction. A nice comic about the topic is here.
Michael Rutz followed with “Breaking Security Barriers”. Again security topics but more related to roles, how they work and what you can do with them.
Magnus Hagander finished the technical sessions with “A TARDIS for your ORM – application level time travel in PostgreSQL”. This session was about implementing a system that allows selects on historical data, e.g. give me state for a person as it was 10 days ago (similar to oracle’s flashback data archive). This was implemented using range types, the tzrange in this case.
Conclusion: I will definitely join the next conference if I find the time for it. For getting new ideas, getting to know more and more people in the community this is the right place to be. By the way: Since a few months there is the Swiss PostgreSQL User-Group. Interested to join?

Cet article Swiss Postgres Conference 2015 est apparu en premier sur Blog dbi services.

Do you need a hundred databases and users for training etc. in PostgreSQL?

Just a few lines of code:

[postgres8@oel7 data]$ cat create_databases.sh
 #!/bin/bash
 NUMOFDBS=100
 for i in `seq ${NUMOFDBS}`; do
 psql -q -c "create user u${i} UNENCRYPTED password 'u${i}' NOCREATEDB NOCREATEROLE NOCREATEUSER" postgres
 createdb -O u${i} db${i}

done

Execute it and you are ready:

[postgres8@oel7 data]$ time ./create_databases.sh
real 1m37.825s
 user 0m0.143s
 sys 0m0.109s

Have a look at the databases and its owners:

postgres=# \l+

Name | Owner | Encoding | Collate | Ctype | Access privileges | Size | Tablespace | Description
 -----------+-----------+----------+-------------+-------------+-------------------------+---------+------------+--------------------------------------------
 db1 | u1 | UTF8 | en_US.UTF-8 | en_US.UTF-8 | | 6705 kB | pg_default |
 db10 | u10 | UTF8 | en_US.UTF-8 | en_US.UTF-8 | | 6705 kB | pg_default |
 db100 | u100 | UTF8 | en_US.UTF-8 | en_US.UTF-8 | | 6705 kB | pg_default |
 db11 | u11 | UTF8 | en_US.UTF-8 | en_US.UTF-8 | | 6705 kB | pg_default |
 db12 | u12 | UTF8 | en_US.UTF-8 | en_US.UTF-8 | | 6705 kB | pg_default |
 ...

Really, really easy. For dropping the databases:

[postgres8@oel7 data]$ cat drop_databases.sh
 #!/bin/bash
NUMOFDBS=100
for i in `seq ${NUMOFDBS}`; do
 dropdb db${i}
 dropuser u${i}
done

Cet article Quickly create a hundred databases and users est apparu en premier sur Blog dbi services.

Indexing queries for like/similarity conditions is not that easy with the usual index types. The only option you have with btree indexes (especially if the wild-card is at the beginning of the filter) is to create a partial index on that columns for a very specific query.

Let’s do a simple example with a btree index. The test data:

drop table if exists t1;
 create table t1 ( a varchar(50) );
 insert into t1 ( select md5(tt::text)
 from generate_series ( 1, 100000) tt
 );

This created 1’000’000 distinct rows:

postgres=# with aa as ( select a, count(*) from t1 group by a having count(*) = 1  )
select count(*) from aa;
 count  
--------
 100000
(1 row)

This is a perfect distribution for creating an index if we look for few rows. What options do we have? The first, obvious one, is to create a btree index:

postgres=# create index i_btree on t1(a);
CREATE INDEX

This will perfectly work for queries like this:

postgres=# explain analyze select * from t1 where a = 'd0e2dc293c21d9628c65b5032fe9dc76';
                                                    QUERY PLAN                                               
     
-------------------------------------------------------------------------------------------------------------
-----
 Index Only Scan using i_btree on t1  (cost=0.42..8.44 rows=1 width=33) (actual time=0.041..0.041 rows=1 loop
s=1)
   Index Cond: (a = 'd0e2dc293c21d9628c65b5032fe9dc76'::text)
   Heap Fetches: 1
 Planning time: 0.135 ms
 Execution time: 0.060 ms
(5 rows)

But what if we want to query the data like this (Please excuse the double quotes around the like. I found no other ways to prevent this blog software from doing funny things with it):

postgres=# explain analyze select * from t1 where a like '"%"e2dc293c21d9628c65b5032fe9dc76';
                                            QUERY PLAN                                             
---------------------------------------------------------------------------------------------------
 Seq Scan on t1  (cost=0.00..2084.00 rows=10 width=33) (actual time=11.656..14.481 rows=1 loops=1)
   Filter: ((a)::text~~ '"%"e2dc293c21d9628c65b5032fe9dc76'::text)
   Rows Removed by Filter: 99999
 Planning time: 0.069 ms
 Execution time: 14.493 ms
(5 rows)
postgres=# explain analyze select * from t1 where a like '"%"e2dc93"%"c21d9628c65b5032fe9dc76';
                                            QUERY PLAN                                             
---------------------------------------------------------------------------------------------------
 Seq Scan on t1  (cost=0.00..2084.00 rows=10 width=33) (actual time=10.990..14.112 rows=1 loops=1)
   Filter: ((a)::text ~~ '"%"e2dc93"%"c21d9628c65b5032fe9dc76'::text)
   Rows Removed by Filter: 99999
 Planning time: 0.046 ms
 Execution time: 14.183 ms
(5 rows)

This will trigger a full table scan on the table as the index can not be used. Any chance to fix this with a btree index? We could create a partial index like this for the first query:

postgres=# create index i_btree_t1 on t1(a) where a like '"%"e2dc293c21d9628c65b5032fe9dc76'; CREATE INDEX postgres=#

Does it help?:

postgres=# explain analyze select * from t1 where a like '"%"e2dc293c21d9628c65b5032fe9dc76';
                                                     QUERY PLAN                                              
       
-------------------------------------------------------------------------------------------------------------
-------
 Bitmap Heap Scan on t1  (cost=4.13..40.97 rows=10 width=33) (actual time=0.013..0.013 rows=1 loops=1)
   Recheck Cond: ((a)::text ~~ '"%"e2dc293c21d9628c65b5032fe9dc76'::text)
   Heap Blocks: exact=1
   ->  Bitmap Index Scan on i_btree_t1  (cost=0.00..4.13 rows=10 width=0) (actual time=0.010..0.010 rows=1 lo
ops=1)
 Planning time: 0.208 ms
 Execution time: 0.029 ms
(6 rows)

Yes, it does help. But only in this case. As soon as the query is slightly modified this will not work anymore:

postgres=# explain analyze select * from t1 where a like '"%"XXdc293c21d9628c65b5032fe9dc76';
                                            QUERY PLAN                                             
---------------------------------------------------------------------------------------------------
 Seq Scan on t1  (cost=0.00..2084.00 rows=10 width=33) (actual time=11.529..11.529 rows=0 loops=1)
   Filter: ((a)::text ~~ '"%"XXdc293c21d9628c65b5032fe9dc76'::text)
   Rows Removed by Filter: 100000
 Planning time: 0.080 ms
 Execution time: 11.541 ms
(5 rows)

So this is only an option for very static queries asking for the same “like” over and over again.
Luckily, in PostgreSQL, there is the pg_trgm extension which is shipped by default. From the documentation: “The pg_trgm module provides functions and operators for determining the similarity of alphanumeric text based on trigram matching, as well as index operator classes that support fast searching for similar strings.”
Can this help us? Let’s try. Installing extensions which are shipped by default is just matter of:

postgres=# create extension pg_trgm;
 CREATE EXTENSION
 postgres=#
Btw.: There is a plsql shortcut to list all the extension currently installed;
postgres=# \dx
                                     List of installed extensions
    Name    | Version |   Schema   |                            Description                            
------------+---------+------------+-------------------------------------------------------------------
 cstore_fdw | 1.2     | public     | foreign-data wrapper for flat cstore access
 hstore     | 1.3     | public     | data type for storing sets of (key, value) pairs
 pg_trgm    | 1.1     | public     | text similarity measurement and index searching based on trigrams
 plpgsql    | 1.0     | pg_catalog | PL/pgSQL procedural language
(4 rows)

Now that the extension is installed let’s create an index using the extension:

postgres=# drop index i_btree;
DROP INDEX
postgres=# drop index i_btree_t1;
DROP INDEX
postgres=# 
postgres=# create index i_rtgm on t1 using gist ( a gist_trgm_ops);
CREATE INDEX
postgres=# 

Does it help for the queries above? :

postgres=# explain analyze select * from t1 where a like '"%"e2dc293c21d9628c65b5032fe9dc76';
                                                   QUERY PLAN                                                
   
-------------------------------------------------------------------------------------------------------------
---
 Bitmap Heap Scan on t1  (cost=4.36..41.20 rows=10 width=33) (actual time=1.654..1.654 rows=1 loops=1)
   Recheck Cond: ((a)::text ~~ '"%"e2dc293c21d9628c65b5032fe9dc76'::text)
   Heap Blocks: exact=1
   ->  Bitmap Index Scan on i_rtgm  (cost=0.00..4.36 rows=10 width=0) (actual time=1.646..1.646 rows=1 loops=
1)
         Index Cond: ((a)::text ~~ '"%"e2dc293c21d9628c65b5032fe9dc76'::text)
 Planning time: 0.113 ms"
 Execution time: 1.682 ms
(7 rows)
postgres=# explain analyze select * from t1 where a like '"%"XXdc293c21d9628c65b5032fe9dc76';
                                                   QUERY PLAN                                                
   
-------------------------------------------------------------------------------------------------------------
---
Bitmap Heap Scan on t1  (cost=4.36..41.20 rows=10 width=33) (actual time=1.633..1.633 rows=0 loops=1)
   Recheck Cond: ((a)::text ~~ '"%"XXdc293c21d9628c65b5032fe9dc76'::text)
   ->  Bitmap Index Scan on i_rtgm  (cost=0.00..4.36 rows=10 width=0) (actual time=1.630..1.630 rows=0 loops=
1)
         Index Cond: ((a)::text ~~ '"%"XXdc293c21d9628c65b5032fe9dc76'::text)
 Planning time: 0.058 ms
 Execution time: 1.650 ms
(6 rows)

Even for multiple wild-cards;

postgres=# explain analyze select * from t1 where a like '"%"e2dc"%"93c21d"%%"62"%%"65b5032fe9dc76';
                                                   QUERY PLAN
   
-------------------------------------------------------------------------------------------------------------
---
 Bitmap Heap Scan on t1  (cost=4.36..41.20 rows=10 width=33) (actual time=1.480..1.481 rows=1 loops=1)
   Recheck Cond: ((a)::text ~~ '"%"e2dc"%"93c21d"%%"62"%%"65b5032fe9dc76'::text)
   Heap Blocks: exact=1
   ->  Bitmap Index Scan on i_rtgm  (cost=0.00..4.36 rows=10 width=0) (actual time=1.475..1.475 rows=1 loops=
1)
         Index Cond: ((a)::text ~~ '"%"e2dc"%"93c21d"%%"62"%%"65b5032fe9dc76'::text)
 Planning time: 0.100 ms
 Execution time: 1.501 ms
(7 rows)

Isn’t that cool?

Cet article Indexing for like/similarity operations est apparu en premier sur Blog dbi services.

On the 2cnd of July the PostgreSQL Global Development Group released an alpha version of the upcoming PostgreSQL 9.5. The same day, Josh Berkus, another of those PostgreSQL core team members released a docker image for testing this alpha release. It’s never been that easy to get started with PostgreSQL or testing new features.

If you are on Linux the docker packages should be available for your distribution. For Oracle Linux 7.1 make sure that the addons yum repository is enabled

cat /etc/yum.repos.d/public-yum-ol7.repo | grep -A 3 addon
 [ol7_addons]
 name=Oracle Linux $releasever Add ons ($basearch)
 baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL7/addons/$basearch/
 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
 gpgcheck=1
 enabled=1

Installing docker is just:

yum install -y docker
systemctl start docker.service

Once docker is running we can search the docker registry:

docker search postgres
NAME                    DESCRIPTION                                     STARS     OFFICIAL   AUTOMATED
postgres                The PostgreSQL object-relational database ...   809       [OK]       
abevoelker/postgres     Postgres 9.3 + WAL-E + PL/V8 and PL/Python...   7                    [OK]
linuxkonsult/postgres   A Postgres trusted build installed with Chef.   5                    [OK]
macadmins/postgres      Postgres that accepts remote connections b...   5                    [OK]
jamesbrink/postgres     Highly configurable PostgreSQL container.       2                    [OK]
kampka/postgres         A postgresql image build on top of an arch...   2                    [OK]
azukiapp/postgres       Docker image to run PostgreSQL by Azuki - ...   2                    [OK]
clkao/postgres-plv8     Docker image for running PLV8 1.4 on Postg...   1                    [OK]
eeacms/postgres         Docker image for PostgreSQL (RelStorage re...   0                    [OK]
bantl23/postgres        postgres development                            0                    [OK]

Get the PostgreSQL 9.5 alpha container and run it:

mkdir -p /var/tmp/docker/postgres
docker run -it --volume "/var/tmp/docker/postgres:/tests" jberkus/postgres95-test:alpha
Unable to find image 'jberkus/postgres95-test:alpha' locally
alpha: Pulling from jberkus/postgres95-test
39bb80489af7: Pull complete 
df2a0347c9d0: Pull complete 
65c152b01d5d: Pull complete 
22e16a33eefa: Pull complete 
b7003ab928a5: Pull complete 
b9ad663e6550: Pull complete 
712b208afbdf: Pull complete 
6db70337094f: Pull complete 
94dc52568254: Pull complete 
af3497b0915c: Pull complete 
203e588497fa: Pull complete 
35133c09fcd4: Already exists 
Digest: sha256:cd722ecdcfb68467d077f1edddf9cfcf1faabb954f544098a854bd8d83b61871
Status: Downloaded newer image for jberkus/postgres95-test:alpha

Looks good so we should be able to use it:

root@5e74f2e32422:/# su - postgres
postgres@5e74f2e32422:~$ pg_ctl start
server starting
postgres@5e74f2e32422:~$ [ @  @ 2015-07-03 06:57:09.840 GMT]:LOG:  redirecting log output to logging collector process
[ @  @ 2015-07-03 06:57:09.840 GMT]:HINT:  Future log output will appear in directory "pg_log".
 
postgres@5e74f2e32422:~$ psql
psql (9.5alpha1)
Type "help" for help.

postgres=# \l
                             List of databases
   Name    |  Owner   | Encoding | Collate | Ctype |   Access privileges   
-----------+----------+----------+---------+-------+-----------------------
 libdata   | libdata  | UTF8     | C       | C     | 
 postgres  | postgres | UTF8     | C       | C     | 
 template0 | postgres | UTF8     | C       | C     | =c/postgres          +
           |          |          |         |       | postgres=CTc/postgres
 template1 | postgres | UTF8     | C       | C     | =c/postgres          +
           |          |          |         |       | postgres=CTc/postgres
(4 rows)

Cool. Minimal effort to get a test infrastructure with docker. (Please notice that I did everything as root. It is always better to setup proper sudo permissions)

Cet article Testing the just released PostgreSQL 9.5 Alpha in a docker container est apparu en premier sur Blog dbi services.

For integrating data from other systems PostgreSQL has the concept of foreign data wrappers. Many of these exist for different types of systems. In this post I’ll look into how you may connect PostgreSQL to Oracle.

The home of the foreign data wrapper for Oracle (as most of the PostgreSQL stuff) is on github. If you do not want to clone the git repository you can download it from the PostgreSQL extension network, too.

For building oracle_fdw you’ll need to download the Basic and SDK package of the Oracle instance client. For simplicity I’ll use the rpm versions.

Make sure that pg_config is in your path and you have installed both the Oracle instant client basic and sdk packages:

postgres@oel7:/home/postgres/$ which pg_config
/u01/app/postgres/product/94/db_1/bin/pg_config
postgres@oel7:/home/postgres/$ unzip oracle_fdw-1.2.0.zip
postgres@oel7:/home/postgres/$ sudo yum localinstall oracle-instantclient12.1-basic-12.1.0.2.0-1.x86_64.rpm oracle-instantclient12.1-devel-12.1.0.2.0-1.x86_64.rpm
postgres@oel7:/home/postgres/$ cd oracle_fdw-1.2.0
postgres@oel7:/home/postgres/oracle_fdw-1.2.0/$ make
postgres@oel7:/home/postgres/oracle_fdw-1.2.0/$ make install

Quite easy. Let’s see if it really works:

postgres@oel7:/home/postgres/oracle_fdw-1.2.0/ [PG1] psql postgres
psql (9.4.1dbi services)
Type "help" for help.

postgres=# create extension oracle_fdw;
ERROR:  could not load library "/u01/app/postgres/product/94/db_1/lib/oracle_fdw.so": libclntsh.so.12.1: cannot open shared object file: No such file or directory

Ok, we need to set LD_LIBRARY_PATH to include the Oracle libraries and restart PostgreSQL:

postgres@oel7:/home/postgres/oracle_fdw-1.2.0/$ export LD_LIBRARY_PATH=/usr/lib/oracle/12.1/client64/lib/
postgres@oel7:/home/postgres/oracle_fdw-1.2.0/$ pg_ctl -D /u02/pgdata/PG1 restart -m fast
postgres@oel7:/home/postgres/oracle_fdw-1.2.0/$ psql postgres
psql (9.4.1dbi services)
Type "help" for help.

postgres=# create extension oracle_fdw;
CREATE EXTENSION

Better. Foreign data wrappers work by defining foreign servers and user mappings:

postgres=# create server oracle foreign data wrapper oracle_fdw options (dbserver '//192.168.22.242/PROD.local' );
CREATE SERVER
postgres=# create user mapping for postgres server oracle options (user 'A', password 'a');
CREATE USER MAPPING

My Oracle database has a user called “A” with “a” as the password. Now we need to create the foreign table that should look like the table in Oracle and once this is done we should be able to select:

postgres=# create foreign table INTERVAL_SALES ( prod_id int not null, time_id date, amount_sold numeric) server oracle options (schema 'A', table 'INTERVAL_SALES');
CREATE FOREIGN TABLE
postgres=# select count(*) from INTERVAL_SALES;
 count 
-------
     3
(1 row)

Cool. We even can write to Oracle (Make sure to read the readme of oracle_fdw for understanding the “key” option which is mandatory for updates):

 -- oracle
SQL> select * from "A"."INTERVAL_SALES";

   PROD_ID TIME_ID   AMOUNT_SOLD
---------- --------- -----------
	 1 04-MAY-15	       1
	 2 08-MAY-16	       1
	 3 13-MAY-17	       1
-- postgres
postgres=# insert into INTERVAL_SALES values ( 4, current_date, 5 );
INSERT 0 1

Lets check the data on the Oracle side:

--oracle
SQL> select * from "A"."INTERVAL_SALES";

   PROD_ID TIME_ID   AMOUNT_SOLD
---------- --------- -----------
	 1 04-MAY-15	       1
	 2 08-MAY-16	       1
	 3 13-MAY-17	       1
	 4 18-AUG-15	       5

Looks good. You now have access to the data hosted in an Oracle database and can start working with it.

Again: make sure you read the readme of oracle_fdw, especially the “problems” and “options” sections.

Cet article Connecting your PostgreSQL instance to an Oracle database est apparu en premier sur Blog dbi services.

As you might know PostgreSQL 9.5 is currently in alpha 2. One of the new features will be the possibility to import foreign schemas. Having this
it is no longer required to create all the foreign tables which saves a lot of work.

Setting up oracle_fdw was done in exactly the same way as in the previous post. Make sure you use the latest version of oracle_fdw as some internals changed in PostgreSQL 9.5 which oracle_fdw depends on. The latest zipped version can be downloaded here

For this little demo I created the SCOTT user in my Oracle database ( $ORACLE_HOME/rdbms/admin/utlsampl.sql ):

SQL> show user
USER is "SCOTT"
SQL> select table_name from user_tables;

TABLE_NAME
--------------------------------------------------------------------------------
BONUS
DEPT
EMP
SALGRADE

SQL> 

On the PostgreSQL side lets create a schema and setup oracle_fdw:

postgres@oel7:/home/postgres/ [PG6] psql postgres
psql (9.5alpha2)
Type "help" for help.

postgres=# create schema scott;
CREATE SCHEMA
postgres=# create server oracle foreign data wrapper oracle_fdw options (dbserver '//192.168.22.242/PROD.local' );
CREATE SERVER
postgres=# create user mapping for postgres server oracle options (user 'SCOTT', password 'tiger');
CREATE USER MAPPING

Before PostgreSQL 9.5 now would be the time to create the foreign tables as they are defined in the Oracle database. But starting with PostgreSQL 9.5 we are able to do this:

postgres=# import foreign schema "SCOTT" from server oracle into scott;
IMPORT FOREIGN SCHEMA
postgres=# 

This should have imported all the table definitions:

postgres=# set search_path='scott';
SET
postgres=# \d 
              List of relations
 Schema |   Name   |     Type      |  Owner   
--------+----------+---------------+----------
 scott  | bonus    | foreign table | postgres
 scott  | dept     | foreign table | postgres
 scott  | emp      | foreign table | postgres
 scott  | salgrade | foreign table | postgres
(4 rows)

postgres=# \d+ dept
                                    Foreign table "scott.dept"
 Column |         Type          | Modifiers | FDW Options  | Storage  | Stats target | Description 
--------+-----------------------+-----------+--------------+----------+--------------+-------------
 deptno | smallint              | not null  | (key 'true') | plain    |              | 
 dname  | character varying(14) |           |              | extended |              | 
 loc    | character varying(13) |           |              | extended |              | 
Server: oracle
FDW Options: (schema 'SCOTT', "table" 'DEPT')

postgres=# \d+ emp
                                          Foreign table "scott.emp"
  Column  |              Type              | Modifiers | FDW Options  | Storage  | Stats target | Description 
----------+--------------------------------+-----------+--------------+----------+--------------+-------------
 empno    | smallint                       | not null  | (key 'true') | plain    |              | 
 ename    | character varying(10)          |           |              | extended |              | 
 job      | character varying(9)           |           |              | extended |              | 
 mgr      | smallint                       |           |              | plain    |              | 
 hiredate | timestamp(0) without time zone |           |              | plain    |              | 
 sal      | numeric(7,2)                   |           |              | main     |              | 
 comm     | numeric(7,2)                   |           |              | main     |              | 
 deptno   | smallint                       |           |              | plain    |              | 
Server: oracle
FDW Options: (schema 'SCOTT', "table" 'EMP')

What happens if we try to insert a row that violates the primary key on emp?

postgres=# insert into emp values (7639,'aa','aa',7902,current_date,123,11,20);
INSERT 0 1
postgres=# insert into emp values (7639,'aa','aa',7902,current_date,123,11,20);
ERROR:  error executing query: OCIStmtExecute failed to execute remote query
DETAIL:  ORA-00001: unique constraint (SCOTT.PK_EMP) violated
postgres=# 

Really, really cool. One command to get a whole schema definition out of an Oracle database. Makes life a lot easier.

Cet article Connecting your PostgreSQL instance to an Oracle database – The PostgreSQL 9.5 way est apparu en premier sur Blog dbi services.

How many applications do you know that need to define a validity of a row? And how many of these do that by adding two columns: valid_from and valid_to or similar column names? Well, in PostgreSQL (you already suspect it ) there is much more elegant way to do that. PostgreSQL knows the concept of range types. Several of these are pre-defined and you get them once you install PostgreSQL:

• int4range — Range of integer
• int8range — Range of bigint
• numrange — Range of numeric
• tsrange — Range of timestamp without time zone
• tstzrange — Range of timestamp with time zone
• daterange — Range of date

The classical example for using range types is a booking system for meeting rooms. Meeting rooms are booked for a period of time and the booking system has to know which periods are free before presenting possible choices. Lets do a simple test setup:

create extension if not exists btree_gist;
drop table if exists meeting_rooms_booked;
drop table if exists meeting_rooms;
create table meeting_rooms ( id int primary key
                           , mname varchar(20)
                           , location varchar(10)
                           );
create table meeting_rooms_booked ( mid int references meeting_rooms(id)
                                  , booking_range tsrange
                                  , exclude using gist (mid with =,booking_range with &&)
                                  );
insert into meeting_rooms ( id, mname, location)
       values ( 1, 'meetingsouth', 'south' )
            , ( 2, 'meetingnorth', 'north' )
            , ( 3, 'meetingwest', 'west' )
            , ( 4, 'meetingeast', 'east' );
insert into meeting_rooms_booked ( mid, booking_range )
       values ( 1, '[2015-01-01 15:00, 2015-01-01 18:30]' )
            , ( 1, '[2015-01-01 08:00, 2015-01-01 08:30]' )
            , ( 2, '[2015-03-01 17:00, 2015-03-01 18:30]' )
            , ( 1, '[2015-03-01 05:00, 2015-03-01 08:30]' )
            , ( 3, '[2015-02-01 15:00, 2015-02-01 18:30]' )
            , ( 4, '[2015-02-01 19:00, 2015-02-01 20:30]' )
            , ( 4, '[2015-03-01 15:00, 2015-03-01 18:30]' );

Lets say we want to know if meeting room number 3 is free at the 1st of February 2015 between 16:00 and 16:30. Using the range type operator “&&” (overlap) this is easy to ask:

postgres=# select * from meeting_rooms_booked where mid = 3;
 mid |                 booking_range                 
-----+-----------------------------------------------
   3 | ["2015-02-01 15:00:00","2015-02-01 18:30:00"]
(1 row)

postgres=# select booking_range && '[2015-02-01 16:00,2015-02-01 16:30)'::tsrange from meeting_rooms_booked where mid = 3;
 ?column? 
----------
 t
(1 row)

Indeed this does overlap. If we ask for a period for which the room is free we get false:

postgres=# select booking_range && '[2015-02-01 18:45,2015-02-01 19:15)'::tsrange from meeting_rooms_booked where mid = 3;
 ?column? 
----------
 f
(1 row)

If you go back to the table definition of meeting_rooms_booked the “&&” operator is used to enforce that there can not be two bookings which overlap in time. This is the reason the btree_gist extension was installed as normal btree indexes can not be created on range type columns.

So, if it is about a booking system or simple valid_from/valid_to definitions or something else which will benefit from this goody: range types are what you need

Cet article Representing ranges with one data type? est apparu en premier sur Blog dbi services.

Before PostgreSQL 9.5 (which is in alpha2 currently) you could grant access to individual columns of a table to users or roles. A little test script to demonstrate this:

(postgres@[local]:5432) [postgres] > select version();
                                                   version                                                    
--------------------------------------------------------------------------------------------------------------
 PostgreSQL 9.4.4 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit
(1 row)

create table t ( id int primary key, name varchar(50), salary bigint );
insert into t ( id, name, salary ) values ( 1,'name1',1000), (2,'name2',20000), (3,'name3',3000);
create user u1 UNENCRYPTED PASSWORD 'u1';
create user u2 UNENCRYPTED PASSWORD 'u2';
grant select (id,name) on t to u1;
grant select (name,salary) on t to u2;

User u1 has the right to select columns “id” and “name” and user u2 has the right to select columns “name” and “salary”:

(u1@[local]:5432) [postgres] > select user;
 current_user 
--------------
 u1
(1 row)

(u1@[local]:5432) [postgres] > select id,name from t;
 id | name  
----+-------
  1 | name1
  2 | name2
  3 | name3
(3 rows)

(u1@[local]:5432) [postgres] > select id,name,salary from t;
ERROR:  permission denied for relation t
(u1@[local]:5432) [postgres] > 


(u2@[local]:5432) [postgres] > select user;
 current_user 
--------------
 u2
(1 row)

(u2@[local]:5432) [postgres] > select name,salary from t;
 name  | salary 
-------+--------
 name1 |   1000
 name2 |  20000
 name3 |   3000
(3 rows)

(u2@[local]:5432) [postgres] > select id,name,salary from t;
ERROR:  permission denied for relation t
(u2@[local]:5432) [postgres] > 

As you can see from the above selecting data from columns the users are not authorized to results in a “permission denied” message.

Now lets build the same test case on PostgreSQL 9.5 alpha2:

postgres=# select version();
                                                     version                                                  
    
--------------------------------------------------------------------------------------------------------------
----
 PostgreSQL 9.5alpha2 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-
bit
(1 row)


create table t ( id int primary key, name varchar(50), salary bigint );
insert into t ( id, name, salary ) values ( 1,'name1',1000), (2,'name2',20000), (3,'name3',3000);
create user u1 UNENCRYPTED PASSWORD 'u1';
create user u2 UNENCRYPTED PASSWORD 'u2';
grant select (id,name) on t to u1;
grant select (name,salary) on t to u2;

The new feature “row level security” enables us to do the following:

alter table t enable row level security;

As a first step row level security needs to be enabled for the table. For this the “alter table” command was extended which is already reflected in the documentation for PostgreSQL 9.5.

But this can not be all we need to do, right? No conditions, no checks at this point in time. This is where the policies come into the game:

create policy policy1 on t for select to u2 using (salary < 3000);

In words this policy does the following: If the user is “u2″ and this user is selecting from “t” then only let the user “u2″ see rows that have a salary less than 3000. Does it work?

postgres=> select user;

 current_user 
--------------
 u2
(1 row)

postgres=> select name,salary from t;
 name  | salary 
-------+--------
 name1 |   1000
(1 row)

Exactly what the policy was defined for. Do the column permissions still apply?

postgres=> select id from t;
ERROR:  permission denied for relation t
postgres=> 

Yes, so starting with PostgreSQL 9.5 a mixture of column and row level security is possible. Btw: Describing tables that have policies applied was enhanced, too:

postgres=> \d t
              Table "public.t"
 Column |         Type          | Modifiers 
--------+-----------------------+-----------
 id     | integer               | not null
 name   | character varying(50) | 
 salary | bigint                | 
Indexes:
    "t_pkey" PRIMARY KEY, btree (id)
Policies:
    POLICY "policy1" FOR SELECT
      TO u2
      USING ((salary < 3000))

In addition there is now a new system information function which tells if the current user has row level security enabled for a table:

postgres=> select user;
 current_user 
--------------
 u2
(1 row)
postgres=> select row_security_active('t');
 row_security_active 
---------------------
 t
(1 row)

Conclusion: Yet another great feature which made it into PostgreSQL. If you are willing to alpha/beta test there is a wiki page which describes the best practices.

Cet article Row level security is coming to PostgreSQL est apparu en premier sur Blog dbi services.

This post takes a look on how to migrate the Oracle 12cR1 sample schemas to PPAS 9.4 (PostgreSQL Plus Advanced Server 9.4). I’ll not dig into how to install PPAS as this was described in detail some time ago. Just follow this post if you need a setup guide.

If you wonder why I am doing this there are two reasons:

• to see if it works, to have fun and to learn
• PostgreSQL and PPAS are real alternatives to Oracle so migrating applications from one to another can make make a lot of sense

To download the Oracle database examples point your browser to the otn download page and download the “Oracle Database 12c Release 1 Examples (12.1.0.2.0) for Linux x86-64″ file.
Installing the sample is quite easy: unzip, start OUI, next, next, next:

oracle@oel12102:/u01/app/oracle/software/ [PROD] unzip linuxamd64_12102_examples.zip
oracle@oel12102:/u01/app/oracle/software/ [PROD] cd examples/
oracle@oel12102:/u01/app/oracle/software/examples/ [PROD] ./runInstaller

Once the scripts are available we can install the schemas. The only important point is that the schemas need to be installed in the right order. So I’ll begin with the HR schema:

@?/demo/schema/human_resources/hr_main.sql
specify password for HR as parameter 1:
Enter value for 1: manager

specify default tablespeace for HR as parameter 2:
Enter value for 2: users

specify temporary tablespace for HR as parameter 3:
Enter value for 3: temp

specify password for SYS as parameter 4:
Enter value for 4: manager

specify log path as parameter 5:
Enter value for 5: /var/tmp/

After that the OE schema can be installed (You’ll need the Multimedia Option installed for this to succeed):

SQL> @?/demo/schema/order_entry/oe_main.sql

specify password for OE as parameter 1:
Enter value for 1: manager

specify default tablespeace for OE as parameter 2:
Enter value for 2: users

specify temporary tablespace for OE as parameter 3:
Enter value for 3: temp

specify password for HR as parameter 4:
Enter value for 4: manager

specify password for SYS as parameter 5:
Enter value for 5: manager

specify directory path for the data files as parameter 6:
Enter value for 6: /var/tmp/

writeable directory path for the log files as parameter 7:
Enter value for 7: /var/tmp/

specify version as parameter 8:
Enter value for 8: v3

The we can continue with the PM schema:

SQL> @?/demo/schema/product_media/pm_main.sql

specify password for PM as parameter 1:
Enter value for 1: manager

specify default tablespeace for PM as parameter 2:
Enter value for 2: users

specify temporary tablespace for PM as parameter 3:
Enter value for 3: temp

specify password for OE as parameter 4:
Enter value for 4: manager

specify password for SYS as parameter 5:
Enter value for 5: manager

specify directory path for the PM data files as parameter 6:
Enter value for 6: /u01/app/oracle/product/12.1.0/db_2_0/demo/schema/product_media/

specify directory path for the PM load log files as parameter 7:
Enter value for 7: /u01/app/oracle/product/12.1.0/db_2_0/demo/schema/product_media/

specify work directory path as parameter 8:
Enter value for 8: /u01/app/oracle/product/12.1.0/db_2_0/demo/schema/product_media/

Then continue with the IX schema:

SQL> @?/demo/schema/info_exchange/ix_main.sql

specify password for IX as parameter 1:
Enter value for 1: manager

specify default tablespeace for IX as parameter 2:
Enter value for 2: users

specify temporary tablespace for IX as parameter 3:
Enter value for 3: temp

specify password for SYS as parameter 4:
Enter value for 4: manager

specify path for log files as parameter 5:
Enter value for 5: /u01/app/oracle/product/12.1.0/db_2_0/demo/schema/info_exchange/

specify version as parameter 6:
Enter value for 6: v3

And finally the SH schema:

SQL> @?/demo/schema/sales_history/sh_main.sql

specify password for SH as parameter 1:
Enter value for 1: manager

specify default tablespace for SH as parameter 2:
Enter value for 2: users

specify temporary tablespace for SH as parameter 3:
Enter value for 3: temp

specify password for SYS as parameter 4:
Enter value for 4: manager

specify directory path for the data files as parameter 5:
Enter value for 5: /u00/app/oracle/product/12.1.0/db_2_0/demo/schema/sales_history/

writeable directory path for the log files as parameter 6:
Enter value for 6: /u00/app/oracle/product/12.1.0/db_2_0/demo/schema/sales_history/

specify version as parameter 7:
Enter value for 7: v3

Once everything is installed we have the following objects available:

SQL> select owner,object_type,count(*) num_obj 
       from dba_objects 
      where owner in ('SH','PM','OE','IX','HR','BI') group by owner,object_type order by 1,2;

OWNER                          OBJECT_TYPE                NUM_OBJ
------------------------------ ----------------------- ----------
HR                             INDEX                           19
HR                             PROCEDURE                        2
HR                             SEQUENCE                         3
HR                             TABLE                            7
HR                             TRIGGER                          2
HR                             VIEW                             1
IX                             EVALUATION CONTEXT               2
IX                             INDEX                           17
IX                             LOB                              3
IX                             QUEUE                            4
IX                             RULE SET                         4
IX                             SEQUENCE                         2
IX                             TABLE                           17
IX                             TYPE                             1
IX                             VIEW                             8
OE                             FUNCTION                         1
OE                             INDEX                           48
OE                             LOB                             15
OE                             SEQUENCE                         1
OE                             SYNONYM                          6
OE                             TABLE                           14
OE                             TRIGGER                          4
OE                             TYPE                            37
OE                             TYPE BODY                        3
OE                             VIEW                            13
PM                             INDEX                           21
PM                             LOB                             17
PM                             TABLE                            3
PM                             TYPE                             3
SH                             DIMENSION                        5
SH                             INDEX                           23
SH                             INDEX PARTITION                196
SH                             MATERIALIZED VIEW                2
SH                             TABLE                           13
SH                             TABLE PARTITION                 56
SH                             VIEW                             1

Having the sample schemas available we are almost ready to start the migration to PPAS 9.4. We’ll use the EDB migration toolkit for this as it automates many tasks. The toolkit itself is documented here. If you do not want to read documentation here is the short version

As the migration toolkit uses jdbc to connect to the Oracle database we’ll need to download the Oracle jdbc drivers. I used the latest one, which is 12.1.0.2 (ojdbc7.jar) at the time of writing. This jar file needs to be copied to the following location:

enterprisedb@centos7:/home/enterprisedb/ [dummy] ls -la /etc/alternatives/jre/lib/ext/
total 11424
drwxr-xr-x. 2 root root    4096 Nov 25 14:46 .
drwxr-xr-x. 9 root root    4096 Nov 25 13:01 ..
-rw-r--r--. 1 root root 4003647 Oct 21 22:19 cldrdata.jar
-rw-r--r--. 1 root root    9444 Oct 21 22:19 dnsns.jar
-rw-r--r--. 1 root root   48732 Oct 21 22:19 jaccess.jar
-rw-r--r--. 1 root root 1204407 Oct 21 22:19 localedata.jar
-rw-r--r--. 1 root root     617 Oct 21 22:19 meta-index
-rw-r--r--. 1 root root 2023751 Oct 21 22:19 nashorn.jar
-rw-r--r--. 1 root root 3698857 Nov 25 14:46 ojdbc7.jar  <=================
-rw-r--r--. 1 root root   30448 Oct 21 22:19 sunec.jar
-rw-r--r--. 1 root root  294143 Oct 21 22:19 sunjce_provider.jar
-rw-r--r--. 1 root root  266680 Oct 21 22:19 sunpkcs11.jar
-rw-r--r--. 1 root root   77887 Oct 21 22:19 zipfs.jar
enterprisedb@centos7:/home/enterprisedb/ [dummy] 

The connection parameters to the source and the target have to be specified in the toolkit.properties file which is located in the edbmtk/etc directory of the ppas installation:

cat /u01/app/postgres/product/9.4/ppas_1_3/edbmtk/etc/toolkit.properties
SRC_DB_URL=jdbc:oracle:thin:@192.168.22.242:1521:PROD
SRC_DB_USER=system
SRC_DB_PASSWORD=manager
TARGET_DB_URL=jdbc:edb://localhost:5444/orasample
TARGET_DB_USER=enterprisedb
TARGET_DB_PASSWORD=manager

I want the Oracle sample schemas in my own database in PPAS so I created the ORASAMPLE database:

(enterprisedb@[local]:5444) [postgres] > create database orasample;
CREATE DATABASE
Time: 624.415 ms

Ready for migrating the first schema?

enterprisedb@centos7:/u01/app/postgres/product/9.4/ppas_1_3/edbmtk/bin/ [CRM] pwd
/u01/app/postgres/product/9.4/ppas_1_3/edbmtk/bin
enterprisedb@centos7:/u01/app/postgres/product/9.4/ppas_1_3/edbmtk/bin/ [CRM] ./runMTK.sh -fastCopy -logBadSQL -fetchSize 10000 -loaderCount 1 -dropSchema true HR

The result:

Running EnterpriseDB Migration Toolkit (Build 48.0.1) ...
Source database connectivity info...
conn =jdbc:oracle:thin:@192.168.22.242:1521:PROD
user =system
password=******
Target database connectivity info...
conn =jdbc:edb://localhost:5444/orasample
user =enterprisedb
password=******
Connecting with source Oracle database server...
Connected to Oracle, version 'Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options'
Connecting with target EnterpriseDB database server...
Connected to EnterpriseDB, version '9.4.1.3'
Importing redwood schema HR...
Creating Schema...hr 
Creating Sequence: DEPARTMENTS_SEQ
Creating Sequence: EMPLOYEES_SEQ
Creating Sequence: LOCATIONS_SEQ
Loading Table Data in 8 MB batches...
Creating Table: COUNTRIES
Loading Table: COUNTRIES ...
[COUNTRIES] Migrated 25 rows.
[COUNTRIES] Table Data Load Summary: Total Time(s): 0.075 Total Rows: 25
Creating Table: DEPARTMENTS
Loading Table: DEPARTMENTS ...
[DEPARTMENTS] Migrated 27 rows.
[DEPARTMENTS] Table Data Load Summary: Total Time(s): 0.038 Total Rows: 27
Creating Table: EMPLOYEES
Loading Table: EMPLOYEES ...
[EMPLOYEES] Migrated 107 rows.
[EMPLOYEES] Table Data Load Summary: Total Time(s): 0.09 Total Rows: 107 Total Size(MB): 0.0087890625
Creating Table: JOBS
Loading Table: JOBS ...
[JOBS] Migrated 19 rows.
[JOBS] Table Data Load Summary: Total Time(s): 0.011 Total Rows: 19
Creating Table: JOB_HISTORY
Loading Table: JOB_HISTORY ...
[JOB_HISTORY] Migrated 10 rows.
[JOB_HISTORY] Table Data Load Summary: Total Time(s): 0.026 Total Rows: 10
Creating Table: LOCATIONS
Loading Table: LOCATIONS ...
[LOCATIONS] Migrated 23 rows.
[LOCATIONS] Table Data Load Summary: Total Time(s): 0.03 Total Rows: 23 Total Size(MB): 9.765625E-4
Creating Table: REGIONS
Loading Table: REGIONS ...
[REGIONS] Migrated 4 rows.
[REGIONS] Table Data Load Summary: Total Time(s): 0.025 Total Rows: 4
Data Load Summary: Total Time (sec): 0.489 Total Rows: 215 Total Size(MB): 0.01
Creating Constraint: JHIST_EMP_ID_ST_DATE_PK
Creating Constraint: EMP_EMP_ID_PK
Creating Constraint: EMP_EMAIL_UK
Creating Constraint: JOB_ID_PK
Creating Constraint: DEPT_ID_PK
Creating Constraint: LOC_ID_PK
Creating Constraint: COUNTRY_C_ID_PK
Creating Constraint: REG_ID_PK
Creating Constraint: JHIST_DEPT_FK
Creating Constraint: JHIST_EMP_FK
Creating Constraint: JHIST_JOB_FK
Creating Constraint: DEPT_MGR_FK
Creating Constraint: EMP_MANAGER_FK
Creating Constraint: EMP_JOB_FK
Creating Constraint: EMP_DEPT_FK
Creating Constraint: DEPT_LOC_FK
Creating Constraint: LOC_C_ID_FK
Creating Constraint: COUNTR_REG_FK
Creating Constraint: JHIST_DATE_INTERVAL
Creating Constraint: EMP_SALARY_MIN
Creating Index: LOC_COUNTRY_IX
Creating Index: LOC_STATE_PROVINCE_IX
Creating Index: LOC_CITY_IX
Creating Index: JHIST_DEPARTMENT_IX
Creating Index: JHIST_EMPLOYEE_IX
Creating Index: JHIST_JOB_IX
Creating Index: DEPT_LOCATION_IX
Creating Index: EMP_NAME_IX
Creating Index: EMP_MANAGER_IX
Creating Index: EMP_JOB_IX
Creating Index: EMP_DEPARTMENT_IX
Creating Trigger: SECURE_EMPLOYEES
Creating Trigger: UPDATE_JOB_HISTORY
Creating View: EMP_DETAILS_VIEW
Creating Procedure: ADD_JOB_HISTORY
Creating Procedure: SECURE_DML

Schema HR imported successfully.

Creating User: HR

Migration process completed successfully.

Migration logs have been saved to /home/enterprisedb/.enterprisedb/migration-toolkit/logs

******************** Migration Summary ********************
Sequences: 3 out of 3
Tables: 7 out of 7
Constraints: 20 out of 20
Indexes: 11 out of 11
Triggers: 2 out of 2
Views: 1 out of 1
Procedures: 2 out of 2
Users: 1 out of 1

Total objects: 47
Successful count: 47
Failed count: 0
Invalid count: 0

*************************************************************

That was quite easy, wasn’t it? Non of the objects failed to migrate. Lets validate this inside PPAS. I installed PPAS in Oracle compatibility mode and therefore have the dba_* views available:

(enterprisedb@[local]:5444) [postgres] > \c orasample
You are now connected to database "orasample" as user "enterprisedb".
(enterprisedb@[local]:5444) [orasample] > select object_type,count(*) 
                                            from dba_objects 
                                           where schema_name = 'HR' and status = 'VALID';
 object_type | count 
-------------+-------
 TRIGGER     |     2
 SEQUENCE    |     3
 VIEW        |     1
 PROCEDURE   |     2
 TABLE       |     7
 INDEX       |    19
(6 rows)

Exactly the same amount of objects as in Oracle, even the PL/SQL procedures are there. You don’t believe it? :

(enterprisedb@[local]:5444) [orasample] > select text from dba_source where schema_name = 'HR';
                                                                                                  text                                                                                                   
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 CREATE OR REPLACE PROCEDURE hr.add_job_history(p_emp_id numeric, p_start_date timestamp without time zone, p_end_date timestamp without time zone, p_job_id character varying, p_department_id numeric)
  AUTHID DEFINER IS
 BEGIN
   INSERT INTO job_history (employee_id, start_date, end_date,
                            job_id, department_id)
     VALUES(p_emp_id, p_start_date, p_end_date, p_job_id, p_department_id);
 END
...

Ok, the HR schema is a simple one. Lets continue with the next one, SH:

enterprisedb@centos7:/u01/app/postgres/product/9.4/ppas_1_3/edbmtk/bin/ [CRM] pwd
/u01/app/postgres/product/9.4/ppas_1_3/edbmtk/bin
enterprisedb@centos7:/u01/app/postgres/product/9.4/ppas_1_3/edbmtk/bin/ [CRM] ./runMTK.sh -fastCopy -logBadSQL -fetchSize 10000 -loaderCount 1 -dropSchema true SH

The result:

******************** Migration Summary ********************
Tables: 11 out of 11
Constraints: 14 out of 14
Indexes: 13 out of 13
Views: 1 out of 3

Total objects: 41
Successful count: 39
Failed count: 2
Invalid count: 0

List of failed objects
======================
Views
--------------------
1. SH.FWEEK_PSCAT_SALES_MV
2. SH.CAL_MONTH_SALES_MV

Not bad, but two views are invalid. Why? As specified the “-logBadSQL” switch there is a separate logfile containing all the sql statements which failed:

enterprisedb@centos7:/home/enterprisedb/.enterprisedb/migration-toolkit/logs/ [CRM] pwd
/home/enterprisedb/.enterprisedb/migration-toolkit/logs
enterprisedb@centos7:/home/enterprisedb/.enterprisedb/migration-toolkit/logs/ [CRM] ls -latr | grep SH
-rw-r--r--. 1 enterprisedb enterprisedb    1286 Nov 28 15:30 mtk_bad_sql_SH_20151128032916.sql
-rw-r--r--. 1 enterprisedb enterprisedb    8097 Nov 28 15:30 mtk_SH_20151128032916.log

This file contains exactly the statements for the two views that failed to create:

-- MTK-15009: Error Creating Materialized View: FWEEK_PSCAT_SALES_MV
-- DB-42601: ERROR: syntax error at or near "PREBUILT" at position 53
-- Line 1: CREATE MATERIALIZED VIEW FWEEK_PSCAT_SALES_MV BUILD PREBUILT
--                                                             ^

CREATE MATERIALIZED VIEW FWEEK_PSCAT_SALES_MV BUILD PREBUILT
 REFRESH FORCE
 ON DEMAND
 AS 
SELECT   t.week_ending_day
  ,        p.prod_subcategory
  ,        sum(s.amount_sold) AS dollars
  ,        s.channel_id
  ,        s.promo_id
  FROM     sales s
  ,        times t
  ,        products p
  WHERE    s.time_id = t.time_id
  AND      s.prod_id = p.prod_id
  GROUP BY t.week_ending_day
  ,        p.prod_subcategory
  ,        s.channel_id
  ,        s.promo_id;

-- MTK-15009: Error Creating Materialized View: CAL_MONTH_SALES_MV
-- DB-42601: ERROR: syntax error at or near "PREBUILT" at position 51
-- Line 1: CREATE MATERIALIZED VIEW CAL_MONTH_SALES_MV BUILD PREBUILT
--                                                           ^

CREATE MATERIALIZED VIEW CAL_MONTH_SALES_MV BUILD PREBUILT
 REFRESH FORCE
 ON DEMAND
 AS 
SELECT   t.calendar_month_desc
  ,        sum(s.amount_sold) AS dollars
  FROM     sales s
  ,        times t
  WHERE    s.time_id = t.time_id
  GROUP BY t.calendar_month_desc;

If we take a look at the syntax for create materialized view it becomes clear why this happened:

(enterprisedb@[local]:5444) [postgres] > \h CREATE MATERIALIZED VIEW 
Command:     CREATE MATERIALIZED VIEW
Description: define a new materialized view
Syntax:
CREATE MATERIALIZED VIEW table_name
    [ (column_name [, ...] ) ]
    [ WITH ( storage_parameter [= value] [, ... ] ) ]
    [ TABLESPACE tablespace_name ]
    AS query
    [ WITH [ NO ] DATA ]

There syntax is just wrong. Maybe this is a bug in the migration toolkit as it seems the statements are not mapped from Oracle to PPAS syntax. It is easy to fix:

(enterprisedb@[local]:5444) [orasample] > CREATE MATERIALIZED VIEW sh.FWEEK_PSCAT_SALES_MV
AS SELECT   t.week_ending_day
  ,        p.prod_subcategory
  ,        sum(s.amount_sold) AS dollars
  ,        s.channel_id
  ,        s.promo_id
  FROM     sh.sales s
  ,        sh.times t
  ,        sh.products p
  WHERE    s.time_id = t.time_id
  AND      s.prod_id = p.prod_id
  GROUP BY t.week_ending_day
  ,        p.prod_subcategory
  ,        s.channel_id
  ,        s.promo_id;
SELECT 11266
Time: 8193.370 ms
(enterprisedb@[local]:5444) [orasample] > CREATE MATERIALIZED VIEW sh.CAL_MONTH_SALES_MV
AS SELECT   t.calendar_month_desc
  ,        sum(s.amount_sold) AS dollars
  FROM     sh.sales s
  ,        sh.times t
  WHERE    s.time_id = t.time_id
  GROUP BY t.calendar_month_desc;
SELECT 48
Time: 396.849 ms

Comparing the amount of objects again we should be fine:

(enterprisedb@[local]:5444) [postgres] > \c orasample
You are now connected to database "orasample" as user "enterprisedb".
(enterprisedb@[local]:5444) [orasample] > select object_type,count(*) 
                                            from dba_objects 
                                           where schema_name = 'SH' and status = 'VALID'
                                           group by object_type;
 object_type | count 
-------------+-------
 TRIGGER     |    60
 VIEW        |     1
 TABLE       |    67
 INDEX       |    19
(4 rows)

Uh, totally different numbers. Table partitions are counted as tables here and each partition gets a trigger created (that is how partitions are implemented in PostgreSQL). There is no concept of partitioned indexes in PostgreSQL but we can create indexes on the partitions. I am not sure what happened to the dimension as I am not familiar with this on the oracle side (I’ll see that I can check this in more detail soon). At least nothing about that is reported in the log file. You can see, comparing the amount of objects is not longer sufficient for being able to tell if everything was migrated. Special Oracle features need special considerations and can not be migrated automatically. Not everything can be migrated easily or without adjusting the application but the migration toolkit automates a lot of work and can give a picture of what is possible and what not.

The next schemas will be a topic for another post. Hope this helped.

Cet article Migrating the Oracle 12cR1 sample schemas to PostgreSQL Plus Advanced Server 9.4 est apparu en premier sur Blog dbi services.

If you are used to patch Oracle databases you probably know how to use opatch to apply PSUs. How does PostgreSQL handle this? Do we need to patch the existing binaries to apply security fixes? The answer is: No.

Lets say you want to patch PostgreSQL from version 9.4.1 to version 9.4.5. What do you need to do?

For this little demo I’ll create a new database and a sample table in my 9.4.1 instance:

(postgres@[local]:5432) [postgres] > select version();
                                                         version                                                          
--------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 9.4.1 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit
(1 row)

Time: 0.483 ms
(postgres@[local]:5432) [postgres] > create database patch;
CREATE DATABASE
Time: 2533.745 ms
(postgres@[local]:5432) [postgres] > \c patch
You are now connected to database "patch" as user "postgres".
(postgres@[local]:5432)  > create table test ( a int );
CREATE TABLE
Time: 2.430 ms
(postgres@[local]:5432)  > insert into test (a) values ( generate_series(1,100));
INSERT 0 100
Time: 0.959 ms

If I now want to bring this version to 9.4.5 the first step I’ll need to do is to install the 9.4.5 binaries in a separate path. The binaries for my 9.4.1 installation are located here:

postgres@oel7:/home/postgres/ [PG1] ps -ef | grep PG1
postgres  2645     1  0 10:51 ?        00:00:00 /u01/app/postgres/product/94/db_1/bin/postgres -D /u02/pgdata/PG1
postgres 14439 11550  0 11:04 pts/1    00:00:00 grep --color=auto PG1

I already installed the 9.4.5 binaries here:

postgres@oel7:/home/postgres/ [PG1] ls /u01/app/postgres/product/94/db_5
bin  include  lib  share

The only tasks I need to do from here on are a) stop the 9.4.1 version:

postgres@oel7:/home/postgres/ [PG1] which pg_ctl
/u01/app/postgres/product/94/db_1/bin/pg_ctl
postgres@oel7:/home/postgres/ [PG1] pg_ctl -D /u02/pgdata/PG1 stop -m fast
waiting for server to shut down.... done
server stopped
postgres@oel7:/home/postgres/ [PG1] ps -ef | grep PG1
postgres 14452 11550  0 11:06 pts/1    00:00:00 grep --color=auto PG1
postgres@oel7:/home/postgres/ [PG1] 

Once the old version is down I just can b) restart with the new binaries:

postgres@oel7:/home/postgres/ [PG1] pg_ctl -D /u02/pgdata/PG1 start
server starting
postgres@oel7:/home/postgres/ [PG1] LOG:  database system was shut down at 2015-12-01 11:06:31 CET
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started

That’s it. The new version is now 9.4.5:

(postgres@[local]:5432) [postgres] > select version();
                                                   version                                                    
--------------------------------------------------------------------------------------------------------------
 PostgreSQL 9.4.5 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit
(1 row)

Time: 20.725 ms
(postgres@[local]:5432) [postgres] > \c patch
You are now connected to database "patch" as user "postgres".
(postgres@[local]:5432)  > select count(*) from test;
 count 
-------
   100
(1 row)

Time: 104.297 ms

Usually, for minor versions, you can just install the new binaries and start the instance from there. But anyway, be sure to read the release notes before doing it.

Cet article Patching PostgreSQL to a new minor release est apparu en premier sur Blog dbi services.

The last post looked into how to upgrade PostgreSQL to a new minor version. In this post I’ll look into how to upgrade PostgreSQL to a new major version. This is not as simple as just installing the binaries and start the instance from there. For major upgrades there are two possibilities:

• dump the old version and restore into the new version by using pg_dump and pg_restore
• use pg_upgrade

I’ll only look into pg_upgrade for this post. For simplicity I’ll upgrade the 9.4.5 PostgreSQL instance from the last post to 9.5 beta2. The binaries for 9.5 beta2 are already there:

postgres@oel7:/u01/app/postgres/software/ [PG1] which pg_upgrade
/u01/app/postgres/product/95/db_b2/bin/pg_upgrade

Obviously we need to stop the current version before performing the upgrade:

postgres@oel7:/u01/app/postgres/software/ [PG1] pg_ctl stop -D /u02/pgdata/PG1
waiting for server to shut down.... done
server stopped

Then we need to create a new database cluster with the version we want to upgrade to (9.5 beta2 in this case):

postgres@oel7:/u01/app/postgres/software/ [PG1] mkdir /u02/pgdata/PG7
postgres@oel7:/u01/app/postgres/software/ [PG1] mkdir /u03/pgdata/PG7
postgres@oel7:/u01/app/postgres/software/ [PG1] mkdir /u90/arch/PG7
postgres@oel7:/u01/app/postgres/software/ [PG1] initdb -D /u02/pgdata/PG7 -X /u03/pgdata/PG7/ 
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locales
  COLLATE:  en_US.UTF-8
  CTYPE:    en_US.UTF-8
  MESSAGES: en_US.UTF-8
  MONETARY: de_CH.UTF-8
  NUMERIC:  de_CH.UTF-8
  TIME:     en_US.UTF-8
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /u02/pgdata/PG7 ... ok
fixing permissions on existing directory /u03/pgdata/PG7 ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /u02/pgdata/PG7/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    pg_ctl -D /u02/pgdata/PG7 -l logfile start

To verify the version we do a quick startup:

postgres@oel7:/u01/app/postgres/software/ [PG7] pg_ctl -D /u02/pgdata/PG7 start
server starting
postgres@oel7:/u01/app/postgres/software/ [PG7] LOG:  database system was shut down at 2015-12-01 12:10:02 CET
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started

postgres@oel7:/u01/app/postgres/software/ [PG7] sqh
Null display is "NULL".
Timing is on.
psql (9.5beta2)
Type "help" for help.

(postgres@[local]:5449) [postgres] > 

Then shut it down again:

postgres@oel7:/u01/app/postgres/software/ [PG7] pg_ctl -D /u02/pgdata/PG7 stop -m fast
waiting for server to shut down....LOG:  received fast shutdown request
LOG:  aborting any active transactions
LOG:  autovacuum launcher shutting down
LOG:  shutting down
LOG:  database system is shut down
 done
server stopped

Now we can begin with the upgrade by specifying four environment variables:

postgres@oel7:/u01/app/postgres/software/ [PG7] export PGDATAOLD=/u02/pgdata/PG1
postgres@oel7:/u01/app/postgres/software/ [PG7] export PGDATANEW=/u02/pgdata/PG7
postgres@oel7:/u01/app/postgres/software/ [PG7] export PGBINOLD=/u01/app/postgres/product/94/db_5/bin
postgres@oel7:/u01/app/postgres/software/ [PG7] export PGBINNEW=/u01/app/postgres/product/95/db_b2/bin/
postgres@oel7:/u01/app/postgres/software/ [PG7] pg_upgrade
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* system OID user data types                ok
Checking for contrib/isn with bigint-passing mismatch       ok
Creating dump of global objects                             ok
Creating dump of database schemas
                                                            ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Analyzing all rows in the new cluster                       ok
Freezing all rows on the new cluster                        ok
Deleting files from new pg_clog                             ok
Copying old pg_clog to new server                           ok
Setting next transaction ID and epoch for new cluster       ok
Deleting files from new pg_multixact/offsets                ok
Copying old pg_multixact/offsets to new server              ok
Deleting files from new pg_multixact/members                ok
Copying old pg_multixact/members to new server              ok
Setting next multixact ID and offset for new cluster        ok
Resetting WAL archives                                      ok
Setting frozenxid and minmxid counters in new cluster       ok
Restoring global objects in the new cluster                 ok
Restoring database schemas in the new cluster
                                                            ok
Creating newly-required TOAST tables                        ok
Copying user relation files
                                                            ok
Setting next OID for new cluster                            ok
Sync data directory to disk                                 ok
Creating script to analyze new cluster                      ok
Creating script to delete old cluster                       ok

Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade so,
once you start the new server, consider running:
    ./analyze_new_cluster.sh

Running this script will delete the old cluster's data files:
    ./delete_old_cluster.sh

Quite easy, isn’t it? As pointed out by pg_upgrade optimizer statistics need to be gathered as this is not done automatically:

postgres@oel7:/u01/app/postgres/software/ [PG7] pg_ctl -D /u02/pgdata/PG7 start
server starting
postgres@oel7:/u01/app/postgres/software/ [PG7] LOG:  database system was shut down at 2015-12-01 12:18:34 CET
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started

postgres@oel7:/u01/app/postgres/software/ [PG7] ./analyze_new_cluster.sh 
This script will generate minimal optimizer statistics rapidly
so your system is usable, and then gather statistics twice more
with increasing accuracy.  When it is done, your system will
have the default level of optimizer statistics.

If you have used ALTER TABLE to modify the statistics target for
any tables, you might want to remove them and restore them after
running this script because they will delay fast statistics generation.

If you would like default statistics as quickly as possible, cancel
this script and run:
    "/u01/app/postgres/product/95/db_b2/bin/vacuumdb" --all --analyze-only

vacuumdb: processing database "bi": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "db1": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "patch": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "postgres": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "template1": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "bi": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "db1": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "patch": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "postgres": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "template1": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "bi": Generating default (full) optimizer statistics
vacuumdb: processing database "db1": Generating default (full) optimizer statistics
vacuumdb: processing database "patch": Generating default (full) optimizer statistics
vacuumdb: processing database "postgres": Generating default (full) optimizer statistics
vacuumdb: processing database "template1": Generating default (full) optimizer statistics

Done

The output above proves that the “patch” database is really there but we can confirm that:

postgres@oel7:/u01/app/postgres/software/ [PG7] sqh
Null display is "NULL".
Timing is on.
psql (9.5beta2)
Type "help" for help.

(postgres@[local]:5449) [postgres] > \c patch
You are now connected to database "patch" as user "postgres".
(postgres@[local]:5449)  > select count(*) from test;
 count 
-------
   100
(1 row)

Time: 0.705 ms

And finally we can delete the old data files:

postgres@oel7:/u01/app/postgres/software/ [PG7] ./delete_old_cluster.sh 
postgres@oel7:/u01/app/postgres/software/ [PG7] ls -al /u02/pgdata/PG1
ls: cannot access /u02/pgdata/PG1: No such file or directory

That’s it.

Cet article Upgrading PostgreSQL to a new major release est apparu en premier sur Blog dbi services.

When developing an application as well as when the application is in production there is the need to identify long running queries. In Oracle one tool you might use for that is the SQL Monitor. In this post I’ll look into what PostgreSQL provides in this area.

PostgreSQL has a very strong logging system. This system can be used to log many, many server messages as well as information about sql queries. To enable to background process that captures the server log messages and redirects them to log files you need to set the logging_collector parameter to on in a first step:

(postgres@[local]:4448) [postgres] > alter system set logging_collector=on;
ALTER SYSTEM
Time: 30.390 ms
(postgres@[local]:4448) [postgres] > show logging_collector;
 logging_collector 
-------------------
 on
(1 row)

Once you have this enabled you need to tell PostgreSQL where you want to log to. This is done by setting the parameter log_directory:

(postgres@[local]:4448) [postgres] > show log_directory;
     log_directory      
------------------------
 /u02/pgdata/PG6/pg_log
(1 row)

In my case this is set to the pg_log directory which is located in my data directory. Additionally we can define how the log files will be named:

(postgres@[local]:4448) [postgres] > show log_filename;
   log_filename    
-------------------
 postgresql-%a.log
(1 row)

The place holders which can be used are the same as in strftime. The default is:

(postgres@[local]:4448) [postgres] > alter system set log_filename='postgresql-%Y-%m-%d_%H%M%S.log';
ALTER SYSTEM
Time: 45.666 ms

I recommend to set the log_rotation_age or log_rotation_size parameter so that the log-files will be rotated:

(postgres@[local]:4448) [postgres] > show log_rotation_size;
 log_rotation_size 
-------------------
 10MB
(1 row)

Time: 1.015 ms
(postgres@[local]:4448) [postgres] > show log_rotation_age;
 log_rotation_age 
------------------
 8d
(1 row)

As we now have the basic settings available lets check if we need to restart the server for the settings to go into effect:

(postgres@[local]:4448) [postgres] > select name,pending_restart 
                                       from pg_settings 
                                      where name in ('log_filename','log_rotation_size'
                                                    ,'log_rotation_age','log_destination','logging_collector');
       name        | pending_restart 
-------------------+-----------------
 log_destination   | f
 log_filename      | f
 log_rotation_age  | f
 log_rotation_size | f
 logging_collector | f
(5 rows)

(postgres@[local]:4448) [postgres] > select pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

Ok, should be fine. Lets quickly check if there is a log file with some recent messages in the directory we specified:

postgres@oel7:/home/postgres/ [PG6] ls -altr /u02/pgdata/PG6/pg_log
total 68
drwx------. 19 postgres postgres 4096 Dec  5 11:01 ..
drwx------.  2 postgres postgres   45 Dec  5 11:01 .
-rw-------.  1 postgres postgres  384 Dec  5 11:01 postgresql-2015-12-05_100103.log

Looks fine. Back to what this post is about. How can we log sql statements? One parameter in this area is log_duration. When we set this to on:

(postgres@[local]:4448) [postgres] > alter system set log_duration=on;
ALTER SYSTEM
Time: 38.978 ms
(postgres@[local]:4448) [postgres] > select name,pending_restart 
                                       from pg_settings 
                                      where name in ('log_duration');
     name     | pending_restart 
--------------+-----------------
 log_duration | f
(1 row)

Time: 2.044 ms

(postgres@[local]:4448) [postgres] > select pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

… the duration of every statement is logged to the log file:

(postgres@[local]:4448) [postgres] > create table tt ( a int );
CREATE TABLE
Time: 23.829 ms
(postgres@[local]:4448) [postgres] > insert into tt (a) values (generate_series(1,1000));
INSERT 0 1000
Time: 37.333 ms
(postgres@[local]:4448) [postgres] > select count(*) from tt;
 count 
-------
  1000
(1 row)

Having a look at the log file we can confirm that the duration is logged:

postgres@oel7:/home/postgres/ [PG6] tail /u02/pgdata/PG6/pg_log/postgresql-2015-12-05_100103.log 
2015-12-05 10:08:07.044 GMT - 4 - 4609 - postgres@postgres LOG:  statement: create table tt ( a int );
2015-12-05 10:08:07.067 GMT - 5 - 4609 - postgres@postgres LOG:  duration: 23.669 ms
2015-12-05 10:08:22.052 GMT - 6 - 4609 - postgres@postgres LOG:  duration: 37.163 ms
2015-12-05 10:08:25.519 GMT - 7 - 4609 - postgres@postgres LOG:  duration: 22.992 ms

Well, is the duration without the text of the statement very helpful? Not really and this is where the log_min_duration_statement parameter comes into the game. Setting this to any value greater than -1 logs each statement that runs longer than what you specified. If you set it to zero all statements will be logged:

(postgres@[local]:4448) [postgres] > alter system set log_min_duration_statement=0;
ALTER SYSTEM

Time: 0.188 ms
(postgres@[local]:4448) [postgres] > select pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

(postgres@[local]:4448) [postgres] > select count(*) tt;
 tt 
----
  1
(1 row)

Time: 0.680 ms

Checking the logfile again:

postgres@oel7:/home/postgres/ [PG6] tail -1 /u02/pgdata/PG6/pg_log/postgresql-2015-12-05_100103.log
2015-12-05 10:13:48.392 GMT - 8 - 4651 - postgres@postgres LOG:  duration: 0.216 ms  statement: select count(*) tt;

Much better: We have the time stamp when the statement was executed, the number of the line in the logfile where we can find the log message (8), the operating system process id and the user which executed the statement.

That’s is for now. Make yourself familiar with the various parameters of the logging system. There are plenty of things you can control and adjust.

The next post will look at another way to identify problematic statements.

Btw: The PostgreSQL version I used here is:

(postgres@[local]:4448) [postgres] > select version();
                                                     version                                                      
------------------------------------------------------------------------------------------------------------------
 PostgreSQL 9.5alpha2 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit
(1 row)

Time: 0.376 ms

Cet article SQL Monitoring in PostgreSQL (1) – the logging system est apparu en premier sur Blog dbi services.

The last post looked into how you can monitor queries using the logging system. This post will introduce pg_stat_statements.

pg_stat_statements is a module that needs to be loaded and is not available in the default configuration. Loading it is quite easy. Create the extension as usual:

postgres@oel7:/home/postgres/ [PG6] sqh
Null display is "NULL".
Timing is on.
psql (9.5alpha2)
Type "help" for help.

(postgres@[local]:4448) [postgres] > create extension pg_stat_statements;
CREATE EXTENSION
Time: 281.765 ms
(postgres@[local]:4448) [postgres] > \dx
                                     List of installed extensions
        Name        | Version |   Schema   |                        Description                        
--------------------+---------+------------+-----------------------------------------------------------
 btree_gist         | 1.1     | public     | support for indexing common datatypes in GiST
 pg_stat_statements | 1.3     | public     | track execution statistics of all SQL statements executed
 plpgsql            | 1.0     | pg_catalog | PL/pgSQL procedural language
 postgres_fdw       | 1.0     | public     | foreign-data wrapper for remote PostgreSQL servers
(4 rows)

After the extension is available we need to adjust the shared_preload_libraries parameter:

(postgres@[local]:4448) [postgres] > show shared_preload_libraries;
 shared_preload_libraries 
--------------------------
 
(1 row)

(postgres@[local]:4448) [postgres] > alter system set shared_preload_libraries='pg_stat_statements';
ALTER SYSTEM
Time: 55.005 ms

(postgres@[local]:4448) [postgres] > select name,pending_restart 
                                       from pg_settings 
                                      where name in ('shared_preload_libraries');
           name           | pending_restart 
--------------------------+-----------------
 shared_preload_libraries | f
(1 row)

Time: 1.517 ms
(postgres@[local]:4448) [postgres] > select pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

Basically pg_stat_statements can be used from now own. But there are some parameters to look at if you want to fine tune. Check the documentation for the description.

(postgres@[local]:4448) [postgres] > show pg_stat_statements.max;
-[ RECORD 1 ]----------+-----
pg_stat_statements.max | 5000

Time: 0.230 ms
(postgres@[local]:4448) [postgres] > show pg_stat_statements.track;
-[ RECORD 1 ]------------+----
pg_stat_statements.track | top

Time: 0.211 ms
(postgres@[local]:4448) [postgres] > show pg_stat_statements.track_utility;
-[ RECORD 1 ]--------------------+---
pg_stat_statements.track_utility | on

Time: 0.215 ms
(postgres@[local]:4448) [postgres] > show pg_stat_statements.save;
-[ RECORD 1 ]-----------+---
pg_stat_statements.save | on

Time: 0.212 ms

When we installed the extension a view was created with the following columns:

(postgres@[local]:4448) [postgres] > \d pg_stat_statements
          View "public.pg_stat_statements"
       Column        |       Type       | Modifiers 
---------------------+------------------+-----------
 userid              | oid              | 
 dbid                | oid              | 
 queryid             | bigint           | 
 query               | text             | 
 calls               | bigint           | 
 total_time          | double precision | 
 min_time            | double precision | 
 max_time            | double precision | 
 mean_time           | double precision | 
 stddev_time         | double precision | 
 rows                | bigint           | 
 shared_blks_hit     | bigint           | 
 shared_blks_read    | bigint           | 
 shared_blks_dirtied | bigint           | 
 shared_blks_written | bigint           | 
 local_blks_hit      | bigint           | 
 local_blks_read     | bigint           | 
 local_blks_dirtied  | bigint           | 
 local_blks_written  | bigint           | 
 temp_blks_read      | bigint           | 
 temp_blks_written   | bigint           | 
 blk_read_time       | double precision | 
 blk_write_time      | double precision | 

We can now query the view for information we are interested in, e.g.:

(postgres@[local]:4448) [postgres] > \x
Expanded display is on.
(postgres@[local]:4448) [postgres] > select userid,query,calls,total_time from pg_stat_statements;
-[ RECORD 1 ]
userid     | 10
query      | alter system set logging_collector=on;
calls      | 1
total_time | 30.13
-[ RECORD 2 ]
userid     | 10
query      | create extension pg_stat_statements;
calls      | 2
total_time | 250.54
-[ RECORD 3 ]
userid     | 10
query      | select name,pending_restart from pg_settings where name in (?,?,?,?,?);
calls      | 1
total_time | 0.627
-[ RECORD 4 ]
userid     | 10
query      | show log_rotation_size;
calls      | 1
total_time | 0.006

Additionally we can call a function which is named exactly the same as the view:

(postgres@[local]:4448) [postgres] > select * from pg_stat_statements(true);
-[ RECORD 1 ]
userid              | 10
dbid                | 13295
queryid             | 780340104
query               | alter system set logging_collector=on;
calls               | 1
total_time          | 30.13
min_time            | 30.13
max_time            | 30.13
mean_time           | 30.13
stddev_time         | 0
rows                | 0
shared_blks_hit     | 0
shared_blks_read    | 0
shared_blks_dirtied | 0
shared_blks_written | 0
local_blks_hit      | 0
local_blks_read     | 0
local_blks_dirtied  | 0
local_blks_written  | 0
temp_blks_read      | 0
temp_blks_written   | 0
blk_read_time       | 0
blk_write_time      | 0
-[ RECORD 2 ]
userid              | 10
dbid                | 13295
queryid             | 1392856018
query               | create extension pg_stat_statements;
calls               | 2
total_time          | 250.54
min_time            | 1.489
max_time            | 249.051
mean_time           | 125.27
stddev_time         | 123.781
rows                | 0
shared_blks_hit     | 1150
shared_blks_read    | 90
Time: 0.742 ms

On top of either the view or the function we can now start to troubleshoot issues with the queries the server executes. Hope this helps.

In the next post I’ll introduce pg_activity.

Cet article SQL Monitoring in PostgreSQL (2) – pg_stat_statements est apparu en premier sur Blog dbi services.

The last posts looked at how the logging system and the pg_stat_statements extension can be used to monitor sql statements in PostgreSQL. This post will introduce pg_activity which is very similar to htop.

There are some dependencies which need to be installed before we can start installing pg_activity. The first one is python. As I am on a redhat based distribution this is quite easy:

[root@oel7 ~] yum install -y python

Then we need to install psycopg which is the database adapter for PostgreSQL for the python language (Note: if you have installed PostgreSQL not in the default location edit the setup.cfg script and provide the path to pg_config. Otherwise the install will fail):

postgres@oel7:/var/tmp/ [dummy] tar -axf psycopg2-2.6.1.tar.gz
postgres@oel7:/var/tmp/ [dummy] cd psycopg2-2.6.1
postgres@oel7:/var/tmp/psycopg2-2.6.1/ [dummy] python setup.py build
postgres@oel7:/var/tmp/psycopg2-2.6.1/ [dummy] sudo python setup.py install

The next (and last) thing we need to have available is psutils, a python library for querying os statistics:

postgres@oel7:/var/tmp/ [dummy] tar -axf psutil-3.3.0.tar.gz
postgres@oel7:/var/tmp/ [dummy] cd psutil-3.3.0
postgres@oel7:/var/tmp/ [dummy] sudo python setup.py install

That’s it. Now we can install pg_activity:

postgres@oel7:/var/tmp/ [dummy] unzip pg_activity-master.zip
postgres@oel7:/var/tmp/ [dummy] cd pg_activity-master
postgres@oel7:/var/tmp/ [dummy] sudo python setup.py install

Quite easy. Lets see what we can do with it. If you are locally on the server where your PostgreSQL instance runs you can just start pg_activity (I fired a sql statement so that you can see at least one sql in the screenshot):

postgres@oel7:/home/postgres/ [PG3] pg_activity

There is a nice summary on the top like in top/htop. The different statements which are currently executing are displayed below.

Hitting “h” for help shows you the various options:

The “F1/2/3″ switches are very nice when you want to display blocking queries only, running queries only or waiting queries only. Another great features is that you do not need to install pg_activity on the server where PostgreSQL is running. The same connections options as in e.g. psql are there so that you can connect to any remote PostgreSQL instance you have access to:

postgres@oel7:/home/postgres/ [PG3] pg_activity --help
Usage: pg_activity [options]

htop like application for PostgreSQL server activity monitoring.

Options:
  --version             show program's version number and exit
  -U USERNAME, --username=USERNAME
                        Database user name (default: "postgres").
  -p PORT, --port=PORT  Database server port (default: "5432").
  -h HOSTNAME, --host=HOSTNAME
                        Database server host or socket directory
                        (default: "localhost").
  -d DBNAME, --dbname=DBNAME
                        Database name to connect to (default: "postgres").
  -C, --no-color        Disable color usage.
  --blocksize=BLOCKSIZE
                        Filesystem blocksize (default: 4096)
  --rds                 Enable support for AWS RDS
  --help                Show this help message and exit.
  --debug               Enable debug mode for traceback tracking.

  Display Options, you can exclude some columns by using them :
    --no-database       Disable DATABASE.
    --no-user           Disable USER.
    --no-client         Disable CLIENT.
    --no-cpu            Disable CPU%.
    --no-mem            Disable MEM%.
    --no-read           Disable READ/s.
    --no-write          Disable WRITE/s.
    --no-time           Disable TIME+.
    --no-wait           Disable W.

Conclusion: pg_activity is small but very useful tool for monitoring an PostgreSQL instance. In the next posts I’ll look into some more feature rich monitoring solutions that are around for monitoring PostgreSQL instances.

Cet article SQL Monitoring in PostgreSQL (3) – pg_activity est apparu en premier sur Blog dbi services.

The last posts introduced the logging system, pg_stat_statements and pg_activity. All of these can be used to monitor sql statements the PostgreSQL server is executing. In this post I’ll look into pgcluu: PostgreSQL Cluster utilization! This is a more complete monitoring solution as it is not only focused on sql statements but gives you information about the database cluster itself and other useful stuff.

All you need to run pgcluu is a modern perl distribution but this should be available if you are on a recent operating system. If you not only want to have statistics about a PostgreSQL instance but also want to have OS statistics you’ll need the sysstat package in addition (this should be available for your distribution). You can install pgcluu on the server where the PostgreSQL instance you want to monitor runs (as I will do for this post) or on a remote host. Installation is quite easy;

postgres@oel7:/var/tmp/ [dummy] tar -axf pgcluu-2.4.tar.gz
postgres@oel7:/var/tmp/ [dummy] cd pgcluu-2.4
postgres@oel7:/var/tmp/ [dummy] perl Makefile.PL
postgres@oel7:/var/tmp/ [dummy] make && sudo make install

pgcluu is divided into two parts:

• The collector which is responsible for collecting the statistics: pgcluu_collectd
• The report generator which generates the reports out of the files the collector generated: pgcluu

To collect statistics start the pgcluu_collectd script as deamon:

postgres@oel7:/home/postgres/ [PG2] mkdir /var/tmp/test_stats
postgres@oel7:/home/postgres/ [PG2] pgcluu_collectd -D -i 60 /var/tmp/test_stats/
postgres@oel7:/home/postgres/ [PG2] LOG: Detach from terminal with pid: 10423

This will collect statistics for the PostgreSQL instance you have the environment set for every 60 seconds and stores the results in the /var/tmp/test_stats/ directory:

postgres@oel7:/var/tmp/pgcluu-2.4/ [postgres] ls -la /var/tmp/test_stats/
total 196
drwxrwxr-x. 2 postgres postgres  4096 Dec  7 16:16 .
drwxrwxrwt. 4 root     root        64 Dec  7 16:05 ..
-rw-rw-r--. 1 postgres postgres  8280 Dec  7 16:16 pg_class_size.csv
-rw-rw-r--. 1 postgres postgres   274 Dec  7 16:16 pg_database_size.csv
-rw-rw-r--. 1 postgres postgres  4214 Dec  7 16:15 pg_hba.conf
-rw-rw-r--. 1 postgres postgres  1636 Dec  7 16:15 pg_ident.conf
-rw-rw-r--. 1 postgres postgres 30694 Dec  7 16:16 pg_settings.csv
-rw-rw-r--. 1 postgres postgres     0 Dec  7 16:15 pg_stat_connections.csv
-rw-rw-r--. 1 postgres postgres   333 Dec  7 16:16 pg_stat_database.csv
-rw-rw-r--. 1 postgres postgres  2682 Dec  7 16:16 pg_statio_user_indexes.csv
-rw-rw-r--. 1 postgres postgres  1040 Dec  7 16:16 pg_statio_user_sequences.csv
-rw-rw-r--. 1 postgres postgres  1582 Dec  7 16:16 pg_statio_user_tables.csv
-rw-rw-r--. 1 postgres postgres  1004 Dec  7 16:16 pg_stat_locks.csv
-rw-rw-r--. 1 postgres postgres   764 Dec  7 16:16 pg_stat_unused_indexes.csv
-rw-rw-r--. 1 postgres postgres  2682 Dec  7 16:16 pg_stat_user_indexes.csv
-rw-rw-r--. 1 postgres postgres  1430 Dec  7 16:16 pg_stat_user_tables.csv
-rw-rw-r--. 1 postgres postgres     0 Dec  7 16:15 pg_tablespace_size.csv
-rw-rw-r--. 1 postgres postgres   343 Dec  7 16:15 postgresql.auto.conf
-rw-rw-r--. 1 postgres postgres 24821 Dec  7 16:15 postgresql.conf
-rw-rw-r--. 1 postgres postgres 56896 Dec  7 16:16 sar_stats.dat
-rw-rw-r--. 1 postgres postgres  3111 Dec  7 16:15 sysinfo.txt

After some time, when hopefully there was some activity in the PostgreSQL instance, stop the deamon:

postgres@oel7:/home/postgres/ [PG2] pgcluu_collectd -k
OK: pgcluu_collectd exited with value 0
postgres@oel7:/home/postgres/ [PG2] 

Once we have some statistics collected we can generate a report:

postgres@oel7:/home/postgres/ [PG2] mkdir /var/tmp/test_report/
postgres@oel7:/home/postgres/ [PG2] pgcluu -o /var/tmp/test_report/ /var/tmp/test_stats/

The report can be viewed in any modern browser that supports javascript and css. Once you open the index.html you are presented with an overview of the system:

On the top there is a menu which allows you to navigate to various reports of your operating system and the PostgreSQL instance, e.g. the cluster:

Reports for various OS statistics are available through the system menu:

To get the best of out this you probably should let the collector running all the time and use the build in rotation functionality:

postgres@oel7:/home/postgres/ [PG2] pgcluu_collectd -D -i 60 --rotate-daily /var/tmp/test_stats/  

Having statistics available for each day of the week helps a lot in troubleshooting. Reports can than be scheduled automatically by cron or any other scheduler.

Conclusion: pgcluu is an easy to setup and easy to use monitoring solution for PostgreSQL instances. Spend some time in thinking about how to collect, how to report and how to archive the reports and you’ll have plenty of information to troubleshoot and to plan capacity.

Cet article Monitoring tools for PostgreSQL – pgcluu est apparu en premier sur Blog dbi services.

The last posts introduced the logging system, pg_stat_statements, pg_activity and pgcluu. This post will look at POWA: PostgreSQL Workload Analyzer.

For getting the most out of POWA the following extension should be installed in the PostgreSQL instance you want to monitor:

• pg_stat_statements (see last post)
• pg_stat_kcache: gathers statistics about reads and writes done by the file system layer
• pg_qualstats: gathers statistics of predicates found in where statements and join clauses
• btree_gist: provides GiST index operator classes that implement B-tree equivalent behavior for various data types

As pg_stat_statements is already installed in my PostgreSQL instance lets start by installing the pg_stat_kcache extension.

postgres@oel7:/var/tmp/ [PG3] unzip pg_stat_kcache-master.zip
postgres@oel7:/var/tmp/ [PG3] cd pg_stat_kcache-master
postgres@oel7:/var/tmp/pg_stat_kcache-master/ [PG3] make
postgres@oel7:/var/tmp/pg_stat_kcache-master/ [PG3] make install

As usual: quite easy. As with pg_stat_statements we need to adjust the shared_preload_libraries parameter to have the extension loaded:

(postgres@[local]:4445) [postgres] > show shared_preload_libraries;
 shared_preload_libraries 
--------------------------
 pg_stat_statements
(1 row)

Time: 0.230 ms
(postgres@[local]:4445) [postgres] > alter system set shared_preload_libraries=pg_stat_statements,pg_stat_kcache;
ALTER SYSTEM
Time: 2.995 ms

After the PostgreSQL instance was restarted:

postgres@oel7:/var/tmp/pg_stat_kcache-master/ [PG3] pg_ctl -D /u02/pgdata/PG3/ stop -m fast
postgres@oel7:/var/tmp/pg_stat_kcache-master/ [PG3] pg_ctl -D /u02/pgdata/PG3/ start

… the extension can be created:

(postgres@[local]:4445) [postgres] > show shared_preload_libraries;
     shared_preload_libraries      
-----------------------------------
 pg_stat_statements,pg_stat_kcache
(1 row)

(postgres@[local]:4445) [postgres] > create extension pg_stat_kcache;
CREATE EXTENSION
Time: 68.483 ms

(postgres@[local]:4445) [postgres] > \dx
                                     List of installed extensions
        Name        | Version |   Schema   |                        Description                        
--------------------+---------+------------+-----------------------------------------------------------
 pg_stat_kcache     | 2.0.2   | public     | Kernel cache statistics gathering
 pg_stat_statements | 1.2     | public     | track execution statistics of all SQL statements executed
 plpgsql            | 1.0     | pg_catalog | PL/pgSQL procedural language
(3 rows)

The next, and final, extension which should be installed is pg_qualstats. The procedure is almost the same:

postgres@oel7:/var/tmp/ [PG3] unzip pg_qualstats-master.zip
postgres@oel7:/var/tmp/ [PG3] cd pg_qualstats-master
postgres@oel7:/var/tmp/pg_qualstats-master/ [PG3] make
postgres@oel7:/var/tmp/pg_qualstats-master/ [PG3] make install

Again we’ll need to adjust shared_preload_libraries:

(postgres@[local]:4445) [postgres] > show shared_preload_libraries;
     shared_preload_libraries      
-----------------------------------
 pg_stat_statements,pg_stat_kcache
(1 row)

Time: 0.215 ms
(postgres@[local]:4445) [postgres] > alter system set shared_preload_libraries=pg_stat_statements,pg_stat_kcache,pg_qualstats;
ALTER SYSTEM
Time: 4.692 ms

Then restart the server:

postgres@oel7:/var/tmp/pg_stat_kcache-master/ [PG3] pg_ctl -D /u02/pgdata/PG3/ stop -m fast
postgres@oel7:/var/tmp/pg_stat_kcache-master/ [PG3] pg_ctl -D /u02/pgdata/PG3/ start

Finally create the extension:

(postgres@[local]:4445) [postgres] > show shared_preload_libraries;
             shared_preload_libraries             
--------------------------------------------------
 pg_stat_statements, pg_stat_kcache, pg_qualstats
(1 row)

Time: 0.285 ms
(postgres@[local]:4445) [postgres] > create extension pg_qualstats;
CREATE EXTENSION
Time: 143.439 ms
(postgres@[local]:4445) [postgres] > \dx
                                     List of installed extensions
        Name        | Version |   Schema   |                        Description                        
--------------------+---------+------------+-----------------------------------------------------------
 pg_qualstats       | 0.0.7   | public     | An extension collecting statistics about quals
 pg_stat_kcache     | 2.0.2   | public     | Kernel cache statistics gathering
 pg_stat_statements | 1.2     | public     | track execution statistics of all SQL statements executed
 plpgsql            | 1.0     | pg_catalog | PL/pgSQL procedural language
(4 rows)

The btree_gist extension is there by default and we just need to add it:

postgres@[local]:4445) [postgres] > \dx
                                     List of installed extensions
        Name        | Version |   Schema   |                        Description                        
--------------------+---------+------------+-----------------------------------------------------------
 pg_qualstats       | 0.0.7   | public     | An extension collecting statistics about quals
 pg_stat_kcache     | 2.0.2   | public     | Kernel cache statistics gathering
 pg_stat_statements | 1.2     | public     | track execution statistics of all SQL statements executed
 plpgsql            | 1.0     | pg_catalog | PL/pgSQL procedural language
(4 rows)

(postgres@[local]:4445) [postgres] > create extension btree_gist;
CREATE EXTENSION
Time: 21.112 ms
(postgres@[local]:4445) [postgres] > \dx
                                     List of installed extensions
        Name        | Version |   Schema   |                        Description                        
--------------------+---------+------------+-----------------------------------------------------------
 btree_gist         | 1.0     | public     | support for indexing common datatypes in GiST
 pg_qualstats       | 0.0.7   | public     | An extension collecting statistics about quals
 pg_stat_kcache     | 2.0.2   | public     | Kernel cache statistics gathering
 pg_stat_statements | 1.2     | public     | track execution statistics of all SQL statements executed
 plpgsql            | 1.0     | pg_catalog | PL/pgSQL procedural language
(5 rows)

Having all the requirements available we now start by installing powa-archivist. This is the POWA extension that gathers the performance statistics. The usual steps apply:

postgres@oel7:/var/tmp/ [PG3] unzip powa-archivist-master.zip
postgres@oel7:/var/tmp/ [PG3] cd powa-archivist-master
postgres@oel7:/var/tmp/powa-archivist-master/ [PG3] make
postgres@oel7:/var/tmp/powa-archivist-master/ [PG3] make install

Again, adjust shared_preload_libraries:

(postgres@[local]:4445) [postgres] > show shared_preload_libraries;
             shared_preload_libraries             
--------------------------------------------------
 pg_stat_statements, pg_stat_kcache, pg_qualstats
(1 row)

Time: 0.243 ms
(postgres@[local]:4445) [postgres] > alter system set shared_preload_libraries=pg_stat_statements, pg_stat_kcache, pg_qualstats, powa;
ALTER SYSTEM
Time: 69.219 ms

Restart the server:

postgres@oel7:/var/tmp/pg_stat_kcache-master/ [PG3] pg_ctl -D /u02/pgdata/PG3/ stop -m fast
postgres@oel7:/var/tmp/pg_stat_kcache-master/ [PG3] pg_ctl -D /u02/pgdata/PG3/ start

Create the extension:

(postgres@[local]:4445) [postgres] > \dx
                                     List of installed extensions
        Name        | Version |   Schema   |                        Description                        
--------------------+---------+------------+-----------------------------------------------------------
 btree_gist         | 1.0     | public     | support for indexing common datatypes in GiST
 pg_qualstats       | 0.0.7   | public     | An extension collecting statistics about quals
 pg_stat_kcache     | 2.0.2   | public     | Kernel cache statistics gathering
 pg_stat_statements | 1.2     | public     | track execution statistics of all SQL statements executed
 plpgsql            | 1.0     | pg_catalog | PL/pgSQL procedural language
(5 rows)

(postgres@[local]:4445) [postgres] > create extension powa;
CREATE EXTENSION
Time: 742.831 ms
(postgres@[local]:4445) [postgres] > \dx
                                     List of installed extensions
        Name        | Version |   Schema   |                        Description                        
--------------------+---------+------------+-----------------------------------------------------------
 btree_gist         | 1.0     | public     | support for indexing common datatypes in GiST
 pg_qualstats       | 0.0.7   | public     | An extension collecting statistics about quals
 pg_stat_kcache     | 2.0.2   | public     | Kernel cache statistics gathering
 pg_stat_statements | 1.2     | public     | track execution statistics of all SQL statements executed
 plpgsql            | 1.0     | pg_catalog | PL/pgSQL procedural language
 powa               | 3.0.0   | public     | PostgreSQL Workload Analyser-core
(6 rows)

At this point in time it is advisable to create a dedicated database for the powa repository and add all the extensions:

(postgres@[local]:4445) [postgres] > create database powa;
CREATE DATABASE
Time: 1664.653 ms
(postgres@[local]:4445) [postgres] > \c powa
You are now connected to database "powa" as user "postgres".
(postgres@[local]:4445) [powa] > create extension pg_stat_statements;
CREATE EXTENSION
Time: 25.448 ms
(postgres@[local]:4445) [powa] > create extension btree_gist;
CREATE EXTENSION
Time: 134.281 ms
(postgres@[local]:4445) [powa] > create extension pg_qualstats;
CREATE EXTENSION
Time: 25.683 ms
(postgres@[local]:4445) [powa] > create extension pg_stat_kcache;
CREATE EXTENSION
Time: 53.798 ms
(postgres@[local]:4445) [powa] > create extension powa;
CREATE EXTENSION
Time: 98.410 ms
(postgres@[local]:4445) [powa] > \dx
                                     List of installed extensions
        Name        | Version |   Schema   |                        Description                        
--------------------+---------+------------+-----------------------------------------------------------
 btree_gist         | 1.0     | public     | support for indexing common datatypes in GiST
 pg_qualstats       | 0.0.7   | public     | An extension collecting statistics about quals
 pg_stat_kcache     | 2.0.2   | public     | Kernel cache statistics gathering
 pg_stat_statements | 1.2     | public     | track execution statistics of all SQL statements executed
 plpgsql            | 1.0     | pg_catalog | PL/pgSQL procedural language
 powa               | 3.0.0   | public     | PostgreSQL Workload Analyser-core
(6 rows)

There are some configuration parameters that allow you to control the powa extension:

(postgres@[local]:4445) [postgres] > show powa.frequency;
 powa.frequency 
----------------
 5min
(1 row)

Time: 0.319 ms
(postgres@[local]:4445) [postgres] > show powa.retention;
 powa.retention 
----------------
 1d
(1 row)

Time: 0.241 ms
(postgres@[local]:4445) [postgres] > show powa.database;
 powa.database 
---------------
 powa
(1 row)

Time: 0.241 ms
(postgres@[local]:4445) [postgres] > show powa.coalesce;
 powa.coalesce 
---------------
 100
(1 row)

Time: 0.362 ms

So far for the work to be done inside the PostgreSQL instance. No we need the web interface. In general you can install the web interface anywhere. I’ll be doing it on the same host by installing the requirements first:

postgres@oel7: [PG3] sudo yum install python-pip python-devel

After that pip can be used to install the web interface:

postgres@oel7: [PG3] sudo pip install powa-web

We need to create small configuration file for the web interface:

postgres@oel7: [PG3] sudo echo "servers={
  'main': {
    'host': 'localhost',
    'port': '5432',
    'database': 'powa'
  }
}
cookie_secret=\"A_SECRET\" " > /etc/powa-web.conf

Once this is available the web interface can be started:

postgres@oel7:/var/tmp/powa-archivist-master/ [PG3] powa-web

You should be able to access the interface at port 8888:

After some time (you’ll need to give powa some time to collect) the dashboard will be populated:

If you select a database you can scroll down to the list of sql statements:

Clicking on one of these gives nice graphs (the following are all graphs for one statement):

Conclusion: POWA is a very nice tool for gathering and displaying statistics around a PostgreSQL instance. Especially that you can store all the statistics in a separate database and can control on how long you want to keep them makes it a very good choice. Traveling back in time to troubleshoot issues becomes very easy.

Cet article Monitoring tools for PostgreSQL – POWA est apparu en premier sur Blog dbi services.

It didn’t made it into the upcoming PostgreSQL 9.5 release but it will probably be coming in 9.6: Parallel sequential scans. If you take a look at the development documentation some new parameters showed up:

• max_parallel_degree: The maximum degree of parallelism for an individual parallel operation.
• parallel_tuple_cost: Sets the planner’s estimate of the cost of transferring a tuple from a parallel worker process to another process. The default is 0.1
• parallel_setup_cost: Sets the planner’s estimate of the cost of launching parallel worker processes. The default is 1000.

Lets do a quick test. If you want to setup a PostgreSQL instance from the nightly development snapshot you can follow this post.

As you can see below I am running a 9.6 development version:

postgres@oel7:/home/postgres/ [dummy] install/bin/psql -p 6666 postgres
Null display is "NULL".
Timing is on.
psql (9.6devel)
Type "help" for help.

For testing parallel sequential scans we need to raise the max_parallel_degree parameter which is zero by default:

(postgres@[local]:6666) [postgres] > show max_parallel_degree;
 max_parallel_degree 
---------------------
 0
(1 row)

Time: 0.302 ms
(postgres@[local]:6666) [postgres] > alter system set max_parallel_degree=5;
ALTER SYSTEM
Time: 5.509 ms

(postgres@[local]:6666) [postgres] > select pg_reload_conf();
LOG:  received SIGHUP, reloading configuration files
LOG:  parameter "max_parallel_degree" changed to "5"
 pg_reload_conf 
----------------
 t
(1 row)

Time: 19.252 ms

Note: The log messages displayed when the configuration was reloaded are new, too.

Now lets create some data to test with:

(postgres@[local]:6666) [postgres] > create table tt ( a int, b int );
CREATE TABLE

Time: 427.362 ms(postgres@[local]:6666) 
[postgres] > insert into tt (a,b) 
                values ( generate_series(1,1000000),generate_series(1,1000000));
INSERT 0 1000000
Time: 4912.641 ms

So, lets see parallel squential scan in action:

(postgres@[local]:6666) [postgres] > explain analyze select count(*) from tt where a between 5 and 50000;
                                                         QUERY PLAN                                                          
-----------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=13800.29..13800.30 rows=1 width=0) (actual time=94.067..94.068 rows=1 loops=1)
   ->  Gather  (cost=1000.00..13677.60 rows=49076 width=0) (actual time=0.655..91.073 rows=49996 loops=1)
         Number of Workers: 2
         ->  Parallel Seq Scan on tt  (cost=0.00..7770.00 rows=49076 width=0) (actual time=8.465..33.548 rows=16665 loops=3)
               Filter: ((a >= 5) AND (a <= 50000))
               Rows Removed by Filter: 316668 ab
 Planning time: 0.280 ms
 Execution time: 94.127 ms
(8 rows)

Note the “Number of Workers” and “Parallel Seq Scan” information in the execution plan. Looks promising … More posts about testing this in more detail to come.

Cet article Parallel sequential scans are coming to PostgreSQL est apparu en premier sur Blog dbi services.

As I could not find any information about this error when I faced it this is just a short hint: If you try to use pg_dump or pg_dumpall out of a PPAS installation to dump or dumpall a community version of PostgreSQL you probably get this:

pg_dump: [archiver (db)] query failed: ERROR: schema "sys" does not exist
pg_dump: [archiver (db)] query was: SET search_path = pg_catalog, sys, dbo

The solution is quite easy but you have to be aware of: The pg_dump and pg_dumpall utilities shipped with PPAS are not compatible with community PostgreSQL and therefore you’ll have to install the PostgreSQL community binaries and use the utilities from there to succeed (even if PPAS is installed in PostgreSQL mode).

Cet article pg_dump: [archiver (db)] query failed: ERROR: schema “sys” does not exist est apparu en premier sur Blog dbi services.