PostgreSQL 9.5 introduces a new parameter which is called: cluster_name. So, what is this good for?

Imagine you have two (or even more) PostgreSQL clusters running on the same host. When looking at the operating system processes this might look like this:

postgres@oel7:/home/postgres/ [PG6] ps -ef | grep "postgres:"
postgres  2325  2324  0 14:01 ?        00:00:00 postgres: logger process   
postgres  2327  2324  0 14:01 ?        00:00:00 postgres: checkpointer process   
postgres  2328  2324  0 14:01 ?        00:00:00 postgres: writer process   
postgres  2329  2324  0 14:01 ?        00:00:00 postgres: wal writer process   
postgres  2330  2324  0 14:01 ?        00:00:00 postgres: autovacuum launcher process   
postgres  2331  2324  0 14:01 ?        00:00:00 postgres: archiver process   last was 0000000100000002000000B2
postgres  2332  2324  0 14:01 ?        00:00:00 postgres: stats collector process   
postgres  2709  2708  0 14:01 ?        00:00:00 postgres: logger process   
postgres  2711  2708  0 14:01 ?        00:00:00 postgres: checkpointer process   
postgres  2712  2708  0 14:01 ?        00:00:00 postgres: writer process   
postgres  2713  2708  0 14:01 ?        00:00:00 postgres: wal writer process   
postgres  2714  2708  0 14:01 ?        00:00:00 postgres: autovacuum launcher process   
postgres  2715  2708  0 14:01 ?        00:00:00 postgres: stats collector process   
postgres  3191  2771  0 14:09 pts/0    00:00:00 grep --color=auto postgres:

It is not possible to see immediately to which instance which process belongs to. Sure, you can look at the postmaster:

postgres@oel7:/home/postgres/ [PG6] ps -ef | grep "postgres -D"
postgres  2324     1  0 14:01 ?        00:00:00 /u01/app/postgres/product/95/db_2/bin/postgres -D /u02/pgdata/PG6
postgres  2708     1  0 14:01 ?        00:00:00 /u01/app/postgres/product/95/db_b2/bin/postgres -D /u02/pgdata/PG7

… and from there make your way to the child processes. But this is not very convenient. This is where cluster_name comes in. Now, with PostgreSQL 9.5 (the community just released RC1), you can set this parameter to value of your choice:

(postgres@[local]:4448) [postgres] > alter system set cluster_name='my_cluster';
ALTER SYSTEM
Time: 29.964 ms
(postgres@[local]:4448) [postgres] > select pending_restart 
                                       from pg_settings 
                                      where name = 'cluster_name';
 pending_restart 
-----------------
 t
(1 row)

As you can see from the last select above: for this parameter to have any effect we need to restart the cluster:

postgres@oel7:/home/postgres/ [PG6] pg_ctl -D /u02/pgdata/PG6/ restart
waiting for server to shut down.... done
server stopped
server starting
postgres@oel7:/home/postgres/ [PG6] 

From now on all the processes belonging to this instance can be easily identified:

postgres@oel7:/home/postgres/ [PG6] ps -ef | grep my_cluster
postgres  3336  3335  0 14:18 ?        00:00:00 postgres: my_cluster: logger process   
postgres  3338  3335  0 14:18 ?        00:00:00 postgres: my_cluster: checkpointer process   
postgres  3339  3335  0 14:18 ?        00:00:00 postgres: my_cluster: writer process   
postgres  3340  3335  0 14:18 ?        00:00:00 postgres: my_cluster: wal writer process   
postgres  3341  3335  0 14:18 ?        00:00:00 postgres: my_cluster: autovacuum launcher process   
postgres  3342  3335  0 14:18 ?        00:00:00 postgres: my_cluster: archiver process   
postgres  3343  3335  0 14:18 ?        00:00:00 postgres: my_cluster: stats collector process   
postgres  3360  2771  0 14:19 pts/0    00:00:00 grep --color=auto my_cluster

A little, but very nice addition which made it into PostgreSQL 9.5.

Cet article Setting your cluster name in PostgreSQL 9.5 est apparu en premier sur Blog dbi services.

Back in 2012 I wrote a small blog post about watching the results of commands in Linux. Well, the same can be done in psql:

(postgres@[local]:4448) [postgres] > create table t1 ( a int );
CREATE TABLE
(postgres@[local]:4448) [postgres] > insert into t1 values ( generate_series ( 1, 10));
INSERT 0 10
(postgres@[local]:4448) [postgres] > select count(*) from t1;
 count 
-------
    10
(1 row)

(postgres@[local]:4448) [postgres] > \watch 
Watch every 2s	Mon Dec 21 07:34:35 2015

 count 
-------
    10
(1 row)

Watch every 2s	Mon Dec 21 07:34:37 2015

 count 
-------
    10
(1 row)

Watch every 2s	Mon Dec 21 07:34:39 2015

 count 
-------
    10
(1 row)

Isn’t that cool? Timing information can be displayed as well:

(postgres@[local]:4448) [postgres] > \timing
Timing is on.
(postgres@[local]:4448) [postgres] > select count(*) from t1;
 count 
-------
    10
(1 row)

Time: 1.355 ms
(postgres@[local]:4448) [postgres] > \watch
Watch every 2s	Mon Dec 21 07:35:50 2015

 count 
-------
    10
(1 row)

Time: 0.246 ms
Watch every 2s	Mon Dec 21 07:35:52 2015

 count 
-------
    10
(1 row)

Time: 0.264 ms
Watch every 2s	Mon Dec 21 07:35:54 2015

 count 
-------
    10
(1 row)

Time: 0.267 ms

Btw: You can adjust the interval at which the statement is executed by specifying the seconds after the “\watch” command:

(postgres@[local]:4448) [postgres] > select count(*) from t1;
 count 
-------
    10
(1 row)

Time: 0.254 ms
(postgres@[local]:4448) [postgres] > \watch 3
Watch every 3s	Mon Dec 21 07:36:51 2015

 count 
-------
    10
(1 row)

Time: 0.250 ms
Watch every 3s	Mon Dec 21 07:36:54 2015

 count 
-------
    10
(1 row)

Time: 0.268 ms
Watch every 3s	Mon Dec 21 07:36:57 2015

 count 
-------
    10
(1 row)

Time: 0.267 ms

Another small, but very useful feature. Have fun watching your statements.

Cet article Watching sql statements in PostgreSQL est apparu en premier sur Blog dbi services.

While setting up a PostgreSQL infrastructure at customer today my colleague Pierre Sicot and I discussed on how time consuming it is to setup standby databases if you compare different vendors. Lets make a challenge out of this. This is the baseline:

1. Install the product from scratch
2. Create a master database
3. Create a hot standby database (on the same host)
4. The standby database needs to be open read only
5. Create a table containing one record on the master
6. Verify that the table and the record is there on the standby

For this PostgreSQL demo I used the just released PostgreSQL 9.5 RC1.

This is the script:

postgres@oel7:/home/postgres/ [dummy] cat create_standby_from_scratch.sh 
#!/bin/bash

BASE=$HOME
SOURCE="${HOME}/postgresql-9.5rc1.tar.bz2"
PGHHOME="${HOME}/pg95rc"

MASTERDATADIR="${HOME}/pgmaster"
STANDBYDATADIR="${HOME}/pgstandby"

PATH=${PHOME}/bin:$PATH
export PATH

pg_ctl stop -D ${HOME}/pgstandby -m fast
pg_ctl stop -D ${HOME}/pgmaster -m fast

rm -rf ${PGHHOME}
rm -rf ${HOME}/postgresql-9.5rc1}
rm -rf ${MASTERDATADIR}
rm -rf ${STANDBYDATADIR}

# configure, compile and install
cd ${HOME}
tar -axf ${SOURCE}
cd ${HOME}/postgresql-9.5rc1
./configure --prefix=${PGHHOME} 
make world
make install

PATH=${PHOME}/bin:$PATH
export PATH

# initialze the master postgresql cluster
initdb -D ${MASTERDATADIR}
echo "listen_addresses = '*'" > ${MASTERDATADIR}/postgresql.conf
echo "port=7777" >> ${MASTERDATADIR}/postgresql.conf
echo "wal_level='hot_standby'" >> ${MASTERDATADIR}/postgresql.conf
echo "max_replication_slots=3" >> ${MASTERDATADIR}/postgresql.conf
echo "max_wal_senders=3" >> ${MASTERDATADIR}/postgresql.conf
pg_ctl start -D ${MASTERDATADIR}
sleep 2
psql -p 7777 -c "select * from pg_create_physical_replication_slot('test');" postgres
pg_ctl stop -D ${MASTERDATADIR} -m fast

# prepare the primary
mkdir ${STANDBYDATADIR}
chmod 700 ${STANDBYDATADIR}
cp -pr ${MASTERDATADIR}/* ${STANDBYDATADIR}/
echo "standby_mode = 'on'" > ${STANDBYDATADIR}/recovery.conf
echo "primary_slot_name = 'test' " >> ${STANDBYDATADIR}/recovery.conf 
echo "primary_conninfo = 'user=postgres port=7777 host=localhost'" >> ${STANDBYDATADIR}/recovery.conf
echo "recovery_target_timeline = 'latest'" >> ${STANDBYDATADIR}/recovery.conf
echo "port=7778" >> ${STANDBYDATADIR}/postgresql.conf
echo "hot_standby='on'" >> ${STANDBYDATADIR}/postgresql.conf
echo "host    replication     postgres        ::1/128            trust" >> ${MASTERDATADIR}/pg_hba.conf

pg_ctl start -D ${MASTERDATADIR}
sleep 2
pg_ctl start -D ${STANDBYDATADIR}

## create a sample table on the master
psql -p 7777 -c "create table t1 ( a int );" postgres
psql -p 7777 -c "insert into t1 values (1);" postgres
sleep 1
# check the table on the standby
psql -p 7778 -c "select * from t1;" postgres

And this is the result (in a VirtualBox VM on my notebook, not a big server):

postgres@oel7:/home/postgres/ [dummy] time ./create_standby_from_scratch.sh
...
LOG:  autovacuum launcher started
server starting
CREATE TABLE
INSERT 0 1
LOG:  database system was shut down at 2015-12-22 19:41:20 GMT
LOG:  entering standby mode
LOG:  consistent recovery state reached at 0/1718740
LOG:  record with zero length at 0/1718740
LOG:  database system is ready to accept read only connections
LOG:  started streaming WAL from primary at 0/1000000 on timeline 1
LOG:  redo starts at 0/1718740
 a 
---
 1
(1 row)
...
real	0m50.061s
user	0m8.908s
sys	0m4.796s

Less than a minute. The battle is open

Cet article How fast can you setup a standby database with PostgreSQL? est apparu en premier sur Blog dbi services.

Today I had a look at PostgreSQL in Amazon RDS. Once you have an Amazon AWS account (this requires a valid credit card for verification) setup bringing up a PostgreSQL instance is pretty easy. The first 12 months are free but be sure to read about the restrictions and what is terminated by default and what is not.

Once your AWS account is ready you can login to the management console which lists all the services amazon offers:

The service I am interested in is RDS (Relational database service):

Once selected the “Instances” menu on the left brings up the instance management page:

Of course I have to click on the “Launch DB Instance” button to bring up a new instance:

Once PostgreSQL is selected take care on the next screen:

Be sure you take the “free” option. Quite a few PostgreSQL versions are available:

Of course we’ll use the latest one and specify the other details (don’t worry about my username. When this post is published the instance will already be deleted ):

Make sure that you always see the “Your current selection is eligible for the free tier.” message on the left side of the screen. If you do not see this you selected an option that prevents you from creation a free instance. Backups and maintenance details can be specified in the next screens:

And launch…ups:

Not so bad Lets correct this and launch again:

A few minutes later the instance is ready:

For sure the first thing I want to do is to connect. How do I do this? Let’s try to connect to the “endpoint” from my local machine:

… and I am in. So far for the setup. In the next post I’ll look at how to restrict access to this instance to specific IP-addresses to strengthen security.

Cet article PostgreSQL on Amazon RDS – Setting up the beast est apparu en premier sur Blog dbi services.

In the last post I looked at how to bring up a PostgreSQL instance in the Amazon cloud. In this post I’ll look into how to restrict access to this instance to specific IP addresses to strengthen security. If you think about moving things to the cloud you for sure want to make the stuff accessible to restricted people or systems only.

On the left side there is a “Security Groups” menu which sounds like what we want:

Seems I have to go to the EC2 console, so lets do that:

When a PostgreSQL instance is created a default security group is created with allows the machine with my local IP address to access the instance. No other clients are accepted by default:

If I adjust this to something else I am no longer able to connect:

The connection attempt just hangs…

… and timeouts after a while:

In fact these are firewall rules and you can choose among various protocols and types:

For the “source” there is the choice between a custom ip address, my own ip address and “hey, I don’t care”, which is “Anyone”:

The same can be down for outbound connections:

Quite easy and powerful and of course you can create new security groups:

When there are discussions about putting things into the cloud security is always a topic and it seems that Amazon did a good job here. At least for these simple tests I could prevent access to my instance very easy and fast and I am able to adjust the security settings to my needs.

In the next post I’ll look at how I can configure my PostgreSQL instance. Only when I am able to adjust the configuration for the workloads I expect the offering is really usable. Stay tuned …

Cet article PostgreSQL on Amazon RDS – Securing the beast est apparu en premier sur Blog dbi services.

In the last post I looked at how you can strengthen the security for your PostgreSQL instance in Amazon RDS. In this post I’ll look at how you can configure your PostgreSQL instance.

When you login to the Amazon console and select your running instance the screen looks like this:

If you want to modify the instance there is a “modify” entry in the “Instance Actions” menu:

This brings you to the main configuration page which is the same as the one which was displayed when we did the setup of the instance. Notice that you can change the security group here, too. This is what we talked about in the last post.

But how can I change the PostgreSQL parameters? Nothing is offered here to do that. This is what the “Parameter Groups” are for:

It is not possible to change the default parameter group:

But we can create a new one:

Parameter groups created in addition to the default one can be edited:

And voila: The full set of PostgreSQL parameters is available. Although it is not possible to edit every parameter (e.g. archive_command or the ssl stuff) most of them can be adjusted. Lets give it a try and change “work_mem” to 128:

… and then save the parameter group. When I log into my instance and check the current_setting of work_mem it is 4MB:

How can we now apply the parameter group we just created? Quite easy: Go back where you can modify the instance and change the parameter group to the group just created:

… and apply the changes. A summary will be displayed on what will be changed:

Once we proceed the new settings will be applied:

Surprisingly the status changes to “pending reboot” after a few seconds which should not be necessary for modifications of work_mem):

Lets do it:

Once the reboot completed lets check if our new value for “work_mem” is really applied:

It is. So, the way to adjust the PostgreSQL settings is via the parameter groups. If you have many instances which different kind of workloads this simplifies configuration a lot.

In the next post I’ll look at how to add a replica database.

Cet article PostgreSQL on Amazon RDS – Configuring the beast est apparu en premier sur Blog dbi services.

In the last post I looked on how you can change the PostgreSQL parameters when the instance is running in the Amazon cloud. In this post we’ll look at how to add a replica to the PostgreSQL instance that is already running. Even if you push things to the cloud you want to make sure that you have a standby database in case the master crashes. At best the replica is running in another data center. Lets see what Amazon offers in this direction.

As usual we start with the instance page of the Amazon console:

One of the available options in the “Instance Actions” is “Create Read Replicate”:

So, lets do this and see what happens:

Not many parameters we have to specify here. Lets go on and create the replica:

Cool, that’s all we need to do? The replica is now in the process of being created and the master gets modified:

Lets wait a few minutes and then check what is there:

Finally the new replica is replicating from the master. Multi AZ is not available with the free version of Amazon RDS so the replica runs in the same availability zone. Automated backups are disabled and if you check the instance settings there is at least a bug in the description:

I was not aware I am doing MySQL here

There is even information about the current lag:

The interesting point is if we can connect to our replica for reads, so lets try:

Cool, works. We should not be able to write to the replica:

Perfect. The interesting point is what happens if we switch (promote) to replica and make it active. Will we lose the master or will it be rebuild as standby automatically? Lets try:

We are offered to specify the backup details for the new master:

And here it is: We can not restart the replication process with the new master. Lets see what happens:

And now we have two masters:

Hm, this is not what I expected. For production use the automated rebuild of the old master to a new standby should be done automatically which is not the case here. Maybe there are more options in the non-free version of Amazon RDS. At least for production this might be a killer. Additionally the promotion of the standby took several minutes which is not as fast as I expected. Promoting a PostgreSQL standby database is usually a matter of seconds. Sure, this was a manual promote but ending up with two masters is at least questionable.

In the next and last post in this series I’ll look at how you can load data to your cloud instance.

Cet article PostgreSQL on Amazon RDS – Adding a replica to the beast est apparu en premier sur Blog dbi services.

The last posts outlined how you can bring up a PostgreSQL instance in the Amazon cloud, how you can restrict access to the instance, how you can configure it and how you can add a read replica. In this post we’ll look at how you can load data to the instance. A database without data does not make much sense, does it?

For having some data available to load I’ll populate my local PostgreSQL 9.4 instance with a new database and some data:

(postgres@[local]:4445) [postgres] > select version();
                                                         version                                       
                   
-------------------------------------------------------------------------------------------------------
-------------------
 PostgreSQL 9.4.1dbi services on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Ha
t 4.8.3-9), 64-bit
(1 row)

Time: 1.256 ms

(postgres@[local]:4445) [postgres] > create database cloud;
CREATE DATABASE
Time: 1948.216 ms
(postgres@[local]:4445) [postgres] > \c cloud
You are now connected to database "cloud" as user "postgres".

(postgres@[local]:4445) [cloud] > create table tcloud1 ( a int );
CREATE TABLE
Time: 196.661 ms

(postgres@[local]:4445) [cloud] > insert into tcloud1 values (generate_series(1,1000000));
INSERT 0 1000000
Time: 6899.367 ms

(postgres@[local]:4445) [cloud] > create index icloud1 on tcloud1 (a);
CREATE INDEX
Time: 5390.778 ms

So, I have a database called “cloud” which contains a table with 1’000’000 rows and an index.

The first obvious method to get this local database to the cloud is pg_dump, so lets create a dump of my local database:

postgres@oel7:/home/postgres/ [PG3] time pg_dump -f /var/tmp/cloud.dmp cloud

real	0m0.217s
user	0m0.018s
sys	0m0.051s
postgres@oel7:/home/postgres/ [PG3] head /var/tmp/cloud.dmp
--
-- PostgreSQL database dump
--

SET statement_timeout = 0;
SET lock_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SET check_function_bodies = false;
SET client_min_messages = warning;

I should be able to load this dump into the cloud database. First I’ll need to create the database in the cloud instance:

daniel1=> create database cloud;
CREATE DATABASE
daniel1=> \l
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   
-----------+----------+----------+-------------+-------------+-----------------------
 cloud     | daniel1  | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 daniel1   | daniel1  | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 postgres  | daniel1  | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 rdsadmin  | rdsadmin | UTF8     | en_US.UTF-8 | en_US.UTF-8 | rdsadmin=CTc/rdsadmin
 template0 | rdsadmin | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/rdsadmin          +
           |          |          |             |             | rdsadmin=CTc/rdsadmin
 template1 | daniel1  | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/daniel1           +
           |          |          |             |             | daniel1=CTc/daniel1
(6 rows)

Having the database available I should be able to load the data:

dwe@dwe:~$ time psql -h daniel1.c8rrtm2sxmym.eu-central-1.rds.amazonaws.com -U daniel1 cloud < /var/tmp/cloud.dmp
Password for user daniel1: 
SET
SET
SET
SET
SET
SET
CREATE EXTENSION
ERROR:  must be owner of extension plpgsql
SET
SET
SET
CREATE TABLE
ERROR:  role "postgres" does not exist
CREATE INDEX
REVOKE
ERROR:  role "postgres" does not exist
ERROR:  role "postgres" does not exist
GRANT

real	0m17.894s
user	0m0.128s
sys	0m0.024s

Looks not so bad. The user postgres does not exist in the cloud instance, therefore the errors. When I check the dump file for that:

dwe@dwe:~$ cat /var/tmp/cloud.dmp | grep postgres
-- Name: tcloud1; Type: TABLE; Schema: public; Owner: postgres; Tablespace: 
ALTER TABLE tcloud1 OWNER TO postgres;
-- Data for Name: tcloud1; Type: TABLE DATA; Schema: public; Owner: postgres
-- Name: icloud1; Type: INDEX; Schema: public; Owner: postgres; Tablespace: 
-- Name: public; Type: ACL; Schema: -; Owner: postgres
REVOKE ALL ON SCHEMA public FROM postgres;
GRANT ALL ON SCHEMA public TO postgres;

… it is just the permissions which I do ignore here. Otherwise I’d have to re-create the database, change the statements in the dump file and all would be fine. The extension plpgsql does exist anyway:

daniel1=> \c cloud
psql (9.3.10, server 9.4.5)
WARNING: psql major version 9.3, server major version 9.4.
         Some psql features might not work.
SSL connection (cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256)
You are now connected to database "cloud" as user "daniel1".

cloud=> \dx
                 List of installed extensions
  Name   | Version |   Schema   |         Description          
---------+---------+------------+------------------------------
 plpgsql | 1.0     | pg_catalog | PL/pgSQL procedural language
(1 row)

The important point is if my data is there:

cloud=> select count(*) from tcloud1;
  count  
---------
 1000000
(1 row)

cloud=> \d+ tcloud1
                       Table "public.tcloud1"
 Column |  Type   | Modifiers | Storage | Stats target | Description 
--------+---------+-----------+---------+--------------+-------------
 a      | integer |           | plain   |              | 
Indexes:
    "icloud1" btree (a)
Has OIDs: no

Yes, it is. As I did add a replica in the last post the data should be there too:

dwe@dwe:~$ psql -h daniel2.c8rrtm2sxmym.eu-central-1.rds.amazonaws.com -U daniel1 cloud -c "select count(*) from tcloud1"
Password for user daniel1: 
  count  
---------
 1000000
(1 row)

Cool. Quite easy. Another option would be to use the copy command. Lets prepare the source:

(postgres@[local]:4445) [postgres] > \c cloud
You are now connected to database "cloud" as user "postgres".
(postgres@[local]:4445) [cloud] > create table tcloud2 as select * from tcloud1;
SELECT 1000000
Time: 5221.107 ms

(postgres@[local]:4445) [cloud] > copy tcloud2 to '/var/tmp/tcloud2.dmp';
COPY 1000000
Time: 151.243 ms

postgres@[local]:4445) [cloud] > \! head /var/tmp/tcloud2.dmp
1
2
3
4
5
6
7
8
9
10

And now load it to the cloud instance:

dwe@dwe:~$ psql -h daniel1.c8rrtm2sxmym.eu-central-1.rds.amazonaws.com -U daniel1
Password for user daniel1: 
psql (9.3.10, server 9.4.5)
WARNING: psql major version 9.3, server major version 9.4.
         Some psql features might not work.
SSL connection (cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256)
Type "help" for help.

daniel1=> \c cloud
psql (9.3.10, server 9.4.5)
WARNING: psql major version 9.3, server major version 9.4.
         Some psql features might not work.
SSL connection (cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256)
You are now connected to database "cloud" as user "daniel1".

cloud=> create table tcloud2 (like tcloud1 including indexes);
CREATE TABLE
cloud=> 

cloud=> copy tcloud2 from '/var/tmp/tcloud2.dmp';
ERROR:  must be superuser to COPY to or from a file
HINT:  Anyone can COPY to stdout or from stdin. psql's \copy command also works for anyone.

Hm, so my user is not allowed to do that as I am not a superuser. But the hint helps. We just can use psql’s “\copy” command instead:

cloud=> \copy tcloud2 from '/var/tmp/tcloud2.dmp';
cloud=> select count(*) from tcloud2;
  count  
---------
 1000000
(1 row)

cloud=> 

All the data is there. Of course you can use graphical tools like pgadmin3 to load the data, too:

Conclusion: It is really easy and fast to bring data from a local PostgreSQL instance to a cloud instance. Happy loading …

Cet article PostgreSQL on Amazon RDS – Loading the beast est apparu en premier sur Blog dbi services.

In Oracle you can use the external table feature to load data from files into the database. The traditional way to do this in PostgreSQL is to use the copy command. But there is another option which makes use of foreign data wrappers. The foreign data wrapper for doing this is file_fdw.

For creating some data here is a little script which generates 1’000 lines of the form: ID, TEXT:

#!/bin/bash

OUTPUT="/var/tmp/data.csv"
`rm -rf ${OUTPUT}`

for i in {1..1000}
do
  STRING=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1)
  echo "${i} , ${STRING}" >> /var/tmp/data.csv
done

For loading that into PostgreSQL the procedure is simple. First we need to create the extension and then define a server and a foreign table:

(postgres@[local]:5000) [postgres] > create extension file_fdw;
CREATE EXTENSION
Time: 752.715 ms
(postgres@[local]:5000) [postgres] > create server srv_file_fdw foreign data wrapper file_fdw;
CREATE SERVER
Time: 23.317 ms
(postgres@[local]:5000) [postgres] > create foreign table t_csv ( a int, b varchar(50) )
                                            server srv_file_fdw 
                                            options ( filename '/var/tmp/data.csv', format 'csv' );
CREATE FOREIGN TABLE
Time: 74.843 ms

From now we can access the table and load the data:

(postgres@[local]:5000) [postgres] > select count(*) from t_csv;
 count 
-------
  1000
(1 row)

Time: 0.707 ms
(postgres@[local]:5000) [postgres] > create table t_csv2 as select * from t_csv;
SELECT 1000
Time: 147.859 ms

Currently, as outlined in the documentation, you can not write to a foreign table defined with file_fdw:

(postgres@[local]:5000) [postgres] > insert into t_csv values (-1,'a');
ERROR:  cannot insert into foreign table "t_csv"
Time: 18.113 ms

Using explain you can get some details of the file:

(postgres@[local]:5000) [postgres] > explain select count(*) from t_csv;
                             QUERY PLAN                             
--------------------------------------------------------------------
 Aggregate  (cost=171.15..171.16 rows=1 width=0)
   ->  Foreign Scan on t_csv  (cost=0.00..167.10 rows=1621 width=0)
         Foreign File: /var/tmp/data.csv
         Foreign File Size: 38893
(4 rows)

Time: 0.521 ms

If you wonder why the estimate for the rows is wrong:

(postgres@[local]:5000) [postgres] > analyze t_csv;
ANALYZE
Time: 79.925 ms
(postgres@[local]:5000) [postgres] > explain select count(*) from t_csv;
                             QUERY PLAN                             
--------------------------------------------------------------------
 Aggregate  (cost=107.50..107.51 rows=1 width=0)
   ->  Foreign Scan on t_csv  (cost=0.00..105.00 rows=1000 width=0)
         Foreign File: /var/tmp/data.csv
         Foreign File Size: 38893
(4 rows)

Time: 0.296 ms

here you go …

Cet article External tables in PostgreSQL? est apparu en premier sur Blog dbi services.

Usually I advice to build PostgreSQL from source as it gives the most flexibility. Another option is to use the official yum repositories. There is one repository for each supported release of PostgreSQL and most of the common RedHat based Linux distributions are listed: CentOS, Oracle Linux, RedHat, Fedora, Scientific Linux and even Amazon Linux.

Currently PostgreSQL 9.6 is in active development and new features get developed and committed. So lets try to add to the PostgreSQL 9.6 repository. I assume I do not need to say that the 9.6 version is not for anything else than testing at the moment.

As I am only Oracle Linux 7 with my VM I downloaded the repository rpm for this version. Once this is transferred to the VM installation is easy:

root@oel7 tmp] yum localinstall pgdg-oraclelinux96-9.6-1.noarch.rpm
Loaded plugins: ulninfo
Examining pgdg-oraclelinux96-9.6-1.noarch.rpm: pgdg-oraclelinux96-9.6-1.noarch
Marking pgdg-oraclelinux96-9.6-1.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package pgdg-oraclelinux96.noarch 0:9.6-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================================================
 Package                                Arch                       Version                   Repository                                            Size
========================================================================================================================================================
Installing:
 pgdg-oraclelinux96                     noarch                     9.6-1                     /pgdg-oraclelinux96-9.6-1.noarch                     2.2 k

Transaction Summary
========================================================================================================================================================
Install  1 Package

Total size: 2.2 k
Installed size: 2.2 k
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : pgdg-oraclelinux96-9.6-1.noarch                                                                                                      1/1 
  Verifying  : pgdg-oraclelinux96-9.6-1.noarch                                                                                                      1/1 

Installed:
  pgdg-oraclelinux96.noarch 0:9.6-1                                                                                                                     

Complete!

Lets see what is available:

[root@oel7 tmp] yum search postgresql96
Loaded plugins: ulninfo
============================================================== N/S matched: postgresql96 ===============================================================
postgresql96-debuginfo.x86_64 : Debug information for package postgresql96
postgresql96.x86_64 : PostgreSQL client programs and libraries
postgresql96-contrib.x86_64 : Contributed source and binaries distributed with PostgreSQL
postgresql96-devel.x86_64 : PostgreSQL development header files and libraries
postgresql96-docs.x86_64 : Extra documentation for PostgreSQL
postgresql96-libs.x86_64 : The shared libraries required for any PostgreSQL clients
postgresql96-plperl.x86_64 : The Perl procedural language for PostgreSQL
postgresql96-plpython.x86_64 : The Python procedural language for PostgreSQL
postgresql96-pltcl.x86_64 : The Tcl procedural language for PostgreSQL
postgresql96-server.x86_64 : The programs needed to create and run a PostgreSQL server

Not as much as in the PostgreSQL 9.5 repo (e.g. all the utilities like pg_top) but it is perfect for testing new features. So lets install PostgreSQL 9.6:

[root@oel7 tmp] yum install postgresql96-server.x86_64
Loaded plugins: ulninfo
Resolving Dependencies
--> Running transaction check
---> Package postgresql96-server.x86_64 0:9.6-git20160121_1PGDG.rhel7 will be installed
--> Processing Dependency: postgresql96-libs(x86-64) = 9.6-git20160121_1PGDG.rhel7 for package: postgresql96-server-9.6-git20160121_1PGDG.rhel7.x86_64
--> Processing Dependency: postgresql96(x86-64) = 9.6-git20160121_1PGDG.rhel7 for package: postgresql96-server-9.6-git20160121_1PGDG.rhel7.x86_64
--> Processing Dependency: postgresql96 = 9.6-git20160121_1PGDG.rhel7 for package: postgresql96-server-9.6-git20160121_1PGDG.rhel7.x86_64
--> Running transaction check
---> Package postgresql96.x86_64 0:9.6-git20160121_1PGDG.rhel7 will be installed
---> Package postgresql96-libs.x86_64 0:9.6-git20160121_1PGDG.rhel7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================================================
 Package                                 Arch                       Version                                            Repository                  Size
========================================================================================================================================================
Installing:
 postgresql96-server                     x86_64                     9.6-git20160121_1PGDG.rhel7                        pgdg96                     4.4 M
Installing for dependencies:
 postgresql96                            x86_64                     9.6-git20160121_1PGDG.rhel7                        pgdg96                     1.2 M
 postgresql96-libs                       x86_64                     9.6-git20160121_1PGDG.rhel7                        pgdg96                     204 k

Transaction Summary
========================================================================================================================================================
Install  1 Package (+2 Dependent packages)

Total download size: 5.8 M
Installed size: 24 M
Is this ok [y/d/N]: y
Downloading packages:
(1/3): postgresql96-9.6-git20160121_1PGDG.rhel7.x86_64.rpm                                                                       | 1.2 MB  00:00:00     
(2/3): postgresql96-libs-9.6-git20160121_1PGDG.rhel7.x86_64.rpm                                                                  | 204 kB  00:00:01     
(3/3): postgresql96-server-9.6-git20160121_1PGDG.rhel7.x86_64.rpm                                                                | 4.4 MB  00:00:01     
--------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                   2.5 MB/s | 5.8 MB  00:00:02     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : postgresql96-libs-9.6-git20160121_1PGDG.rhel7.x86_64                                                                                 1/3 
  Installing : postgresql96-9.6-git20160121_1PGDG.rhel7.x86_64                                                                                      2/3 
  Installing : postgresql96-server-9.6-git20160121_1PGDG.rhel7.x86_64                                                                               3/3 
  Verifying  : postgresql96-9.6-git20160121_1PGDG.rhel7.x86_64                                                                                      1/3 
  Verifying  : postgresql96-server-9.6-git20160121_1PGDG.rhel7.x86_64                                                                               2/3 
  Verifying  : postgresql96-libs-9.6-git20160121_1PGDG.rhel7.x86_64                                                                                 3/3 

Installed:
  postgresql96-server.x86_64 0:9.6-git20160121_1PGDG.rhel7                                                                                              

Dependency Installed:
  postgresql96.x86_64 0:9.6-git20160121_1PGDG.rhel7                        postgresql96-libs.x86_64 0:9.6-git20160121_1PGDG.rhel7                       

Complete!

When you install from the official repositories all will be installed here:

[root@oel7 tmp] ls /usr/pgsql-9.6/
bin  lib  share
[root@oel7 tmp] ls /usr/pgsql-9.6/bin/
clusterdb   dropdb    pg_archivecleanup  pg_controldata  pg_isready      pg_rewind       pg_xlogdump                postmaster
createdb    droplang  pg_basebackup      pg_ctl          pg_receivexlog  pg_test_fsync   postgres                   psql
createlang  dropuser  pgbench            pg_dump         pg_resetxlog    pg_test_timing  postgresql96-check-db-dir  reindexdb
createuser  initdb    pg_config          pg_dumpall      pg_restore      pg_upgrade      postgresql96-setup         vacuumdb

Creating, starting and connecting to a new PostgreSQL 9.6 instance is straight forward from now on:

[root@oel7 tmp] su - postgres
Last login: Fri Jan 22 09:37:34 CET 2016 on pts/1
postgres@oel7:/home/postgres/ [dummy] mkdir data
postgres@oel7:/home/postgres/ [dummy] /usr/pgsql-9.6/bin/initdb -D data/
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /usr/pgsql-9.6/bin/pg_ctl -D data/ -l logfile start

postgres@oel7:/home/postgres/ [dummy] /usr/pgsql-9.6/bin/pg_ctl -D data/ -l /var/tmp/pg96.log start
server starting

postgres@oel7:/home/postgres/ [dummy] psql postgres
Null display is "NULL".
Timing is on.
psql (9.6devel)
Type "help" for help.

Here we go.

Btw: There are apt repositories as well, if you are on a debian based distribution.

Cet article Using the official PostgreSQL yum repositories est apparu en premier sur Blog dbi services.

In PostgreSQL every database contains the public schema by default. Every user that gets created and can login is able to create objects there. Here is a little demo: I’ll create a new user named u1 which is allowed to login. No additional privileges are granted:

postgres=# create user u1 login password 'u1';
CREATE ROLE
postgres=# \c postgres u1
You are now connected to database "postgres" as user "u1".

From now on this user is able to connect to any database in the cluster and is able to create objects in the public schema:

postgres=> \c postgres u1
You are now connected to database "postgres" as user "u1".
postgres=> create table t1 ( a int );
CREATE TABLE
postgres=> insert into t1 values (1);
INSERT 0 1
postgres=> select schemaname,tablename,tableowner 
             from pg_tables 
            where tablename = 't1';
 schemaname | tablename | tableowner 
------------+-----------+------------
 public     | t1        | u1
(1 row)

postgres=> \c edb u1
You are now connected to database "edb" as user "u1".
edb=> create table t1 ( a int );
CREATE TABLE
edb=> select schemaname,tablename,tableowner 
        from pg_tables 
       where tablename = 't1';
 schemaname | tablename | tableowner 
------------+-----------+------------
 public     | t1        | u1
(1 row)

This is probably not what you want as such a user can fill your database and therefore may cause major issues (performance, disk full etc. ). How can we avoid that?

One way to do it is to revoke everything from public:

postgres=# revoke all on schema public from public;
REVOKE

If we now re-connect to the postgres database and try to create a table this will fail:

postgres=# \c postgres u1
You are now connected to database "postgres" as user "u1".
postgres=> create table t2 ( a int );
ERROR:  no schema has been selected to create in
postgres=> create table public.t2 ( a int );
ERROR:  permission denied for schema public

The issue with this approach is that if we connect to another database we are still allowed to create tables:

postgres=# \c edb u1
You are now connected to database "edb" as user "u1".
edb=> create table t2 ( a int );
CREATE TABLE
edb=> 

Only when we do the same revoke in this database …

postgres=# \c edb
You are now connected to database "edb" as user "enterprisedb".
edb=# revoke all on schema public from public;    
REVOKE

.. we are not able to create tables anymore:

edb=# \c edb u1
You are now connected to database "edb" as user "u1".
edb=> create table t3 ( a int );
ERROR:  no schema has been selected to create in
edb=> create table public.t3 ( a int );
ERROR:  permission denied for schema public

Seems a little bit complicated. What else can we do? As every new database is created from template1 by default we can do our revokes there:

postgres=# \c template1
You are now connected to database "template1" as user "enterprisedb".
template1=# revoke all on schema public from public;
REVOKE

Every new database should have this included already:

postgres=# create database db1;
CREATE DATABASE
postgres=# \c db1 u1
You are now connected to database "db1" as user "u1".
db1=> create table t1 ( a int );
ERROR:  no schema has been selected to create in
db1=> create table public.t1 ( a int ); 
ERROR:  permission denied for schema public
db1=> 

Much better.

But remember that it is usually not that easy to connect at all because of pg_hba.conf. When we try to connect from outside:

dwe@dwe:~$ psql -h 192.168.22.243 -p 5444 -U u1 db1
psql: FATAL:  no pg_hba.conf entry for host "192.168.22.1", user "u1", database "db1", SSL off

We have much more control here. If we do not grant access to a specific database we will not be able to connect. So we might grant access to the db1 database but not to the postgres database by adding this line to pg_hba.conf:

host     db1             u1                  192.168.22.1/32         md5

Once the server was reloaded or restarted we are able to connect:

dwe@dwe:~$ psql -h 192.168.22.243 -p 5444 -U u1 db1
Password for user u1: 
psql (9.3.10, server 9.4.5.12)
WARNING: psql major version 9.3, server major version 9.4.
         Some psql features might not work.
Type "help" for help.

db1=> 

But we are not able to connect to any other database:

dwe@dwe:~$ psql -h 192.168.22.243 -p 5444 -U u1 postgres
psql: FATAL:  no pg_hba.conf entry for host "192.168.22.1", user "u1", database "postgres", SSL off
dwe@dwe:~$ psql -h 192.168.22.243 -p 5444 -U u1 template1
psql: FATAL:  no pg_hba.conf entry for host "192.168.22.1", user "u1", database "template1", SSL off

Conclusion: Always make sure that you allow connections from trusted sources only ( via pg_hba.conf ) and think about the public schema before granting access.

Cet article Avoiding access to the public schema in PostgreSQL est apparu en premier sur Blog dbi services.

In the last posts (1, 2, 3, 4, 5) I looked at PostgreSQL in Amazon RDS. Another option you have if you want to go to the cloud is to use the The Postgres Plus® Cloud Database offered by EnterpriseDB.

I will not go through the whole setup here as this is described in detail here.

So once you went through the registration and setup process you should be able to login. The login URL is depending on the region you selected so this link will not work for you if you not selected a region in the EU (Ireland, in my case):

You’ll land at the EnterpriseDB console after a successful login:

Obviously we want to bring up our first cluster, so lets go. You’ll have the choice between community PostgreSQL and Postgres Plus Advanced Server. The other values are more or less the same than those of the Amazon RDS service:

As soon as you choose continuous archiving a storage container for holding all the archived wal files will be created automatically:

As soon as the creation of the storage finished an email should arrive in your inbox:

From: support@theclouddatabase.com
To: "daniel westermann" 
Sent: Wednesday, January 13, 2016 2:16:18 PM
Subject: WAL Archive Storage Container Created

A storage container (bucket) named xxxxx-ppcd has been created. All Postgres Plus Cloud Database clusters configured for Continuous Archiving (Point-in-Time Recovery) will use this location to store archived WAL files. This container should not be deleted once created as it will cause WAL archiving to stop functioning.

Waiting a few minutes for the process to complete and the new PostgreSQL cluster will be ready:

As with the storage container you’ll receive notification emails once an instance is ready:

From: support@theclouddatabase.com
To: "daniel westermann" 
Sent: Wednesday, January 13, 2016 2:32:09 PM
Subject: Database State Changed to RUNNING i-7ce1b6f1 t2.micro

The Master database server ec2-52-18-219-36.eu-west-1.compute.amazonaws.com in cluster pgclust1 is now RUNNING in location eu-west-1a using port 1234.

From: support@theclouddatabase.com
To: "daniel westermann" 
Sent: Wednesday, January 13, 2016 2:40:44 PM
Subject: Database State Changed to RUNNING i-9e4c7d15 t2.micro

The Replica database server ec2-52-48-111-239.eu-west-1.compute.amazonaws.com t2.micro in cluster pgclust1 is now RUNNING in location eu-west-1c.

Quite easy. A very good features is that a load balances will be added automatically which listens on port 9999:

Lets see if we can connect to the load balancer:

dwe@dwe:~$ psql -h ec2-52-18-219-36.eu-west-1.compute.amazonaws.com -p 9999 -U postgres postgres
Password for user postgres: 
psql (9.3.10, server 9.4.4)
WARNING: psql major version 9.3, server major version 9.4.
         Some psql features might not work.
Type "help" for help.

postgres=# \l
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
-----------+----------+----------+------------+------------+-----------------------
 postgres  | postgres | UTF8     | en_US.UTF8 | en_US.UTF8 | 
 template0 | postgres | UTF8     | en_US.UTF8 | en_US.UTF8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF8 | en_US.UTF8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
(3 rows)

postgres=# 

In contrast to the Amazon RDS service we have full control over the cluster as we are able to connect with the postgres superuser.
When the cluster was created an initial backup was created automatically:

Another interesting feature is auto scaling:

You have two options there: Auto scale the cluster when a specified threshold for storage usage is reached or auto scale the cluster when a specified threshold on concurrent connections is reached. But keep in mind that this will increase costs as well.

You even can choose what happens when there is a major issue on the master:

Do you want to replace the master with a new master or do you want to promote to a replica? Promoting to a replica is faster but you might loose transactions as replication in this case is asynchronous. In case the master is replaced with a new master the storage volumes are switch to the new master so no transaction will be lost but this will take longer than promoting a replica.

Configuring the instances is almost the same as in Amazon RDS. You’ll have to create a new configuration and then attach it to a running instance. Additionally you may configure pgpool, which is the load balancer from above:

The monitoring part is slightly limited. You can take a look at the storage used, the amount of connections and the load. This help when defining the auto scaling options but does not replace a full monitoring solution:

The events sections lists all events that happened to your cluster:

Let’s try to manually scale up our cluster by adding another replica:

Quite easy as well. Now we have two replicas which replicate from the same master. So, in the end, this might be a solution to consider.

Cet article The Postgres Plus® Cloud Database est apparu en premier sur Blog dbi services.

The just released version of EDB Postgres Advanced Server 9.5 introduces profiles very much the same as in Oracle. Lets have a look at it.

As in Oracle there is a default profile:

(enterprisedb@[local]:5445) [postgres] > \x
Expanded display is on.
(enterprisedb@[local]:5445) [postgres] > select * from edb_profile;
-[ RECORD 1 ]-----------+--------
prfname                 | default
prffailedloginattempts  | -2
prfpasswordlocktime     | -2
prfpasswordlifetime     | -2
prfpasswordgracetime    | -2
prfpasswordreusetime    | -2
prfpasswordreusemax     | -2
prfpasswordverifyfuncdb | NULL
prfpasswordverifyfunc   | NULL

You can also query the dba_profiles view exactly as in Oracle:

(enterprisedb@[local]:5445) [postgres] > \x
Expanded display is off.
(enterprisedb@[local]:5445) [postgres] > select * from dba_profiles;
 profile |      resource_name       | resource_type |   limit   | common 
---------+--------------------------+---------------+-----------+--------
 DEFAULT | FAILED_LOGIN_ATTEMPTS    | PASSWORD      | UNLIMITED | NO
 DEFAULT | PASSWORD_GRACE_TIME      | PASSWORD      | UNLIMITED | NO
 DEFAULT | PASSWORD_LIFE_TIME       | PASSWORD      | UNLIMITED | NO
 DEFAULT | PASSWORD_LOCK_TIME       | PASSWORD      | UNLIMITED | NO
 DEFAULT | PASSWORD_REUSE_MAX       | PASSWORD      | UNLIMITED | NO
 DEFAULT | PASSWORD_REUSE_TIME      | PASSWORD      | UNLIMITED | NO
 DEFAULT | PASSWORD_VERIFY_FUNCTION | PASSWORD      | NULL      | NO
(7 rows)

In fact the dba_profiles view is just a view on top of edb_profile. You can verify this by:

(enterprisedb@[local]:5445) [postgres] >  \d+ dba_profiles                                                        
                           View "sys.dba_profiles"
    Column     |          Type          | Modifiers | Storage  | Description 
---------------+------------------------+-----------+----------+-------------
 profile       | character varying(128) |           | extended | 
 resource_name | character varying(32)  |           | extended | 
 resource_type | character varying(8)   |           | extended | 
 limit         | character varying(128) |           | extended | 
 common        | character varying(3)   |           | extended | 
View definition:
 SELECT
        CASE
...

When there are profiles the “create user/role” syntax should have been extended, too:

(enterprisedb@[local]:5445) [postgres] > \h create user
Command:     CREATE USER
Description: define a new database role
Syntax:
CREATE USER name [ [ WITH ] option [ ... ] ]

where option can be:

      SUPERUSER | NOSUPERUSER
    | CREATEDB | NOCREATEDB
    | CREATEROLE | NOCREATEROLE
    | CREATEUSER | NOCREATEUSER
    | INHERIT | NOINHERIT
    | LOGIN | NOLOGIN
    | REPLICATION | NOREPLICATION
    | CONNECTION LIMIT connlimit
    | [ ENCRYPTED | UNENCRYPTED ] PASSWORD 'password'
    | VALID UNTIL 'timestamp'
    | PROFILE profile_name      <XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    | ACCOUNT { LOCK | UNLOCK }
    | LOCK TIME 'timestamp'
    | PASSWORD EXPIRE [ AT 'timestamp' ]
    | IN ROLE role_name [, ...]
    | IN GROUP role_name [, ...]
    | ROLE role_name [, ...]
    | ADMIN role_name [, ...]
    | USER role_name [, ...]
    | SYSID uid

I would expect that every new user gets the default profile if we do not specify one:

(enterprisedb@[local]:5445) [postgres] > create user u1 login password 'u1';
CREATE ROLE
(enterprisedb@[local]:5445) [postgres] > \du+ u1
                     List of roles
 Role name |   Attributes    | Member of | Description 
-----------+-----------------+-----------+-------------
 u1        | Profile default | {}        | 

As expected. So lets play with a new profile:

(enterprisedb@[local]:5445) [postgres] > create profile my_profile;
CREATE PROFILE
Time: 18.296 ms
(enterprisedb@[local]:5445) [postgres] > alter profile my_profile limit FAILED_LOGIN_ATTEMPTS 1;
ALTER PROFILE
Time: 1.799 ms
(enterprisedb@[local]:5445) [postgres] > alter profile my_profile limit PASSWORD_LOCK_TIME 1;   
ALTER PROFILE
Time: 0.979 ms
(enterprisedb@[local]:5445) [postgres] > alter profile my_profile limit PASSWORD_LIFE_TIME 1;

Pretty much the same as in Oracle. For a full list and a description of the limits check the documentation.

Lets attach the new profile to our user:

(enterprisedb@[local]:5445) [postgres] > alter user u1 profile my_profile ;
ALTER ROLE
Time: 1.931 ms
(enterprisedb@[local]:5445) [postgres] > \du u1
               List of roles
 Role name |     Attributes     | Member of 
-----------+--------------------+-----------
 u1        | Profile my_profile | {}

An important point to understand is that profiles are not per database but per instance. Once a profile is created it is available in all databases:

(enterprisedb@[local]:5445) [postgres] > \c edb
You are now connected to database "edb" as user "enterprisedb".
(enterprisedb@[local]:5445) [edb] > select distinct profile from dba_profiles;
  profile   
------------
 DEFAULT
 MY_PROFILE
(2 rows)

Time: 63.333 ms
(enterprisedb@[local]:5445) [edb] > \c postgres                               
You are now connected to database "postgres" as user "enterprisedb".
(enterprisedb@[local]:5445) [postgres] > select distinct profile from dba_profiles;
  profile   
------------
 DEFAULT
 MY_PROFILE
(2 rows)

Time: 2.130 ms

(enterprisedb@[local]:5445) [postgres] > create database db1;
CREATE DATABASE
Time: 1226.746 ms
(enterprisedb@[local]:5445) [postgres] > \c db1
You are now connected to database "db1" as user "enterprisedb".
(enterprisedb@[local]:5445) [db1] > select distinct profile from dba_profiles;
  profile   
------------
 DEFAULT
 MY_PROFILE
(2 rows)

When profiles are global what about a password verify function you may attach to the profile? Functions are per database, aren’t they? Lets create a simple function:

create or replace function my_func ( password in varchar(30) ) return boolean
is
begin
  if ( length (password) < 5 )
  then
    raise_application_error ( -20001, 'to short');
  end if;
  return true;
end my_func;
/

And then assign this function to the profile:

(enterprisedb@[local]:5445) [postgres] > \df
                             List of functions
 Schema |  Name   | Result data type |    Argument data types     |  Type  
--------+---------+------------------+----------------------------+--------
 public | my_func | boolean          | password character varying | normal
(1 row)

(enterprisedb@[local]:5445) [postgres] > alter profile my_profile limit password_verify_function my_func;
ERROR:  function my_func does not exist
Time: 0.479 ms

Hm. The function clearly exists. The issue here is that there are some specific requirements for the password verify function:

• It needs to be owned by a super user
• It needs to be in the sys schema
• It needs to have the following signature: function_name (varchar2, varchar2, varchar2) return boolean

Knowing this let’s try again with this function:

CREATE OR REPLACE FUNCTION sys.my_func (username varchar2, password varchar2, old_password varchar2)  
RETURN boolean IMMUTABLE  
IS  
BEGIN  
  if ( length (password) < 5 )  
  then  
    raise_application_error ( -20001, 'too short');  
  end if;  
  return true;  
END my_func;  

To be sure lets set the owner of this function to a super user

(enterprisedb@[local]:5445) [postgres] > alter function my_func (varchar2,varchar2,varchar2) owner to enterprisedb;
ALTER FUNCTION
Time: 0.311 ms

Do we succeed now?

(enterprisedb@[local]:5445) [postgres] > alter profile my_profile limit password_verify_function my_func;
ALTER PROFILE
Time: 21.048 ms

Yes, much better. Lets do a simple test:

(enterprisedb@[local]:5445) [postgres] > alter user u1 password 'u1';
ERROR:  EDB-20001: too short
CONTEXT:  edb-spl function my_func(character varying,character varying,character varying) line 5 at procedure/function invocation statement
Time: 67.705 ms

Ok, now it really works. But what does happen if I am connected to another database where the password verify function does not exist?

(enterprisedb@[local]:5445) [postgres] > \c edb
You are now connected to database "edb" as user "enterprisedb".
(enterprisedb@[local]:5445) [edb] > create user u2 password 'u2' profile my_profile;
ERROR:  password verify function is stored in database postgres
HINT:  You must connect to this database in order to assign a password.
Time: 0.644 ms
(enterprisedb@[local]:5445) [edb] > 

Ok, at least the server is telling me where I need to connect to.

This does not lock like a big new feature but it is another little piece that makes migrations from Oracle less problematic especially if there are requirements for password policies.

Cet article EDB Postgres Advanced Server 9.5 new features – Profiles est apparu en premier sur Blog dbi services.

When it comes to designing a PostgreSQL architecture that fits your requirements there are a couple of things to think about: Do you need a HA setup? How shall backups and restores being done and how shall all of these components get monitored? Do you need vendor support for all the tools that make up the architecture? At dbi we have a reference architecture that we can apply to most of our projects. It is not a one size fits all architecture it is more a multi stage model. Each stage adds more redundancy, more control and more automation. In this post I’ll look at the various stages using the tools EnterpriseDB provides.

Lets start with the “minimal” configuration:

On the top there is the application (web based or not) which fires requests against the PostgreSQL master database. The master database streams all changes to a standby database (in another data center, if possible). At the bottom there is BART (which is the backup and recovery tool provided by EnterpriseDB) which performs automated backups. Retention policies ensure proper deletion of expired backups. Test and development databases might be refreshed from the BART host with minimal effort. The downsides with this architecture are:

• The application probably needs to be reconfigured to connect to the standby database in case the master goes down for any reason
• There is no integrated monitoring solution which alerts in case of issues. Sure, monitoring can be done by using scripts, check_postgres or other tools but it needs extra effort

This brings us to the extended stage:

In addition, this stage adds the EDB failover manager which avoids the reconfiguration of the application in case the master goes down. This works by leveraging virtual IP addresses as you probably know it from other cluster solutions. When the master goes down the VIP is switched to the standby and the standby is opened for read/write automatically. The application can continue working without manual intervention. But there is still a downside with this solution: As with the minimal solution there is no integrated monitoring.

This is where the next stage comes into the game:

In addition to the extended stage PEM (Postgres Enterprise Manager) is added. This allows monitoring and alerting of the complete infrastructure. This can either be agent-less or agent based depending on how much information you want to get from the operating systems and how many tasks you want to have available from the PEM console (e.g. remote restarting of PostgreSQL instances). This architecture protects from failures, requires no manual intervention in case of disasters, provides easy and robust monitoring and is able to backup and restore the PostgreSQL instances to any other host for testing or development purposes.

But we can even do more by adding a second replica to offload reporting:

Adding a second replica for reporting purposes takes away load from the master and frees resources while leveraging the resources that are available on the host where the replica runs on.

Sure, this is not a complete picture but these 4 stages provide a minimal overview which helps customers to understand what is possible, where to look at and how they can scale. More replicas can easily be added (even cascading) and load balancers can be placed in front of the databases (e.g. pgpool-II) which strengthens the architecture even more. It all depends on the requirements and for sure there are use cases for every of these stages. In the end it is the customer who decides but at least we can give recommendations based on what we did in the past and what we believe is the best solution for the given requirements.

Of course, all of this can be backed by a SLA according to the customers needs. Interested? Let us know …

Cet article The dbi services PostgreSQL reference architecture (1) – The commercial approach est apparu en premier sur Blog dbi services.

In the last post I looked at how dbi services designs a PostgreSQL architecture based on the products of EnterpriseDB. In this post I’ll look at the same questions but using pure community tools. If you do not need vendor support and do trust the community for providing help and fixes this is another way to go. And usually you can trust the community around PostgreSQL very well. The architecture itself is pretty much the same as in the last post. The tools are different.

As in the last post there is a “minimal” configuration:

The picture is almost the same as with the commercial approach. The only little thing that changed is the backup and recovery tool: This time it is Barman instead of BART. As just one tool changes but not the architecture as a whole the same restrictions apply as with the commercial setup:

• The application probably needs to be reconfigured to connect to the standby database in case the master goes down for any reason
• There is no integrated monitoring solution which alerts in case of issues. Sure, monitoring can be done by using scripts, check_postgres or other tools but it needs extra effort

Once there is the need for automated failover in case the primary goes down the next stage adds repmgr. But as repmgr (at least as of now) is not working with VIPs we’ll need to add another tool to the landscape: pgpool-II. Combining both connections can be redirected: Writes will go to the master and reads can go to the standby. In case of a failover pgpool will redirect both, reads and writes, to the surviving node.

What is missing here, same issue than in the last post, is a monitoring solution over all components. This is where the dbi services DMK comes into the game. DMK brings a set of monitoring scripts and pre-defined thresholds that can be used to monitor all components.

The integration into monitoring tools such as Nagios, Icinga, Zabbix and many others is straight forward and easy. Its all based on customizable scripts and free for our customers. What else is missing? To add even more protection, flexibility and load balancing more replicas can be added to the landscape (one more in the picture below):

Again, this is a simplified overview but it should help in understanding the concepts. Technically the implementation requires a bit more planning and needs a lot of testing especially from the application side (this is true for any HA implementation and not limited to PostgreSQL, of course). But once this up and running you’ll have a reliable, robust and safe PostgreSQL architecture all based on community products.

Cet article The dbi services PostgreSQL reference architecture (2) – The community approach est apparu en premier sur Blog dbi services.

When you are working on Oracle you probably learned that each DDL does an implicit commit in the background. There is no way to have DDLs wrapped in a real transaction (with the exception of the “create schema” command which I blogged about a long time ago). If you take a look a the SQL:2008 standard (ISO/IEC 9075-1:2008) you can find this: “An SQL-transaction (transaction) is a sequence of executions of SQL-statements that is atomic with respect to recovery. That is to say: either the execution result is completely successful, or it has no effect on any SQL schemas or SQL-data.” Well, a schema is what DDL is about, isn’t it? Lets see how PostgreSQL handles this.

When you create objects like you would do it in Oracle you will probably do it like this:

(postgres@[local]:5000) [postgres] > create table tab1 ( a int, b date );
CREATE TABLE
Time: 60.109 ms
(postgres@[local]:5000) [postgres] > create view view1 as select a from tab1;
CREATE VIEW
Time: 64.288 ms
(postgres@[local]:5000) [postgres] > create sequence seq1;
CREATE SEQUENCE
Time: 36.861 ms

Creating object this way behaves pretty much the same as Oracle does. There are autocommits in the background and all objects got created:

(postgres@[local]:5000) [postgres] > \d tab1
     Table "public.tab1"
 Column |  Type   | Modifiers 
--------+---------+-----------
 a      | integer | 
 b      | date    | 

(postgres@[local]:5000) [postgres] > \d view1
     View "public.view1"
 Column |  Type   | Modifiers 
--------+---------+-----------
 a      | integer | 

(postgres@[local]:5000) [postgres] > \d seq1
            Sequence "public.seq1"
    Column     |  Type   |        Value        
---------------+---------+---------------------
 sequence_name | name    | seq1
 last_value    | bigint  | 1
 start_value   | bigint  | 1
 increment_by  | bigint  | 1
 max_value     | bigint  | 9223372036854775807
 min_value     | bigint  | 1
 cache_value   | bigint  | 1
 log_cnt       | bigint  | 0
 is_cycled     | boolean | f
 is_called     | boolean | f

But is this what you really want? What do you do when one ore more of the statements fail? Drop all objects that got created and start from the beginning? Imagine you want to create hundred of objects and you want to either all of them to succeed or roll back everything if one ore more fail. This is where transactions come into the game: A transaction is atomic which means: Either everything inside the transaction succeeds or everything is rolled back. Transactions are not only for DML as you might think. PostgreSQL lets you do this:

(postgres@[local]:5000) [postgres] > begin;
BEGIN
Time: 0.075 ms
(postgres@[local]:5000) [postgres] > create table tab1 ( a int, b date );
CREATE TABLE
Time: 0.543 ms
(postgres@[local]:5000) [postgres] > create view view1 as select a from tab1;
CREATE VIEW
Time: 0.520 ms
(postgres@[local]:5000) [postgres] > create sequence seq1;
CREATE SEQUENCE
Time: 0.576 ms
(postgres@[local]:5000) [postgres] > end;
COMMIT
Time: 0.854 ms

All the DDLs are executed in one transaction. As soon as then transaction is closed by issuing “end;” a commit is performed. All objects are there:

(postgres@[local]:5000) [postgres] > \d tab1
     Table "public.tab1"
 Column |  Type   | Modifiers 
--------+---------+-----------
 a      | integer | 
 b      | date    | 

(postgres@[local]:5000) [postgres] > \d view1
     View "public.view1"
 Column |  Type   | Modifiers 
--------+---------+-----------
 a      | integer | 

(postgres@[local]:5000) [postgres] > \d seq1
            Sequence "public.seq1"
    Column     |  Type   |        Value        
---------------+---------+---------------------
 sequence_name | name    | seq1
 last_value    | bigint  | 1
 start_value   | bigint  | 1
 increment_by  | bigint  | 1
 max_value     | bigint  | 9223372036854775807
 min_value     | bigint  | 1
 cache_value   | bigint  | 1
 log_cnt       | bigint  | 0
 is_cycled     | boolean | f
 is_called     | boolean | f

The pretty cool thing is what happens if one or more of the statements fail:

(postgres@[local]:5000) [postgres] > begin;
BEGIN
Time: 0.076 ms
(postgres@[local]:5000) [postgres] > create table tab1 ( a int, b date );
CREATE TABLE
Time: 0.500 ms
(postgres@[local]:5000) [postgres] > create view view1 as select a from tab99999;
ERROR:  relation "tab99999" does not exist

As the table referenced by the view does not exist the create view statements fails. What is the status of my transaction now? Lets try to continue with the create sequence statement as it would happen if all the DDLs are executed inside a script:

(postgres@[local]:5000) [postgres] > create sequence seq1;
ERROR:  current transaction is aborted, commands ignored until end of transaction block
Time: 0.126 ms

The transaction was aborted and everything will be ignored until we close the transaction. So lets do it:

(postgres@[local]:5000) [postgres] > end;
ROLLBACK
Time: 0.108 ms

And here we go: Instead of a commit a rollback was executed. Does the table tab1 exist?

(postgres@[local]:5000) [postgres] > \d tab1
Did not find any relation named "tab1".

No. So, without worrying about cleaning up anything that was created successfully we can just fix the issue and start again. Isn’t that a nice feature?

Cet article Transactional DDL est apparu en premier sur Blog dbi services.

When a database gets shutdown for patching, operating system maintenance or other reasons you completely lose the contents of the buffer cache. No surprise. Memory is not persistent. Wouldn’t it be nice to load the most important tables to the cache when the instance starts automatically? This would reduce the time for the most important queries to return the results as blocks would not be needed to read from disk. PostgreSQL has a solution for that which is called pg_prewarm. This is a contrib module which is available by default.

The extension provided by default can be installed by just issuing the “create extension” statement:

(postgres@[local]:5001) [postgres] > \! ls /u01/app/postgres/product/95/db_0/share/extension/*prewarm*
/u01/app/postgres/product/95/db_0/share/extension/pg_prewarm--1.0.sql  /u01/app/postgres/product/95/db_0/share/extension/pg_prewarm.control
(postgres@[local]:5001) [postgres] > create extension pg_prewarm;
CREATE EXTENSION
Time: 174.232 ms
(postgres@[local]:5001) [postgres] > \dx pg_pre*
             List of installed extensions
    Name    | Version | Schema |      Description      
------------+---------+--------+-----------------------
 pg_prewarm | 1.0     | public | prewarm relation data
(1 row)

So, how does it work? Lets create some sample data in a first step:

(postgres@[local]:5001) [postgres] > create table prewarm ( a int, b varchar(100) );
CREATE TABLE
Time: 51.165 ms
(postgres@[local]:5001) [postgres] > with generator as ( select generate_series(1,1000000) a )
insert into prewarm ( select a, md5(a::text) from generator );

To get some idea on how long it takes to read the whole table I’ll restart the instance and do a count(*):

postgres@oel7:/home/postgres/ [PG9] pg_ctl -D /u02/pgdata/PG9 stop -m fast
postgres@oel7:/home/postgres/ [PG9] sudo sh -c 'echo 3 >/proc/sys/vm/drop_caches'
postgres@oel7:/home/postgres/ [PG9] pg_ctl -D /u02/pgdata/PG9 start

Note: the sudo command drops the kernel cache (see here for details).

postgres@oel7:/home/postgres/ [PG9] time psql -c "select count(*) from prewarm" postgres
  count  
---------
 1000000
(1 row)

real	0m0.908s
user	0m0.005s
sys	0m0.000s

Almost a second to scan through the 1’000’000 rows to get the count. Lets do the same test but pre-warming the table before:

postgres@oel7:/home/postgres/ [PG9] pg_ctl -D /u02/pgdata/PG9 stop -m fast
postgres@oel7:/home/postgres/ [PG9] sudo sh -c 'echo 3 >/proc/sys/vm/drop_caches'
postgres@oel7:/home/postgres/ [PG9] pg_ctl -D /u02/pgdata/PG9 start
postgres@oel7:/home/postgres/ [PG9] psql -c "select pg_prewarm('prewarm')" postgres
 pg_prewarm 
------------
       8334
(1 row)

Note: The number returned by pg_prewarm is the number of pages affected/pre-warmed.
How long does it now take to get the result:

postgres@oel7:/home/postgres/ [PG9] time psql -c "select count(*) from prewarm" postgres
  count  
---------
 1000000
(1 row)


real	0m0.189s
user	0m0.002s
sys	0m0.004s

Not so bad, isn’t it? There is a second parameter for pg_prewarm which specifies the method to use (copied from the documentation):

• prefetch: issues asynchronous prefetch requests to the operating system, if this is supported, or throws an error otherwise
• read: reads the requested range of blocks; unlike prefetch, this is synchronous and supported on all platforms and builds, but may be slower
• buffer: reads the requested range of blocks into the database buffer cache

The default is “buffer”. If you write a little script which pre-warms the tables you need and put this at the end of your PostgreSQL startup script you can save the time it takes to load the blocks/pages from disk when the first requests come in after a restart of the instance.

There is another extension called pg_hibernater which automatically stores the IDs of the buffers in the buffer cache when the server shuts down and re-reads them again automatically when the server starts.

To install this module: Download, extract, make install as the postgres user:

postgres@oel7:/var/tmp/ [PG9] unzip pg_hibernator-master.zip 
Archive:  pg_hibernator-master.zip
32b344fab58242552730e142adffa8650c7df4ac
   creating: pg_hibernator-master/
  inflating: pg_hibernator-master/LICENSE  
  inflating: pg_hibernator-master/Makefile  
  inflating: pg_hibernator-master/README.md  
  inflating: pg_hibernator-master/misc.c  
  inflating: pg_hibernator-master/pg_hibernate.c  
  inflating: pg_hibernator-master/pg_hibernate_9.3.c  
  inflating: pg_hibernator-master/pg_hibernator.h  
   creating: pg_hibernator-master/tests/
  inflating: pg_hibernator-master/tests/pg_hibernator_comparison_chart.ods  
  inflating: pg_hibernator-master/tests/test_run.txt  
postgres@oel7:/var/tmp/ [PG9] cd pg_hibernator-master
postgres@oel7:/var/tmp/pg_hibernator-master/ [PG9] make install

We’ll need to add the module to the shared_preload_libraries parameter for pg_hibernator to start up:

(postgres@[local]:5001) [postgres] > show shared_preload_libraries;
 shared_preload_libraries 
--------------------------
 
(1 row)

Time: 0.283 ms

(postgres@[local]:5001) [postgres] > alter system set shared_preload_libraries='pg_hibernator';
ALTER SYSTEM
Time: 59.374 ms

Restart the server:

postgres@oel7:/home/postgres/ [PG9] pg_ctl -D /u02/pgdata/PG9 restart -m fast

Double-check the setting:

(postgres@[local]:5001) [postgres] > show shared_preload_libraries ;
 shared_preload_libraries 
--------------------------
 pg_hibernator
(1 row)

Time: 0.307 ms

There are a few parameters which control the behavior (check the documentation for details):

(postgres@[local]:5001) [postgres] > show pg_hibernator.enabled;
 pg_hibernator.enabled 
-----------------------
 on
(1 row)

Time: 0.148 ms
(postgres@[local]:5001) [postgres] > show pg_hibernator.parallel;
 pg_hibernator.parallel 
------------------------
 off
(1 row)

Time: 0.140 ms
(postgres@[local]:5001) [postgres] > show pg_hibernator.default_database;
 pg_hibernator.default_database 
--------------------------------
 postgres
(1 row)

Time: 0.142 ms

Once the module is running there is a new background process called “Buffer Saver”. This process will save the buffer IDs when the server shuts down.

postgres@oel7:/home/postgres/ [PG9] ps -ef | grep "Buffer Saver"
postgres 22773 22764  0 16:12 ?        00:00:00 postgres: bgworker: Buffer Saver   
postgres 22810 20779  0 16:17 pts/2    00:00:00 grep --color=auto Buffer Saver

If you check your data directory you should find that pg_hibernater writes a file per database which has buffers in the cache on shutdown of the server:

postgres@oel7:/home/postgres/ [PG9] ls -la $PGDATA/pg_hibernator/
total 12
drwx------.  2 postgres postgres   32 Mar  9 16:18 .
drwx------. 19 postgres postgres 4096 Mar  9 16:18 ..
-rw-------.  1 postgres postgres   91 Mar  9 16:18 1.save
-rw-------.  1 postgres postgres  619 Mar  9 16:18 2.save

So, lets do the same test again:

postgres@oel7:/home/postgres/ [PG9] pg_ctl -D /u02/pgdata/PG9 stop -m fast
postgres@oel7:/home/postgres/ [PG9] sudo sh -c 'echo 3 >/proc/sys/vm/drop_caches'
postgres@oel7:/home/postgres/ [PG9] pg_ctl -D /u02/pgdata/PG9 start

How fast is it?

postgres@oel7:/home/postgres/ [PG9] time psql -c "select count(*) from prewarm" postgres
  count  
---------
 1000000
(1 row)


real	0m0.623s
user	0m0.004s
sys	0m0.000s

Not as fast as with the pg_prewarm method but still faster as without any pre-warming. Maybe this is because the implementation is different, I did not test it entirely. At least it automates the save and restore of the buffer cache. For taking this into production some more testing is required.

Cet article Pre-warming the buffer cache in PostgreSQL est apparu en premier sur Blog dbi services.

There are many foreign data wrappers available for PostgreSQL. One of them can be used to query rss feeds. In this post I’ll show you how you can feed summary information of a blog into your PostgreSQL database.

The foreign data wrapper you’ll need to download and install is called multicorn. This foreign data wrapper in fact brings more than a rss wrapper which is documented here. Installation is quite easy:

postgres@oel7:/var/tmp/ [PG8] unzip multicorn-1.3.2.zip
postgres@oel7:/var/tmp/ [PG8] cd multicorn-1.3.2
postgres@oel7:/var/tmp/multicorn-1.3.2/ [PG8] make
postgres@oel7:/var/tmp/multicorn-1.3.2/ [PG8] sudo make install

This should install without any issues:

(postgres@[local]:5001) [postgres] > create extension multicorn;
CREATE EXTENSION
Time: 319.934 ms
(postgres@[local]:50
i(postgres@[local]:5001) [postgres] > \dx multi*
                                   List of installed extensions
   Name    | Version | Schema |                            Description
-----------+---------+--------+-------------------------------------------------------------------
 multicorn | 1.3.2   | public | Multicorn Python bindings for Postgres 9.2.* Foreign Data Wrapper
(1 row)

As usual we’ll have to create a foreign server on top of the extension:

CREATE SERVER rss_srv foreign data wrapper multicorn options (
    wrapper 'multicorn.rssfdw.RssFdw'
);

All we need to do from now on is to create a foreign table which will hold the summary information. The column names must much the rss item definition:

(postgres@[local]:5001) [postgres] > CREATE FOREIGN TABLE planet_postgres_blogs (
    "pubDate" timestamp,
    description character varying,
    title character varying,
    link character varying
) server rss_srv options (
    url     'http://planet.postgresql.org/rss20.xml'
);
CREATE FOREIGN TABLE

And here we go:

(postgres@[local]:5001) [postgres] > select title from planet_postgres_blogs;
                                               title                                               
---------------------------------------------------------------------------------------------------
 Robert Haas: No More Full-Table Vacuums
 Ernst-Georg Schmid: More fun with a integrarelational DBMS: SoilGrids
 Hubert 'depesz' Lubaczewski: Waiting for 9.6 – Add a generic command progress reporting facility.
 Magnus Hagander: JSON field constraints
 Robins Tharakan: Separate Index Creation from Data Population during pg_restore
 Andrew Dunstan: Json dates, times, and binary data
 Rubens Souza: And Barman 1.6.0 is out!
 Leo Hsu and Regina Obe: Paris OSGEO Code Sprint 2016 Highlights
 Paul Ramsey: Paris Code Sprint, PostGIS Recap
 Shaun M. Thomas: PG Phriday: Being A Tattletale
 Alexander Korotkov: Pg_pathman Beta Release
 Reuven Lerner: Yes, you can master regular expressions!
 solaimurugan vellaipandian: Installing TeamPostgreSQL on  64 bit Ubuntu 14.x OS
 gabrielle roth: PDXPUG: March meeting in two weeks
 Jamey Hanson: Building JSON Documents from Relational Tables
 Szymon Lipiński: Loading JSON Files Into PostgreSQL 9.5
 Reuven Lerner: Using regexps in PostgreSQL
 Hubert 'depesz' Lubaczewski: Waiting for 9.6 – Add new system view, pg_config
 Szymon Lipiński: Converting JSON to PostgreSQL values, simply
 Craig Kerstiens: Hands On Postgres Sharding
 Oleg Bartunov: 20-th Anniversary of PostgreSQL in Saint Petersburg !
 Shaun M. Thomas: PG Phriday: Corralling the Hordes
 Reuven Lerner: [Video 452] Jignesh Shah: PostgreSQL and Linux Containers
 solaimurugan vellaipandian: RPostgreSQL Data analytics on PostgreSQL data using R
 Bruce Momjian: The Plan for FDW-Based Sharding
 Gulcin Yildirim: Are we ready for Nordic PGDay?
 Szymon Lipiński: Storing Statistics JSON Data in PostgreSQL
 US PostgreSQL Association: PgDay: LFNW!
 Gulcin Yildirim: FLOSS UK Spring Conference
 Chris Travers: A couple annoyances (and solutions) regarding partitioned tables
(30 rows)

Cool, isn’t it?

Cet article Feeding blogs/rrs items directly into your PostgreSQL database est apparu en premier sur Blog dbi services.

When your infrastructure landscape becomes larger and larger you definitely need a tool which helps you in managing your servers, no matter if pḧysical or virtual, running in the cloud or not. One tool (many others are around) which assists you in provisioning, configuration management and mass deployment is Ansible. If you look at the man page you can find this sentence: “Ansible is an extra-simple tool/framework/API for doing ‘remote things’ over SSH.”. What is better than learning by doing? Lets start:

I have three CentOS 7.2 core installations:

All of these hosts are on the same CentOS release (although that does not really matter):

[root@ansible-control~]$ cat /etc/centos-release
CentOS Linux release 7.2.1511 (Core)

Obviously the first step is to get Ansible installed on the control host. There is nothing we need to install on the hosts we want to manage. Ansible does not require to setup any agents which is one of the main advantages over other tools. The only requirement is ssh. There are various ways to get Ansible installed but for the scope of this post we’ll go with yum as Ansible is available in the EPEL repository. Lets add the repository:

[root@ansible-control~]$ yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
[root@ansible-control~]$ rpm -qa | grep epel
epel-release-7-5.noarch

From there on installing Ansible is just a matter of using yum:

[root@ansible-control~]$ yum install -y ansible

At the time of writing this will install version 1.9.4 of Ansible:

[ansible@ansible-control ~]$ ansible --version
ansible 1.9.4
  configured module search path = None

Yum will take care of all the dependencies (e.g. python) and setup everything for us. Before starting to work with Ansible (because we do not want to work with root) all three hosts get a group and a user created. So, for all hosts:

[root@ansible-control~]$ groupadd ansible
[root@ansible-control~]$ useradd -g ansible ansible
[root@ansible-control~]$ passwd ansible 
[root@ansible-control~]$ su - ansible
[ansible@ansible-control~]$ ssh-keygen -t rsa

The generation of the ssh keys is important as we’ll use password-less ssh authentication for talking from the control host to the managed hosts. To succeed with this we’ll need to copy the ssh ids from the control host to the managed hosts:

[ansible@ansible-control ~]$ ssh-copy-id -i .ssh/id_rsa.pub ansible@192.168.22.171
[ansible@ansible-control ~]$ ssh-copy-id -i .ssh/id_rsa.pub ansible@192.168.22.172

How does the control host know which hosts it shall manage if nothing gets installed on the clients? Quite easy: there is a configuration file in /etc/ansible. As we do want to work with our dedicated user we’ll change the permissions:

[root@ansible-control~]$ chown -R ansible:ansible /etc/ansible/*

Using the Ansible user we can now create our first hosts file:

[ansible@ansible-control ~]$ cat /etc/ansible/hosts
[postgres-servers]
192.168.22.171
192.168.22.172

The name in brackets is a so called group name. This means be referencing “postgres-servers” in the Ansible commands Ansible resolves the group name to the server names listed for that group. Lets do a basic test to check if we can talk to our clients:

[ansible@ansible-control ~]$ ansible postgres-servers -a "/bin/echo 11"
192.168.22.171 | success | rc=0 >>
11

192.168.22.172 | success | rc=0 >>
11

Cool, works. We can run the same command on both clients. If there are more clients you can tell Ansible to use N parallel forks to execute the commands by using the “-f” switch (5 is the default):

[ansible@ansible-control ~]$ ansible postgres-servers -a "/bin/echo 11" -f 5
192.168.22.171 | success | rc=0 >>
11

192.168.22.172 | success | rc=0 >>
11

You might already guessed it from the naming of the clients that there will be something about PostgreSQL in this post. And, you’re right To outline one use case for Ansible we’ll setup PostgreSQL on two hosts in exactly the same way using Ansible in the next post.

Cet article Getting started with Ansible – Preparations est apparu en premier sur Blog dbi services.

Some time ago I blogged about how to install PL/JAVA in Postgres Plus Advanced Server 9.5. This resulted in an interesting comment from the maintainer of PL/Java. After exchanging some emails and some more testing it turned out that there really is an issue with my build of PostgreSQL. I usually compile PostgreSQL from source and use the “–with-extra-version” flag when I do the “./configure” and that seems to be an issue (at least in 1.5.0-BETA3). So, lets try to install PL/JAVA into a PostgreSQL 9.5.1 instance which was not compiled with the “–with-extra-version” option.

If you take a look at the build instructions for PL/JAVA there are a few requirements before we can start:

• We need to be able to compile and link C code
• We need a java compiler
• We need pg_config. This is available by default once PostgreSQL is installed.
• We need Maven for building PL/JAVA

As I am on a RedHat based distribution installing the C and Java compiler is just a matter of using yum:

[root@oel7 ~]$ yum install gcc-c++ gcc java-1.8.0-openjdk-devel

Once these are installed the first two requirements are met:

postgres@oel7:/home/postgres/ [PG11] g++ -version
g++: error: unrecognized command line option ‘-version’
g++: fatal error: no input files
compilation terminated.
postgres@oel7:/home/postgres/ [PG11] g++ --version
g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

postgres@oel7:/home/postgres/ [PG11] javac -version
javac 1.8.0_71

For pg_config I do not need to do anything as my PostgreSQL environment is already set:

postgres@oel7:/home/postgres/ [PG11] which pg_config
/u01/app/postgres/product/95/db_1/bin/pg_config
postgres@oel7:/home/postgres/ [PG11] pg_config
BINDIR = /u01/app/postgres/product/95/db_1/bin
DOCDIR = /u01/app/postgres/product/95/db_1/doc/postgresql
HTMLDIR = /u01/app/postgres/product/95/db_1/doc/postgresql
INCLUDEDIR = /u01/app/postgres/product/95/db_1/include
PKGINCLUDEDIR = /u01/app/postgres/product/95/db_1/include/postgresql
INCLUDEDIR-SERVER = /u01/app/postgres/product/95/db_1/include/postgresql/server
LIBDIR = /u01/app/postgres/product/95/db_1/lib
PKGLIBDIR = /u01/app/postgres/product/95/db_1/lib/postgresql
LOCALEDIR = /u01/app/postgres/product/95/db_1/share/locale
MANDIR = /u01/app/postgres/product/95/db_1/share/man
SHAREDIR = /u01/app/postgres/product/95/db_1/share/postgresql
SYSCONFDIR = /u01/app/postgres/product/95/db_1/etc/postgresql
PGXS = /u01/app/postgres/product/95/db_1/lib/postgresql/pgxs/src/makefiles/pgxs.mk
CONFIGURE = '--with-libs=/opt/local/Current/lib' '--with-includes=/opt/local/Current/include/libxml2:/opt/local/Current/include' '--prefix=/mnt/hgfs/pginstaller.auto/server/staging/linux-x64' '--with-ldap' '--with-openssl' '--with-perl' '--with-python' '--with-tcl' '--with-tclconfig=/opt/local/EnterpriseDB/LanguagePack/9.5/Tcl-8.5/lib' '--with-pam' '--enable-thread-safety' '--with-libxml' '--with-ossp-uuid' '--docdir=/mnt/hgfs/pginstaller.auto/server/staging/linux-x64/doc/postgresql' '--with-libxslt' '--with-libedit-preferred' '--with-gssapi' 'LD_LIBRARY_PATH=/opt/local/Current/lib' 'CFLAGS=-DMAP_HUGETLB=0x40000'
CC = gcc
CPPFLAGS = -D_GNU_SOURCE -I/opt/local/Current/include/libxml2 -I/opt/local/Current/include
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -DMAP_HUGETLB=0x40000
CFLAGS_SL = -fpic
LDFLAGS = -L../../../src/common -L/opt/local/Current/lib -Wl,--as-needed -Wl,-rpath,'/mnt/hgfs/pginstaller.auto/server/staging/linux-x64/lib',--enable-new-dtags
LDFLAGS_EX = 
LDFLAGS_SL = 
LIBS = -lpgcommon -lpgport -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lz -ledit -lrt -lcrypt -ldl -lm 
VERSION = PostgreSQL 9.5.1

The last requirement is Maven and this is trivial to install, too:

postgres@oel7:/home/postgres/ [PG11] mkdir -p /u01/app/postgres/product/maven
postgres@oel7:/home/postgres/ [PG11] cd /u01/app/postgres/product/maven
postgres@oel7:/u01/app/postgres/product/maven/ [PG11] wget http://mirror.easyname.ch/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
postgres@oel7:/u01/app/postgres/product/maven/ [PG11] tar -axf apache-maven-3.3.9-bin.tar.gz 
postgres@oel7:/u01/app/postgres/product/maven/ [PG11] cd apache-maven-3.3.9
postgres@oel7:/u01/app/postgres/product/maven/apache-maven-3.3.9/ [PG11] bin/mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T17:41:47+01:00)
Maven home: /u01/app/postgres/product/maven/apache-maven-3.3.9
Java version: 1.8.0_71, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.8.13-118.4.1.el7uek.x86_64", arch: "amd64", family: "unix"

That’s it. Maven is ready and we can start with building PL/JAVA ( Release 1.5.0-BETA3 for the scope of this post ):

postgres@oel7:/u01/app/postgres/product/maven/apache-maven-3.3.9/ [PG11] mkdir -p /u01/app/postgres/product/pljava
postgres@oel7:/u01/app/postgres/product/maven/apache-maven-3.3.9/ [PG11] cd /u01/app/postgres/product/pljava
postgres@oel7:/u01/app/postgres/product/pljava/ [PG11] wget https://github.com/tada/pljava/archive/V1_5_0b3.tar.gz
postgres@oel7:/u01/app/postgres/product/pljava/ [PG11] tar -axf V1_5_0b3.tar.gz 
postgres@oel7:/u01/app/postgres/product/pljava/ [PG11] cd pljava-1_5_0b3/
postgres@oel7:/u01/app/postgres/product/pljava/pljava-1_5_0b3/ [PG11] export PATH=/u01/app/postgres/product/maven/apache-maven-3.3.9/bin/:$PATH
postgres@oel7:/u01/app/postgres/product/pljava/pljava-1_5_0b3/ [PG11] mvn clean install

The last command produces plenty of output. The important notes are at the end:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] PostgreSQL PL/Java ................................. SUCCESS [  6.739 s]
[INFO] PL/Java API ........................................ SUCCESS [  5.920 s]
[INFO] PL/Java backend Java code .......................... SUCCESS [  3.444 s]
[INFO] PL/Java backend native code ........................ SUCCESS [ 46.330 s]
[INFO] PL/Java Deploy ..................................... SUCCESS [  3.397 s]
[INFO] PL/Java Ant tasks .................................. SUCCESS [  1.346 s]
[INFO] PL/Java examples ................................... SUCCESS [  3.731 s]
[INFO] PL/Java packaging .................................. SUCCESS [  7.674 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:21 min
[INFO] Finished at: 2016-03-23T10:25:14+01:00
[INFO] Final Memory: 38M/93M
[INFO] ------------------------------------------------------------------------

Looks fine so far. Following the installation instructions to install PL/JAVA:

postgres@oel7:/u01/app/postgres/product/pljava/pljava-1_5_0b3/ [PG11] java -jar pljava-packaging/target/pljava-pg9.5-amd64-Linux-gpp.jar
/u01/app/postgres/product/95/db_1/lib/postgresql/libpljava-so-1.5.0-BETA3.so as bytes
/u01/app/postgres/product/95/db_1/share/postgresql/pljava/pljava-1.5.0-BETA3.jar as bytes
/u01/app/postgres/product/95/db_1/share/postgresql/pljava/pljava-api-1.5.0-BETA3.jar as bytes
/u01/app/postgres/product/95/db_1/share/postgresql/pljava/pljava-examples-1.5.0-BETA3.jar as bytes
/u01/app/postgres/product/95/db_1/share/postgresql/extension/pljava.control as lines (ASCII)
/u01/app/postgres/product/95/db_1/share/postgresql/pljava/pljava--1.5.0-BETA3.sql as lines (UTF8)
/u01/app/postgres/product/95/db_1/share/postgresql/pljava/pljava--unpackaged--1.5.0-BETA3.sql as lines (UTF8)
/u01/app/postgres/product/95/db_1/share/postgresql/pljava/pljava--1.5.0-BETA2--1.5.0-BETA3.sql as lines (UTF8)
/u01/app/postgres/product/95/db_1/share/postgresql/pljava/pljava--1.5.0-BETA1--1.5.0-BETA3.sql as lines (UTF8)

Looks fine, too. Everything needed by PostgreSQL to install the extension should be there now. Lets try:

postgres@oel7:/u01/app/postgres/product/pljava/pljava-1_5_0b3/ [PG11] psql postgres
Null display is "NULL".
Timing is on.
psql.bin (9.5.1)
Type "help" for help.

(postgres@[local]:7777) [postgres] > create extension pljava;
CREATE EXTENSION
Time: 1947.125 ms
(postgres@[local]:7777) [postgres] > \dx
                                   List of installed extensions
  Name   |   Version   |   Schema   |                         Description                          
---------+-------------+------------+--------------------------------------------------------------
 pljava  | 1.5.0-BETA3 | sqlj       | PL/Java procedural language (https://tada.github.io/pljava/)
 plpgsql | 1.0         | pg_catalog | PL/pgSQL procedural language
(2 rows)

Ready to be used …

Cet article Install PL/JAVA in PostgreSQL 9.5.1 est apparu en premier sur Blog dbi services.