Tuesday 26 November 2013

Configure Hot Standby with Streaming Replication on Postgresql 9.1 and above

NOTE: Please note that this does NOT work on Windows! Backups only work on Unix machines.

NOTE: We need at least postgres 9.1 to make this back up work.

NOTE: You must be admin to restart the server. And you must be user postgres to make edits to the configuration files. All edits assume your base directory is /var/lib/pgsql/

Hot Standby is the name for the capability to run queries on a database that is currently performing archive recovery. Log Shipping replication allows you to create one or more standby nodes that are replicas of the primary node (or master node). Standby nodes can then be used for read-only query access.

To enable hot standby, make the following changes to the postgres.conf on the master:
wal_level = hot_standby
checkpoint_segments = 8
max_wal_senders = 3
wal_keep_segments = 8
Please note that the segment size of the write ahead log (wal) is a fixed 16MB. Should the server undergo huge stress, it could be that the streaming process cannot keep up with WAL, in which case, some data might be lost. To prevent this, increase the size of checkpoint_segments and wal_keep_segments.

Add the following changes to pg_hba.conf on the master:
host     replication     postgres        192.168.2.37/32         trust
Where 192.168.2.37 is the slave. The above line is saying that we would like the postgres user on host 192.168.2.37 to gain replication access to this master server.

Save these changes and restart the server. It is now time to configure the slave server.

First stop the postgres server on the slave and create a backup of the existing server. I do this by moving the entire data directory to old.
mv 9.2 9.2.old
Now that we have an empty data directory, copy all the contents from the master database with the pg_basebackup command. Note: this only works on versions of postgres greater than or equal to 9.1
pg_basebackup -D 9.2/data --host=10.1.1.7 -v -P -xlog -U postgres 
Where 10.1.1.7 is your master server. This in essence copies the entire database over to the slave. Now we need to make some edits. If your access for the master and the slave are the same, you do not need to edit pg_hba.conf. Edit postgresql.conf:
hot_standby = on
This tells postgres that the server should act as a standby server.

Now we need to create a new file, called recovery.conf. In essence this file contains the information of the master server:
standby_mode = 'on'
primary_conninfo = 'host=10.1.1.7 port=5432 user=postgres password=postgres'
Now start the slave server. You must verify that the server is in recovery; this can be done by opening an sql pane and executing the query:
select pg_is_in_recovery();

Tuesday 19 November 2013

Hibernate Associations and Discards/Evicts

Right, so your sitting at work thinking how life would be without the Hibernate ORM...and you would go.... very frustrating indeed. Most of the time, you'd be busy mapping your SQL to your Java instances and vice versa.

But then again, Hibernate does come with its own set of headaches. Here is something I learnt recently.. that hibernate will not tell you.

You have your domain instance and your hibernate session and for whatever reason, you decide to evict it (also known as discard in the Grails world). For single entities, you see that this works just fine...but most system work with a graph of entities....not just one.

Suppose you have domain class A that has one or more class B objects. Should you decide to evict on class A, and you find class B is dirty (meaning some attribute has changed there), you will see that your discard just gets ignored. After all, if class B is updated and class A relies on it, the relationship between A and B needs to get refreshed and as a result, Hibernate just cannot ignore not persisting A.

So what is the solution you ask....you need to evict EVERY B as well as A. Essentially, this tells hibernate that nothing is dirty and now finally, A will not get persisted.

Friday 15 November 2013

Grails Data Binding Gotcha

Grails Data Binding is one of the most powerful tools I've used in the Web MVC world. Not only does it automatically bind correctly to data types, but you can also bind to lists and maps.

So yesterday, I was building an app; and following was the code:
 1 <g:each in="${agents}" var="agent" status="i">
 2  <!-- start row -->
 3  <div class="row">
 4   <div class="span12 control-group-holder">
 5    <div class="row">
 6     <div class="span4 control-group">
 7      <g:include params="[ext_id: agent.ext_id, readonly: true]" controller="agent"
 8           action="show"/>
 9      <g:hiddenField name="agents[$i].ext_id" value="${agent.ext_id}"/>
10      <g:hiddenField name="agents[$i].version" value="${agent.version}"/>
11     </div>
12 
13     <div class="span8 control-group">
14      <g:radioGroup name="agents[$i].photoStatus" class=""
15           labels="${photoStatusValues*.name()*.toLowerCase()*.capitalize()}"
16           values="${photoStatusValues}" value="${agent.photoStatus}">
17       <label class="radio inline" style="margin-left: 0px;">${it.radio} ${it.label}</label>
18      </g:radioGroup>
19      <div style="visibility: hidden" class="photoDeniedReason">
20      <g:message code="text.admin.reviewPhotos.reason.for.rejection" /><br/>
21      <textarea name="agents[${i}].photoDeniedReason" maxlength="4000" class="rejection_reason_required"></textarea>
22      </div>
23     </div>
24    </div>
25   </div>
26  </div>
27  <hr/>
28  <!-- end row -->
29 </g:each>
This is being bound to the command object:
 1 @Validateable
 2 class BatchUpdatePhotosCommand {
 3  List<Agent> agents = new ArrayList<Agent>()
 4 }
 5 
 6 @Validateable
 7 class AgentPhotoCommand {
 8  String ext_id
 9  String version
10  String photoStatus
11 }

Except that it just would not bind. I would get:
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.ArrayList.RangeCheck(ArrayList.java:547) at java.util.ArrayList.get(ArrayList.java:322) at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:195) at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662)
The solution was updating BatchUpdatePhotosCommand with the 'withLazyDefault' and everything worked:
 1 @Validateable
 2 class BatchUpdatePhotosCommand {
 3  List<AgentPhotoCommand> agents = new ArrayList<AgentPhotoCommand>().withLazyDefault {new AgentPhotoCommand()}
 4 }
 5 
 6 @Validateable
 7 class AgentPhotoCommand {
 8  String ext_id
 9  String version
10  String photoStatus
11 }