GSoC_2015_scalability

GSoC 2015 project reporting page

GSoC student: Ambrosini Luca

Project outline

istSos is a server implementation that collects,redistribute and manages data received from sensor according to SOS. The server need to store and retrieve data in a database, based on postgresSQL. This software is heavily dependent on database performance and also needs a database with the lowest possible downtime.
To fulfill big data requirements database change is needed. PostgreSQL+Pgpool—2 allows connections pooling, load balancing, query partitioning and database replication.
Using pgpool-2 between clients and database will improve istSOS performance and his reliability.

[Ambrosini Luca]

** Week 1 ****

What do i have completed this week ?

• Installed istSOS VM
• Installed pgpool server on istSOS VM

What am I going to achieve for next week?

• Set up a wiki page for my report
• Configure pgpool replication
• Test pgpool speedup

Is there any blocking issue?

No blocking issue

[Ambrosini Luca]

** Week 2 ****
What do i have completed this week ?

• Splitted server roles
• Installed/Configured postegresql + pgAdmin
• Configured pgpool2 + pgpool2admin
• Created Wiki page (https://sourceforge.net/p/istsos/wiki/GSoC_2015_scalability/ )

What am I going to achieve for next week?

• Configure database master-slave replication
• Test database master-slave replication

Is there any blocking issue?

• Had some issues in finding compatible versions, but now everything seems to be ok

[Ambrosini Luca]

** Week 3 ****
What do i have completed this week ?

• Reviewed last weeks work ,switched to more LTS versions and for documentation/compatilibity
  • Migrated from Vbox to VmWare
  • Downgraded from ubuntu 15.04 to 14.04
  • Downgraded from postgres 9.4 to 9.3.8
  • Analyzed pgpool methods and switched from pgpool master/slave to replication

What am I going to achieve for next week?

• Wrote documentation/guide for pgpool/postgres installation and configuration
• Start replication

Is there any blocking issue?

• At pgpool start i got an shmem_exit(1) error. I've wrote to pgpool mailing list and i waiting for some help in debugging.

[Ambrosini Luca]

** Week 4 ****
What do i have completed this week ?

• Installed pgpool extension for postgres on clients
• Succeded in connecting client to master sever
• Fixed permission to allow create and insert from pgpool master sever

What am I going to achieve for next week?

• Replicating query with pgpool and not with hardcoded scripts

Is there any blocking issue?

• No

This last week end the next one i will be on my summer exams, so i'll try to work as much as possible

[Ambrosini Luca]

** Week 5 ****

What do i have completed this week ?

• Installed pgpool extension for online_recovery on client and server
• Succeded in replication
• Start writing documentation

What am I going to achieve for next week?

• Write hotplug node scripts
• Install pgpool on istSos simulated network

Is there any blocking issue?

• No

[Ambrosini Luca]

** Week 6 ****

What do i have completed this week ?

• Wrote documentation and guide for replication
• Developed online recovery scripts
• Configured ssh-without authentication

What am I going to achieve for next week?

• Completely fix online recovery
• Wrote online recovery documentation

Is there any blocking issue?

• No

[Ambrosini Luca]

** Week 7 ****
What do i have completed this week ?

• Fixed online recovery and hot plug bash scripts
• Fixed pgpool 3.3.4 with postgresql 9.4
• Wrote online recovery documentation

What am I going to archieve for next week?

• Implements pgpool on istSOS Db

Completely fix online recovery

Is there any blocking issue?

• No

[Ambrosini Luca]

** Week 8 ****

What do i have completed this week ?

• Tested pgpool pitr recovery scripts
• Installed istsos and pgpool network

What am I going to achieve for next week?

• Test and eventually debug problems with istSOS pgpool version

Is there any blocking issue?

• No blocking issue

[Ambrosini Luca]

** Week 9 ****

What do i have completed this week ?

• Tested istSOS with pgpool data backend
• Started istSOS benchmarking system development
• Implemented getCapabilities, descriveSensor, getObservation (90%) benchmark functions

What am I going to achieve for next week?

• Add random time slice for getObservation
• Implement registerSensor and insertObservation benchmark functions

Is there any blocking issue?

• No blocking issue

[Ambrosini Luca]

* Week 10 ***

What do i have completed this week ?

• Completed getObservation (random time slice choice)
• Implemented registerSensor, insertObservation
• Added benchmark configuration files

What am I going to achieve for next week?
- Debug benchmark functions in details
- Start using benchmark to see pgpool performance improvements

Is there any blocking issue?

• No blocking issue

[Ambrosini Luca]

* Week 11 ***

What do i have completed this week ?

• Improved performance and bug fixing
• Implemented intensive insertObservation for network creation
• Start configuring benchmark network

What am I going to achieve for next week?

• Write data logger script for test execution
• Complete benchmark network
• Start benchmarking

Is there any blocking issue?

• No

[Ambrosini Luca]

* Week 12 ***

What do i have completed this week ?

• Added manual log feature in locust
• Switched to locust.io forked version with time control
• Wrote demo-data creator script
• Wrote simple grapher and data analyser for test result analysis
• Executed tests with different apache mpm_xxx, hardware and user numbe

What am I going to achieve for next week?

• Execute test with different sensor network
• Execute test with pgpool network
• Wrote documentations and examples

Is there any blocking issue?

• No

[Ambrosini Luca]

* Week 13 *

What do i have completed this week ?

• Completed test
• Modified IO and RS response routines to avoid replication problems
• Completed GSOC documentation
• Submitted GSOC final evalutation