On November 5 (Monday) following an individual meeting with Dr. Fujinoki, I talked to Greg Bartholomew. He informed us that the CS department has provided us with the Mac Pro server in the Server room of the tutor lab.
Ubuntu Server 12.04.1 LTS ("long term support") is installed on the server. Static IP addressing was already set up.
The server can be accessed via SSH using either the domain name (recommended) or the IP address.
The domain name of the server is: cloud.cs.siue.edu
The IP address of the server is: 146.163.150.19
The server can be access off campus.
There were some problems with the server when we first received it.
Much of the software that is standard with Ubuntu Server 12.04 was not present, and using apt-get to retrieve software was broken. apt-get would only install software from a physical CD (which was not in the server), and not from the online repositories. Greg and Isaac were contacted to help open the CD-ROM tray (apparantly there was no button to open the drive on the server case, and opening the CD-ROM tray from the bash shell was glitched). This has since been fixed, and software can now be downloaded from the online repositories.
The server has two hard disks. The first of these disks (/dev/sda) appears to contain the stock Mac Server OS.
For our project, ideally we will want to use two disks - one disk will contain the Ubuntu operating system, boot manager, and swap space (this appears to have been set up on /dev/sdb; the second hard drive), and the other disk will contain logical volumes (every time a virutal machine is spawned, a logical volume is created on the disk; in other words the logical volume acts as a virtual harddrive for the VM).
I have contact Jared Edens (via cs-support@siue.edu) to ask if it is okay to format /dev/sda. If we're not allowed to format that disk, however, we can probably settle for dividing the main partition on /dev/sdb into two partitions. In that case, the OS will be installed on one half, and the volumes will be stored on the other half.
The following still needs to be done during this sprint:
I need to install an FTP server.
I need to install OpenStack.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I recommend placing a PuTTY shortcut to each of the three servers in our project on your desktop. This way, we can access a server via SSH just by double-clicking it's icon. To do this:
Find your putty.exe.
Right click it
Choose "Copy"
Go to your Desktop
Right click it
Choose "Paste Shortcut"
Rename the shortcut to something indicating what server it connects to.
Example: Putty (cloud.cs.siue.edu)
Another: Cloud Server
Right click the shortcut.
Choose properties.
In the "Target" box, put this at the end, after putty.exe:
-ssh -P 22 cloud.cs.siue.edu
It should now look something like
E:\Putty\putty.exe -ssh -P 22 cloud.cs.siue.edu
Click OK to save changes.
Double click the icon and try to connect. If it works it should open a terminal window right away, or ask you to add an RSA fingerprint to the registry or something.
Do this for each server (note you can just copy/paste the shortcuts and modify their properties):
cloud.cs.siue.edu
cloudoa.cs.siue.edu
cloudui.cs.siue.edu
You're done!
Things that still need to be done with regards to the UI / Owner Agent servers:
PHP needs to be installed on the Webserver and configured to work with nginx
FTP servers need to be set up on the UI and Owner Agent servers
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Jared Edens replied to me and requested that the second disk should NOT be formatted. He did, however, give us permission to split the Ubuntu partition into two halves.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
FTP is now available on the Owner Agent and User Interface servers.
I still need to set up FTP on the Cloud server. I'll probably do that in the meeting today at 5:00 PM.
Or, if you need to upload files or download several files at once, you can access FTP through an FTP client (I recommend FileZilla).
Here are some instructions on using Filezilla to access FTP:
Download the FileZilla client.
Extract/install it somewhere.
Run the FileZilla client.
For the IP/Hostname any of these should work:
cloudui.cs.siue.edu
cloudoa.cs.siue.edu
cloud.cs.siue.edu
For the port, you can leave it blank.
For the username, type your username (see this for usernames if your forgot)
For the password, type your password (again see this if you forgot)
Click on the Quick Connect button to connect to the server. If it works it should display a list of files in the right pane, and give you some confirmation in the console below the toolbar.
From here, you can transfer files. The pane on the left side of the screen displays files on your local machine. The pane on the right side of the screen displays files on the remote machine. Double click on a file on either side to upload/download it to the other side. You can also select multiple files and drag/drop to the other side.
Once you're done transferring files hit the disconnect button and close FileZilla.
Note: FileZilla has a Server Browser feature that lets you save commonly used servers in a drop-down list. I'd recommend making some entries for cloudos, cloudui, and cloud.
Last edit: Jeff Hutchins 2012-11-08
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Today (November 8) was not very productive. [date corrected from November 8 to November 9, then back to November 8 on 12/5/2012]
Before OpenStack can be installed on the Cloud Server, I need to resize the partition the Ubuntu Server OS resides on. However, I can't do this while Ubuntu Server is running, because the OS mounts the filesystem on that partition when it boots.
In other words, I need to boot an OS with a partition editor that DOESN'T mount the file system on the Ubuntu partition. The best way to do this is with a live CD (meaning we need physical access to the server to insert the disk / work with the keyboard & mouse). Live media lets you run an Operating System off of a Floppy, Flash Drive, or CD-ROM. They don't store files on the hard disk - instead, they create a RAM disk (a temporary "hard drive" that exists in RAM).
I believed that the Ubuntu Server installation disk was a Live CD, like Arch Linux's CD was, but apparently the Ubuntu Server disk doesn't work like that. The Ubuntu Desktop CD does, however. Unfortunately I didn't have the Arch / Ubuntu desktop disks with me at the time, and Greg didn't seem to know where a Live CD would be either. I'll bring the Arch and Ubuntu desktop disks with me... Friday, if I have the time.
The other option was booting into the Mac OS on the Cloud Server and somehow editing the partitions on that server, but this failed as well; it was password protected and the person who knew the password wasn't present at the time.
Besides that, we attempted to install PHP to the UI server and configure it with nginx, but this seems to have failed miserably. We can't even connect to the server locally or remotely, so I'll likely need to give it some maintenance in person (perhaps the networking service needs to be restarted or something). We installed the wrong packages the first time around and uninstalled them. There may also be configuration errors with the nginx package (I had to leave for a class before I could resolve these issues, and can't fix them from home at the moment).
Last edit: Jeff Hutchins 2012-12-05
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Friday November 9 at about 1:00 PM,
I was in the tutor lab downstairs trying to resize the partition on the cloud server with a Live CD. I wasn't available for the meeting with Dr. Fujinoki 2:00 because it was one hour before my lab and I needed the time to prepare for it. In the future please try to schedule these meetings earlier if possible (preferably around 10:00 AM).
I successfully resized it, but in the process it seems to have broken the Ubuntu Server installation on the cloud server. I then tried to reinstall Ubuntu Server 12.04 on the cloud server and I believe I installed the boot loader to the wrong partition. I ran out of time to fix the problem, so I'll be doing this work on Saturday. I'm scheduling a meeting in case anyone wants to attend.
Last edit: Jeff Hutchins 2012-11-11
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Reinstalled Ubuntu Server 12.04 on the cloud server on Saturday at around 1:00 PM after being let into the server room by Jared Edens. But SSH on the server is broke now; I didn't realize this until Jared had left. Since I can't log into the server remotely, I'll have to wait until Monday so the CS support staff can let me into the server room again and I can fix the issue in person.
Also, I fixed a problem with the UI server, apparently this is a bug with Ubuntu Server and can occur on any of the 3 machines we have it installed on. Basically, if you have a static IP address set up on an Ubuntu Server machine and you restart the server, the DNS hosts specified in the /etc/network/interfaces aren't written to the /etc/resolv.conf file. So, when the server tries to turn a domain name (e.g. cloudoa.cs.siue.edu or www.google.com ) into an IP address, this will fail with a name resolution error. Note, when this happens, we will still be able to connect to the server remotely, the server itself will just have trouble turning hostnames into IPs, so things like updating / installing packages could fail.
A temporary fix to this problem is:
Edit /etc/network/interfaces with nano by doing:
sudo nano /etc/network/interfaces
Change "static" to "dhcp"
Comment out all the lines that come after that line by placing # at the beginning of the line.
Save the file.
Restart the networking service by doing:
sudo /etc/init.d/networking restart
Edit the /etc/network/interfaces file again:
sudo nano /etc/network/interfaces
Change "dhcp" back to "static"
Uncomment the lines we commented out before by getting rid of the # at the beginning of those lines
Save the file.
Restart the networking service again:
sudo /etc/init.d/networking restart
The good news is that we got PHP working on the UI server. We've started working on a login screen. Here are the pages we have so far:
There seems to be some kind of issue accessing the UI server. I can't access the website or log in through PuTTY.
I suspect someone has turned the machine off or restarted it.
There is a problem on the UI Server machine; whenever it boots, it will get stuck at a screen that says "Could not find SATA device 1, press any key to continue". If the servers were restarted, chances are it's probably stuck at this screen, waiting for someone to press a key. After the key is pressed, Ubuntu Server will boot like it normally does.
My guess as to why it can't find SATA device 1... it's probably referring to a hard disk that was once connected to the computer via the 1st SATA port on the motherboard. Maybe at one time, the computer had two disks installed in it, but the 1st one was removed? That would mean we installed Ubuntu Server on the disk connected on the 2nd SATA port. The boot order is probably set up to boot from the device at SATA 1 (which I would reason is still enabled in the BIOS, even though it's no longer present) before booting from SATA 2. This is just a theory, however.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
November 12, 2012.
Fixed two problems with the UI server.
The BIOS on the UI Server shows that SATA0 and SATA1 were enabled.
The disk we installed Ubuntu Server to is connected to SATA0.
And as expected, SATA1 was enabled, but had no drive connected to it.
I disabled SATA1 and rebooted the machine. It no longer gets stuck at the "Could not find SATA device 1" screen.
This issue has been fixed, but it turns out it wasn't the same issue that caused our server to go offline. The server was still running when I arrived. I ran the following command when I logged on to the machine:
ifconfig
Then I noticed the IP address was different than what it was supposed to be (it should end in 51, but ended in 182 instead).
Turns out, due to a bug in Ubuntu Server, after editing /etc/network/interfaces to use DHCP, restarting the networking, editing /etc/network/interfaces to use a static address, then restarting the networking again, the DHCP client on the machine keeps running.
So, what does this mean? When using DHCP, the machine automatically acquires an IP address from the SIUE network. DHCP addresses have a "lease" on them - in other words, they're only allowed to use these address for a certain amount of time before they expire. When they expire, it's the job of the DHCP client to acquire a new address from the SIUE network.
Because we're using a static IP address, however, the IP address of the machine is never supposed to change (cloud.cs.siue.edu always points to 146.163.150.51). But because the DHCP client kept running, whenever the old DHCP address expired, it replaced our static IP address with a new DHCP address!
I think I've fixed the problem for now. I looked to see if the DHCP server was running:
ps -ef | grep dhclient
Then I saw a line that looked something like this:
This means that the DHCP client is running, and it's process ID (the second number) is 2563. To kill the DHCP client, I did:
sudo kill 2563
This should fix the problem in case we encounter it again. (it should happen any time we use the fix for the name resolution problem I described in the two posts above)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Attempted to install OpenStack on the cloud server 11/14/2012.
During the installation we ran into some kind of configuration error. Not clear on what is going wrong here.
Here's what we did
Was installing OpenStack Compute, entered the following values when prompted:Enter the primary ethernet interface IP: 146.163.150.19Enter the fixed network (eg. 10.0.2.32/27): 146.163.150.53/32Enter the fixed starting IP (eg. 10.0.2.33): 146.163.150.53#######################################################################################The floating range can be a subset of your current network. Configure your DHCP serverto block out the range before you choose it here. An example would be 10.0.1.224-255#######################################################################################Enter the floating network (eg. 10.0.1.224/27): 146.163.150.54/32Enter the floating netowrk size (eg. 32): 1Log:2012-11-14 17:42:13 DEBUG nova.utils [-] backend <module 'nova.db.sqlalchemy.migration' from '/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.pyc$2012-11-14 17:43:04 DEBUG nova.utils [-] backend <module 'nova.db.sqlalchemy.migration' from '/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.pyc$2012-11-14 17:44:06 WARNING nova.utils [-] /usr/lib/python2.7/dist-packages/sqlalchemy/pool.py:639: SADeprecationWarning: The 'listeners' argument to Pool (and$Pool.__init__(self, creator, **kw)2012-11-14 17:44:06 WARNING nova.utils [-] /usr/lib/python2.7/dist-packages/sqlalchemy/pool.py:145: SADeprecationWarning: Pool.add_listener is deprecated. Use$self.add_listener(l)2012-11-14 17:44:06 AUDIT nova.db.sqlalchemy.fix_dns_domains [-] Applying database fix for Essex dns_domains table.2012-11-14 17:44:07 DEBUG nova.utils [req-bd5b0d80-43dd-4fb8-940a-cddeff8a9190 None None] backend <module 'nova.db.sqlalchemy.api' from '/usr/lib/python2.7/dis$2012-11-14 17:44:07 CRITICAL nova [req-bd5b0d80-43dd-4fb8-940a-cddeff8a9190 None None] index out range for address range size!2012-11-14 17:44:07 TRACE nova Traceback (most recent call last):2012-11-14 17:44:07 TRACE nova File "/usr/bin/nova-manage", line 1746, in <module>2012-11-14 17:44:07 TRACE nova main()2012-11-14 17:44:07 TRACE nova File "/usr/bin/nova-manage", line 1733, in main2012-11-14 17:44:07 TRACE nova fn(*fn_args, **fn_kwargs)2012-11-14 17:44:07 TRACE nova File "/usr/bin/nova-manage", line 812, in create2012-11-14 17:44:07 TRACE nova fixed_cidr=fixed_cidr)2012-11-14 17:44:07 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1365, in create_networks2012-11-14 17:44:07 TRACE nova net['gateway'] = gateway or str(subnet_v4[1])2012-11-14 17:44:07 TRACE nova File "/usr/lib/python2.7/dist-packages/netaddr/ip/__init__.py", line 713, in __getitem__2012-11-14 17:44:07 TRACE nova raise IndexError('index out range for address range size!')2012-11-14 17:44:07 TRACE nova IndexError: index out range for address range size!2012-11-14 17:44:07 TRACE nova
Please advise!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Megha and I discovered what was causing the previous problem above. The problem is the range of IP addresses we specify for the fixed network must contain at least 4 IP addresses. This was not specified anywhere in the documentation - we discovered it after looking in the scripts the errors were occurring in and seeing the assumption first-hand.
We need to reserve a range of IP addresses before spawning VMs. After that Dr. Fujinoki wanted me to set up an "image" with the cloud software set up on it but I don't have any idea how to do this. This page goes into it but reading through it I still have no idea what I am supposed to do. All of this needs to be done within the next week.
I'm sorry but I'm just not going to be able to get the cloud server set up. It's proving to be far more difficult than I anticipated and we're running out of time. I keep encountering significant impediments when trying to work with the cloud:
No background in the technologies OpenStack employs (e.g. virtual machines, networking)
OpenStack documentation and guides are too difficult to understand.
"Cloud software" is a new thing, less than 2 years old - it's so esoteric I can't Google for answers to the problems I'm encountering.
OpenStack's error handling isn't very good; like you can see above, it spits out a non-specific error and a stack-trace. The error logs are huge and I can't find any relevant information in them.
Physical access to our server is restricted and requires us to work CS support schedule.
Network configuration (such as reserving blocks of IP addresses on the SIUE network) is also restricted and requires us to work through the SIUE network. We can't do this on demand; every time we discover we need more IP addresses we need to wait until CS support is available to do this for us.
So, because of all the issues I'm running into, and the limited amount of time we have remaining, I sent an e-mail to Dr. Fujinoki asking for him to settle for alternatives.
In place of the Cloud Server we should probably use the time we have remaining to work on the UI and Owner agent programs. MAYBE we can get the cloud software working next semester.
Here is a copy of the e-mail I sent to Dr. Fujinoki:
Dr. Fujinoki,
I've been trying to get the cloud software set up for the past two to three
weeks, and I can't make any progress with it whatsoever.
We need to present our project's progress to a December 7, 2012, and the way
things are going we will have nothing to show for it.
Nobody in this university, least of all our team, has any background with any of
this. We keep running into so many problems trying to get this working, and this
cloud technology is so new and esoteric, we just can't find any solutions to
these problems online. All of the documentation for OpenStack and guides I can
find online are targeted towards IT experts who do this kind of thing for a
living; this isn't me. I have no background in any of this - I just can't get it
working!
To put it simply: We can't set up a cloud server. We're running out of time and
we need to consider alternatives.
Would you be willing to settle for a single PC running a database and the
Lineage Layer software?
--Jeff H.
SIUE Web Mail
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I created a database for the cloud server for the data that will be provided by USTRANSCOM.
I structured the database exactly according to the clients specifications. I have saved and uploaded the SQL queries in the repository for everyone's review. Please look it over and let me know if there is anything that is missing. We may have to see how the tables in the database interact with each other and decide if we need any foreign keys, constraints or triggers.
For this semester we will have to create our own dummy data to populate these tables.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I added two databases to the UI server. One to store the user names along with their passwords and the other for the actual USTRANSCOM data.
They are named as follows:
user_list
main_records
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The UI server is inaccessible until Monday. The bug where the UI server couldn't resolve hosts popped up again (someone probably restarted the servers), so I went to fix it like I outlined in posts above, except I made a mistake.
Instead of calling /etc/init.d/networking restart, you're supposed to call ifdown eth0, followed by ifup eth0 (because /etc/init.d/networking restart is deprecated).
So, to shut down the networking I ran:
ifdown eth0
then I was going to type this to bring it back up:
ifup eth0
But running ifdown shut down my SSH connection and made it impossible to reconnect, since I never got a chance to issue the ifup.
I went downstairs to manually turn it back on, but the doors to the room they're stored in were locked, and nobody was available to let us in! What a surprise.
For future reference, my mistake was running ifdown on one line, then trying to run ifup on another line. I believe I should have done this:
ifdown eth0; ifup eth0;
So, sorry guys.
Also, I think at the meeting with Fujinoki on Monday we should ask for permission to move the servers to the Senior Project lab, or ask for the keys we need to access the room the machines are stored in or something.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Kyle and I got a dummy security gateway and a functional authorization server working.
The authorization server accepts a connection, waits for a username and password to be sent, then queries a database of users on the Owner Agent server. If a username and password is found, a "1" is sent back. Otherwise, a "0" is sent back.
The dummy security gateway is basically just a proxy at this point. You pass it a username and password, and it sends it to the authorization server. Whatever the authorization server responds with is forwarded back to the client.
Anyway, when the UI server comes back online, we can actually handle logins by authenticating via the security gateway, as was originally intended. Of course, if this proves too problematic in the coming week, we can always fall back on the list of users on the UI server.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Made some modifications to the login.php on the Cloud UI server.
I rewrote it to contact the security gateway and authenticate the UserID and password given to the script. Tested and it works!
To add more user logins, use mysql to add them to the users table in the userList database on the owner agent.
I'm going to commit our html directory's contents to the SVN.
I'm also going to commit Kyle's security gateway / authentication server code & the compile shell script we've been using.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
On November 5 (Monday) following an individual meeting with Dr. Fujinoki, I talked to Greg Bartholomew. He informed us that the CS department has provided us with the Mac Pro server in the Server room of the tutor lab.
Ubuntu Server 12.04.1 LTS ("long term support") is installed on the server. Static IP addressing was already set up.
The server can be accessed via SSH using either the domain name (recommended) or the IP address.
The domain name of the server is: cloud.cs.siue.edu
The IP address of the server is: 146.163.150.19
The server can be access off campus.
There were some problems with the server when we first received it.
Much of the software that is standard with Ubuntu Server 12.04 was not present, and using apt-get to retrieve software was broken. apt-get would only install software from a physical CD (which was not in the server), and not from the online repositories. Greg and Isaac were contacted to help open the CD-ROM tray (apparantly there was no button to open the drive on the server case, and opening the CD-ROM tray from the bash shell was glitched). This has since been fixed, and software can now be downloaded from the online repositories.
The server has two hard disks. The first of these disks (/dev/sda) appears to contain the stock Mac Server OS.
For our project, ideally we will want to use two disks - one disk will contain the Ubuntu operating system, boot manager, and swap space (this appears to have been set up on /dev/sdb; the second hard drive), and the other disk will contain logical volumes (every time a virutal machine is spawned, a logical volume is created on the disk; in other words the logical volume acts as a virtual harddrive for the VM).
I have contact Jared Edens (via cs-support@siue.edu) to ask if it is okay to format /dev/sda. If we're not allowed to format that disk, however, we can probably settle for dividing the main partition on /dev/sdb into two partitions. In that case, the OS will be installed on one half, and the volumes will be stored on the other half.
The following still needs to be done during this sprint:
I need to install an FTP server.
I need to install OpenStack.
The owner agent and UI agent servers are up and running.
Megha, Chris, and I have completed the following:
In addition, I have tested the connection: both machines can be accessed from off campus.
The domain name and IP address for the UI server:
The domain name and IP address for the Owner Agent server:
I recommend placing a PuTTY shortcut to each of the three servers in our project on your desktop. This way, we can access a server via SSH just by double-clicking it's icon. To do this:
Rename the shortcut to something indicating what server it connects to.
Right click the shortcut.
Choose properties.
In the "Target" box, put this at the end, after putty.exe:
It should now look something like
Click OK to save changes.
Double click the icon and try to connect. If it works it should open a terminal window right away, or ask you to add an RSA fingerprint to the registry or something.
Do this for each server (note you can just copy/paste the shortcuts and modify their properties):
You're done!
Things that still need to be done with regards to the UI / Owner Agent servers:
Jared Edens replied to me and requested that the second disk should NOT be formatted. He did, however, give us permission to split the Ubuntu partition into two halves.
FTP is now available on the Owner Agent and User Interface servers.
I still need to set up FTP on the Cloud server. I'll probably do that in the meeting today at 5:00 PM.
You can access FTP through your web browser:
ftp://cloudui.cs.siue.edu/
ftp://cloudoa.cs.siue.edu/
Or, if you need to upload files or download several files at once, you can access FTP through an FTP client (I recommend FileZilla).
Here are some instructions on using Filezilla to access FTP:
Once you're done transferring files hit the disconnect button and close FileZilla.
Note: FileZilla has a Server Browser feature that lets you save commonly used servers in a drop-down list. I'd recommend making some entries for cloudos, cloudui, and cloud.
Last edit: Jeff Hutchins 2012-11-08
Today (November 8) was not very productive. [date corrected from November 8 to November 9, then back to November 8 on 12/5/2012]
Before OpenStack can be installed on the Cloud Server, I need to resize the partition the Ubuntu Server OS resides on. However, I can't do this while Ubuntu Server is running, because the OS mounts the filesystem on that partition when it boots.
In other words, I need to boot an OS with a partition editor that DOESN'T mount the file system on the Ubuntu partition. The best way to do this is with a live CD (meaning we need physical access to the server to insert the disk / work with the keyboard & mouse). Live media lets you run an Operating System off of a Floppy, Flash Drive, or CD-ROM. They don't store files on the hard disk - instead, they create a RAM disk (a temporary "hard drive" that exists in RAM).
I believed that the Ubuntu Server installation disk was a Live CD, like Arch Linux's CD was, but apparently the Ubuntu Server disk doesn't work like that. The Ubuntu Desktop CD does, however. Unfortunately I didn't have the Arch / Ubuntu desktop disks with me at the time, and Greg didn't seem to know where a Live CD would be either. I'll bring the Arch and Ubuntu desktop disks with me... Friday, if I have the time.
The other option was booting into the Mac OS on the Cloud Server and somehow editing the partitions on that server, but this failed as well; it was password protected and the person who knew the password wasn't present at the time.
Besides that, we attempted to install PHP to the UI server and configure it with nginx, but this seems to have failed miserably. We can't even connect to the server locally or remotely, so I'll likely need to give it some maintenance in person (perhaps the networking service needs to be restarted or something). We installed the wrong packages the first time around and uninstalled them. There may also be configuration errors with the nginx package (I had to leave for a class before I could resolve these issues, and can't fix them from home at the moment).
Last edit: Jeff Hutchins 2012-12-05
Friday November 9 at about 1:00 PM,
I was in the tutor lab downstairs trying to resize the partition on the cloud server with a Live CD. I wasn't available for the meeting with Dr. Fujinoki 2:00 because it was one hour before my lab and I needed the time to prepare for it. In the future please try to schedule these meetings earlier if possible (preferably around 10:00 AM).
I successfully resized it, but in the process it seems to have broken the Ubuntu Server installation on the cloud server. I then tried to reinstall Ubuntu Server 12.04 on the cloud server and I believe I installed the boot loader to the wrong partition. I ran out of time to fix the problem, so I'll be doing this work on Saturday. I'm scheduling a meeting in case anyone wants to attend.
Last edit: Jeff Hutchins 2012-11-11
Reinstalled Ubuntu Server 12.04 on the cloud server on Saturday at around 1:00 PM after being let into the server room by Jared Edens. But SSH on the server is broke now; I didn't realize this until Jared had left. Since I can't log into the server remotely, I'll have to wait until Monday so the CS support staff can let me into the server room again and I can fix the issue in person.
Also, I fixed a problem with the UI server, apparently this is a bug with Ubuntu Server and can occur on any of the 3 machines we have it installed on. Basically, if you have a static IP address set up on an Ubuntu Server machine and you restart the server, the DNS hosts specified in the /etc/network/interfaces aren't written to the /etc/resolv.conf file. So, when the server tries to turn a domain name (e.g. cloudoa.cs.siue.edu or www.google.com ) into an IP address, this will fail with a name resolution error. Note, when this happens, we will still be able to connect to the server remotely, the server itself will just have trouble turning hostnames into IPs, so things like updating / installing packages could fail.
A temporary fix to this problem is:
The good news is that we got PHP working on the UI server. We've started working on a login screen. Here are the pages we have so far:
http://cloudui.cs.siue.edu/
http://cloudui.cs.siue.edu/login.php
http://cloudui.cs.siue.edu/phpinfo.php
Last edit: Jeff Hutchins 2012-11-11
There seems to be some kind of issue accessing the UI server. I can't access the website or log in through PuTTY.
I suspect someone has turned the machine off or restarted it.
There is a problem on the UI Server machine; whenever it boots, it will get stuck at a screen that says "Could not find SATA device 1, press any key to continue". If the servers were restarted, chances are it's probably stuck at this screen, waiting for someone to press a key. After the key is pressed, Ubuntu Server will boot like it normally does.
My guess as to why it can't find SATA device 1... it's probably referring to a hard disk that was once connected to the computer via the 1st SATA port on the motherboard. Maybe at one time, the computer had two disks installed in it, but the 1st one was removed? That would mean we installed Ubuntu Server on the disk connected on the 2nd SATA port. The boot order is probably set up to boot from the device at SATA 1 (which I would reason is still enabled in the BIOS, even though it's no longer present) before booting from SATA 2. This is just a theory, however.
November 12, 2012.
Fixed two problems with the UI server.
The BIOS on the UI Server shows that SATA0 and SATA1 were enabled.
The disk we installed Ubuntu Server to is connected to SATA0.
And as expected, SATA1 was enabled, but had no drive connected to it.
I disabled SATA1 and rebooted the machine. It no longer gets stuck at the "Could not find SATA device 1" screen.
This issue has been fixed, but it turns out it wasn't the same issue that caused our server to go offline. The server was still running when I arrived. I ran the following command when I logged on to the machine:
Then I noticed the IP address was different than what it was supposed to be (it should end in 51, but ended in 182 instead).
Turns out, due to a bug in Ubuntu Server, after editing /etc/network/interfaces to use DHCP, restarting the networking, editing /etc/network/interfaces to use a static address, then restarting the networking again, the DHCP client on the machine keeps running.
So, what does this mean? When using DHCP, the machine automatically acquires an IP address from the SIUE network. DHCP addresses have a "lease" on them - in other words, they're only allowed to use these address for a certain amount of time before they expire. When they expire, it's the job of the DHCP client to acquire a new address from the SIUE network.
Because we're using a static IP address, however, the IP address of the machine is never supposed to change (cloud.cs.siue.edu always points to 146.163.150.51). But because the DHCP client kept running, whenever the old DHCP address expired, it replaced our static IP address with a new DHCP address!
I think I've fixed the problem for now. I looked to see if the DHCP server was running:
Then I saw a line that looked something like this:
This means that the DHCP client is running, and it's process ID (the second number) is 2563. To kill the DHCP client, I did:
This should fix the problem in case we encounter it again. (it should happen any time we use the fix for the name resolution problem I described in the two posts above)
November 12, 2012
I also just fixed the problem where we couldn't log into cloud.cs.siue.edu over SSH.
There were two issues:
I've since fixed this and we can log into the cloud server now.
I'll start installing/configuring VSFTPD and setting up users on the machine again.
Last edit: Jeff Hutchins 2012-11-12
Attempted to install OpenStack on the cloud server 11/14/2012.
During the installation we ran into some kind of configuration error. Not clear on what is going wrong here.
Here's what we did
Please advise!
Megha and I discovered what was causing the previous problem above. The problem is the range of IP addresses we specify for the fixed network must contain at least 4 IP addresses. This was not specified anywhere in the documentation - we discovered it after looking in the scripts the errors were occurring in and seeing the assumption first-hand.
We need to reserve a range of IP addresses before spawning VMs. After that Dr. Fujinoki wanted me to set up an "image" with the cloud software set up on it but I don't have any idea how to do this. This page goes into it but reading through it I still have no idea what I am supposed to do. All of this needs to be done within the next week.
I'm sorry but I'm just not going to be able to get the cloud server set up. It's proving to be far more difficult than I anticipated and we're running out of time. I keep encountering significant impediments when trying to work with the cloud:
So, because of all the issues I'm running into, and the limited amount of time we have remaining, I sent an e-mail to Dr. Fujinoki asking for him to settle for alternatives.
In place of the Cloud Server we should probably use the time we have remaining to work on the UI and Owner agent programs. MAYBE we can get the cloud software working next semester.
Here is a copy of the e-mail I sent to Dr. Fujinoki:
Dr. Fujinoki,
I've been trying to get the cloud software set up for the past two to three
weeks, and I can't make any progress with it whatsoever.
We need to present our project's progress to a December 7, 2012, and the way
things are going we will have nothing to show for it.
Nobody in this university, least of all our team, has any background with any of
this. We keep running into so many problems trying to get this working, and this
cloud technology is so new and esoteric, we just can't find any solutions to
these problems online. All of the documentation for OpenStack and guides I can
find online are targeted towards IT experts who do this kind of thing for a
living; this isn't me. I have no background in any of this - I just can't get it
working!
To put it simply: We can't set up a cloud server. We're running out of time and
we need to consider alternatives.
Would you be willing to settle for a single PC running a database and the
Lineage Layer software?
--Jeff H.
SIUE Web Mail
I created a database for the cloud server for the data that will be provided by USTRANSCOM.
I structured the database exactly according to the clients specifications. I have saved and uploaded the SQL queries in the repository for everyone's review. Please look it over and let me know if there is anything that is missing. We may have to see how the tables in the database interact with each other and decide if we need any foreign keys, constraints or triggers.
For this semester we will have to create our own dummy data to populate these tables.
I added two databases to the UI server. One to store the user names along with their passwords and the other for the actual USTRANSCOM data.
They are named as follows:
Sunday, December 2, 2012.
The UI server is inaccessible until Monday. The bug where the UI server couldn't resolve hosts popped up again (someone probably restarted the servers), so I went to fix it like I outlined in posts above, except I made a mistake.
Instead of calling /etc/init.d/networking restart, you're supposed to call ifdown eth0, followed by ifup eth0 (because /etc/init.d/networking restart is deprecated).
So, to shut down the networking I ran:
then I was going to type this to bring it back up:
But running ifdown shut down my SSH connection and made it impossible to reconnect, since I never got a chance to issue the ifup.
I went downstairs to manually turn it back on, but the doors to the room they're stored in were locked, and nobody was available to let us in! What a surprise.
For future reference, my mistake was running ifdown on one line, then trying to run ifup on another line. I believe I should have done this:
So, sorry guys.
Also, I think at the meeting with Fujinoki on Monday we should ask for permission to move the servers to the Senior Project lab, or ask for the keys we need to access the room the machines are stored in or something.
Sunday, December 2, 2012.
Kyle and I got a dummy security gateway and a functional authorization server working.
The authorization server accepts a connection, waits for a username and password to be sent, then queries a database of users on the Owner Agent server. If a username and password is found, a "1" is sent back. Otherwise, a "0" is sent back.
The dummy security gateway is basically just a proxy at this point. You pass it a username and password, and it sends it to the authorization server. Whatever the authorization server responds with is forwarded back to the client.
Anyway, when the UI server comes back online, we can actually handle logins by authenticating via the security gateway, as was originally intended. Of course, if this proves too problematic in the coming week, we can always fall back on the list of users on the UI server.
Monday, December 3, 2012.
The UI server is back up now.
Tuesday, December 4, 2012.
Made some modifications to the login.php on the Cloud UI server.
I rewrote it to contact the security gateway and authenticate the UserID and password given to the script. Tested and it works!
To add more user logins, use mysql to add them to the users table in the userList database on the owner agent.
I'm going to commit our html directory's contents to the SVN.
I'm also going to commit Kyle's security gateway / authentication server code & the compile shell script we've been using.