Hi all,
As part of my day job I've been running a team playing with some Micro Services stuff. As part of the work I wanted a side project to experiment with some things which I got carried away with.
I have taken the core JAR from the Mars-Sim project and use Spring Boot and other projects to create a web service API onto the simulation engine. The simulation runs in the background and the updates can be seen via some endpoints as JSON responses. This simple service has Settlements and Persons available plus a basic information on the Simulation. It can also be paused and resumed.
The code is in BitBucket as a complet Maven project that can be run.
It contains an embedded Swagger UI HTML interface that provides an easy to use interface.
Yes quite a bit can easily be achieved without much coding.
I think completing the end points for Person & Settlement would be good step. So something like /settlements/{$id}/buildings or /persons/{$id}/health. These entities have quite deep object graphs.
The ideal next step would be for someone to create a web ui using a modern JavaScript frameworks like Angular or Ember. This would allow a songle simulation instance to be accessed by multiple people across the internet.
Based on that people could have accounts and be put in control of certain Units. The key is the simulation engine is very powerful and accurate, way better than some of the simulation apps you find on iTunes. So maybe it could be turned into an online presence that people can login in and take part.
One problem i see is how everything is stored. A better way to store the state is needed than using object serialisation. But if there is a RESTful API available the backend elements can be changed over time.
One mad idea I had was to create a Twitter account and get the engine to generate tweets when something interesting happen. Spring has a Twitter integration project and I have used this before to autogenerate posts.
Barry
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
his would allow a songle simulation instance to be accessed by multiple people across the internet... Based on that people could have accounts and be put in control of certain Units.
I can imagine that this web version of mars-sim opens up the possibility of (1) co-op feature and (2) persistent world.
So I have no doubt to believe that this new developmental path will take mars-sim to a new height !
Last year, I've wanted to give mars-sim multiplayer option but my approach so far limits it to running within a home network and is not robust.
If we ever set up a new story mode and code an role/rank assignment system that let a user play as a settler that has a rank/role out of a command structure or a hierachy, each user could log in as the 1st person mode instead of being in the typical 3rd person "God"-mode in a lot of strategy games.
I imagine each user will have unique advantage to perform different tasks. We can even make a user's tasks dependent upon the assistance received from other users or NPCs.
So let's spell out more right here or in the developer/user mailing list the things what we need to do exactly to get there.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Let me have a go over the next few days to flesh out the person and settlement end points. But we would need some web ui to be developed that utilised the web service. This should ge something like React or Angular coupled with a responsive design like Bootstrap so it could be usable from mobiles as well as laptops. Whilst i have played with Angular i am just a novice.
Or a mobile native app.
I still think there could be a problem with the persistency aspect of the simulation state. Serialization is fine for single user but muliuser needs something more robust. Plus thinking ahead it should be possible to run the simulator engine in a seperate VM from the web service, allowing the latter to scale with multiple instances.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Serialization is fine for single user but muliuser needs something more robust.
Yes serialization will have to be reworked.
For one, we'll need to lower the network bandwidth. In the last build r3608, I just switched from using GZIP to LZMA2 to compress the save file. Under gzip, a new sim costs ~620kB. With LZMA2, it's down to ~370kB now. However, it's still unrealistic to sync, 370kB world data between, say 10 users, once a few frames.
We've been using Java's ObjectInputStream and ObjectOutputStream which is binary. If we swtich to JAXB, will it solve the problem ?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If the application uses a web based interface then there is no need to sync the large save file between the users. Users would just be downloding JSON responses into their browser.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think the aspects of moving MSP from client-side Java to a web app are very very interesting. I see CS Java going the same way as Flash, Sliverlight etc. and I'd be happy to not require people to use any other tool than their HTML5-capable browser.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
if we're going with server side java, how do we pay for the traffic/usage ? there's no telling how much traffic it will be and is there a free open source hosting option rather than AWS ?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
In terms of the potential of utilizing HTML/CSS UI,
we could make use of a lot of those nice UI templates out there with free animations such as this CSS weather animation for mars-sim android app: http://codepen.io/davidkpiano/full/ByNPQw/
I also found this nice short and sweet solar system animation written in pure css. Can be adapted to mars-sim for web: http://codepen.io/kowlor/pen/ZYYQoy
I'm nearly giving up building new UI using JavaFX--in which we have to build UI from the ground up-- and go vote for HTML/javascript/CSS--where a lot of good-looking templates are readily accessible that provide better UI/UX.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The first important aspect is how to achieve the change to a client server approach. Creating the client running inthe browser is a significant undertaking. What technology to use? Ruby on Rails, JavaScript framework?
The current engine, whilst massively impressive in its detail and modelling, needs to be able to run for days without problem.
Hosting the back end service is a simpler activity. Either a IaaS like AWS is easy but something like Heroku or CloudFoundry is a great match for Java Spring Boot. The cost of running these is not excessive to get something running. If it takes off with many users then thats a great problem to have; maybe Google Ads or getting people to donate.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
See https://vaadin.com/license :
Vaadin Framework is distributed for free as Open Source, and it is licensed under permissive Apache License, version 2.0. This means for example that you may freely download and use Vaadin Framework for commercial, personal and non-commercial purposes. You may also modify the framework, use it as part of your own software and distribute it.
From https://vaadin.com/faq
"The browser side rendering engine of Vaadin is implemented using GWT. At runtime there are no GWT dependencies in the server side, and the browser side is pure Javascript with no dependencies to Google. The javascript code shipping with Vaadin is precompiled, and no GWT compilation is required when using them. "
I'm no web UI expert, my day job is about creating large enterprise solutions using cooperating services or micro services as they are called these days. I think its important to have a stable and well documented service that any client could use. So i feel if we do want a client server answer the client side should use a technology that allows rapid development in that sphere and independent of the back end.
However my personal opinion is a JavaScript framework like Ember or React is the best choice; reuse what is already out there. These are widely used and certianly on the rise it terms of popularity. Previously I have played with AngularJS which I really like but the new version appears very different and has a lot of doubters out there. Apache Condova also looks impressive. But i dont have the knowledge to create such a web UI from scratch currently.
I think what i am saying it would be good to know someone who is an expert in this field and get them to write it :-)
I intent to develop the back end service further over the next few weeks and see how that works out delivering the complex entities like Settlements over JSON?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
a JavaScript framework like Ember or React is the best choice; reuse what is already out there. These are widely used and certianly on the rise it terms of popularity.
Sure. it's better we take advantage of a javascript framework.
Can we refine our ideas further and lay out a list of objectives as we are going to fork our development to create a mars-sim version for web ?
Also, will it simply be carrying the multiplayer feature (in addition to the single player/standalone) OR becoming a Massively Multiplayer Online (MMO) type of game/simulation ?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have pushed a later version of the web service code to my BitBucket repo over the weekend. it covers some more of the Settlement endpoint which is probably the largest as it has the largest object graph.
I plan to add the container concept so it can show what is stored in a Unit.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I can see you created 3 folders, namely, \controllers, \mappers and \model in \src\main\java\org\mars_sim\restws and with a bunch of new java classes in them.
It'll take a while for me to comprehend how everything works and how you implement REST in mars-sim.
btw, I've seen how people build a backend server app in node.js (or if just a very simple web page they use sail.js). Then they use angular to build the front end Web UI.
So, in our case, how easy/difficult is it for us to build a server app (aka mars-sim server) that can be run in any PC?
This way, any users can deploy their own mars-sim on their own network cuz we should probably first make mars-sim multiplayer capable.
As it stands right now, even after we've converted everything in mars-sim-core to a web service, It still lack multiplayer-specific features.
Besides having to buid a Web UI from the ground up, we need to define different levels of interactions. Say, if multiple users log in, how do we assign what they can and cannot do to make the co-op experience to execute in a smooth way?
So, I'm thinking we should first lay out a list of multiplayer features we want to see and implement in the next few releases.
After we test out these new multiplayer features for a year or two, mars-sim will be matured enough to move to the next level of a becoming MMO.
By then we would have laid out a full plan for marketing mars-sim and get support to be funded and host it in AWS.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I see the eventual solution should have 3 repos:
1. existing mars sim code base for the core simulator and maybe a desktop version
2. the REST WS code base that pligs on to the core
3. a seperate web client using just the published interface
Keeping the client seperate allows multiple clients to be developed
The readme in the BitBucket repo explains how to use the maven spring boot plugin to run the server locally.
How do you want to capture the potential multi user features? A page on the Wiki in Source Forge. Or use something like Trello to create a board to manage some cards.
I have now added some items and resources end points on Persons and Settlements.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'll fresh out some multiplayer features I have in mind in a wiki page.
Yes I've also just set up a team called mars-sim in trello. see https://trello.com/marssim. Are you still in mars-sim developer's mailing list? Let me know your name in trello and/or email address so that I can add you as a member and we can add more cards.
Meanwhile, I'll add that maven spring boot plugin into mars-sim's pom.xml.
Do you have the latest development environment for 3.8-snapshot set up up running in eclipse ?
We're coming very close to wrapping up everything for the final release. May be we can start working off the 3.8-snapshot source code, rather than 3.7.
We could create two additional maven modules such as called mars-sim-server and mars-sim-webui. If we can keep everything under one single repo, it'll be nice. Having one repo rather than multiple repos simplifies how to sync up the new changes.
Can you DEMONSTRATE THE EXACT STEPS it would take for me to integerate spring tool into mars-sim so that I may load it up as a server app on my PC (not in AWS) ?
If we are fast, we could quickly build a web ui to run in a broswer with just a few elements/things to look up. It will be like creating an "observer mode" to just observe how things are going in a running settlement.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
My account on Trello is @barryevans4. I forgot that SourceForge uses SVN; haven't used it for years. I only use Git these days. Can you my sourceforge account back to the access list.
I will add some ore notes to the readme at weekend when I have some time. But in short:
1. clone the Git repo from BitBucket
2. run mvn spring-boot:run
3. this will build everything needed and start the server on port 8080
4. go to http://localhost:8080/swagger-ui.html and you get that to access the WS endpoints or go directly.
So you are thinking to add new top level folders for a service and web-UI
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Can you put together quickly a working app (say with just a couple of java class) and run on spring and deploy it locally and run in with just a few basic web ui controls? This way, I can observe how different pieces works together.
I've sent you some mails on your SF account under your name. Do you by any chance receive them?
What's your github username? I'd like to add you onto our site.
Manny
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Can you put together quickly a working app (say with just a couple of java class) and run on spring and deploy it locally and run in with just a few basic web ui controls? This way, I can observe how different pieces works together.
I've sent you some mails on your SF account under your name. Do you by any chance receive them?
What's your github username? I'd like to add you onto our site.
Hi all,
As part of my day job I've been running a team playing with some Micro Services stuff. As part of the work I wanted a side project to experiment with some things which I got carried away with.
I have taken the core JAR from the Mars-Sim project and use Spring Boot and other projects to create a web service API onto the simulation engine. The simulation runs in the background and the updates can be seen via some endpoints as JSON responses. This simple service has Settlements and Persons available plus a basic information on the Simulation. It can also be paused and resumed.
The code is in BitBucket as a complet Maven project that can be run.
It contains an embedded Swagger UI HTML interface that provides an easy to use interface.
https://bitbucket.org/bevans2000/mars-sim-service
I've also deployed a working version onto AWS EC2 to see how long the simulation runs. It is running on a t2.micro so only has very limited resources.
http://ec2-52-49-180-86.eu-west-1.compute.amazonaws.com:8080/swagger-ui.html
Tell me what you think.
Barry
Hi Barry,
It's interesting work that you just make it possible for mars-sim to run as a web service !
Do you have more examples/experiments that we can set up running ?
What do you think we can play with next?
-Manny
Yes quite a bit can easily be achieved without much coding.
I think completing the end points for Person & Settlement would be good step. So something like /settlements/{$id}/buildings or /persons/{$id}/health. These entities have quite deep object graphs.
The ideal next step would be for someone to create a web ui using a modern JavaScript frameworks like Angular or Ember. This would allow a songle simulation instance to be accessed by multiple people across the internet.
Based on that people could have accounts and be put in control of certain Units. The key is the simulation engine is very powerful and accurate, way better than some of the simulation apps you find on iTunes. So maybe it could be turned into an online presence that people can login in and take part.
One problem i see is how everything is stored. A better way to store the state is needed than using object serialisation. But if there is a RESTful API available the backend elements can be changed over time.
One mad idea I had was to create a Twitter account and get the engine to generate tweets when something interesting happen. Spring has a Twitter integration project and I have used this before to autogenerate posts.
Barry
I can imagine that this web version of mars-sim opens up the possibility of (1) co-op feature and (2) persistent world.
So I have no doubt to believe that this new developmental path will take mars-sim to a new height !
Last year, I've wanted to give mars-sim multiplayer option but my approach so far limits it to running within a home network and is not robust.
If we ever set up a new story mode and code an role/rank assignment system that let a user play as a settler that has a rank/role out of a command structure or a hierachy, each user could log in as the 1st person mode instead of being in the typical 3rd person "God"-mode in a lot of strategy games.
I imagine each user will have unique advantage to perform different tasks. We can even make a user's tasks dependent upon the assistance received from other users or NPCs.
So let's spell out more right here or in the developer/user mailing list the things what we need to do exactly to get there.
Let me have a go over the next few days to flesh out the person and settlement end points. But we would need some web ui to be developed that utilised the web service. This should ge something like React or Angular coupled with a responsive design like Bootstrap so it could be usable from mobiles as well as laptops. Whilst i have played with Angular i am just a novice.
Or a mobile native app.
I still think there could be a problem with the persistency aspect of the simulation state. Serialization is fine for single user but muliuser needs something more robust. Plus thinking ahead it should be possible to run the simulator engine in a seperate VM from the web service, allowing the latter to scale with multiple instances.
Yes serialization will have to be reworked.
For one, we'll need to lower the network bandwidth. In the last build r3608, I just switched from using GZIP to LZMA2 to compress the save file. Under gzip, a new sim costs ~620kB. With LZMA2, it's down to ~370kB now. However, it's still unrealistic to sync, 370kB world data between, say 10 users, once a few frames.
We've been using Java's ObjectInputStream and ObjectOutputStream which is binary. If we swtich to JAXB, will it solve the problem ?
If the application uses a web based interface then there is no need to sync the large save file between the users. Users would just be downloding JSON responses into their browser.
I think the aspects of moving MSP from client-side Java to a web app are very very interesting. I see CS Java going the same way as Flash, Sliverlight etc. and I'd be happy to not require people to use any other tool than their HTML5-capable browser.
if we're going with server side java, how do we pay for the traffic/usage ? there's no telling how much traffic it will be and is there a free open source hosting option rather than AWS ?
In terms of the potential of utilizing HTML/CSS UI,
we could make use of a lot of those nice UI templates out there with free animations such as this CSS weather animation for mars-sim android app:
http://codepen.io/davidkpiano/full/ByNPQw/
I also found this nice short and sweet solar system animation written in pure css. Can be adapted to mars-sim for web:
http://codepen.io/kowlor/pen/ZYYQoy
A lot more on dribbble:
https://dribbble.com/search?q=ux+animation
I'm nearly giving up building new UI using JavaFX--in which we have to build UI from the ground up-- and go vote for HTML/javascript/CSS--where a lot of good-looking templates are readily accessible that provide better UI/UX.
The first important aspect is how to achieve the change to a client server approach. Creating the client running inthe browser is a significant undertaking. What technology to use? Ruby on Rails, JavaScript framework?
The current engine, whilst massively impressive in its detail and modelling, needs to be able to run for days without problem.
Hosting the back end service is a simpler activity. Either a IaaS like AWS is easy but something like Heroku or CloudFoundry is a great match for Java Spring Boot. The cost of running these is not excessive to get something running. If it takes off with many users then thats a great problem to have; maybe Google Ads or getting people to donate.
Hi Barry,
Will you consider using a framework to bridge between JVM and Web UI ?
What do you think about these two frameworks, namely, Vaadin and Dukescript ?
Using either one of them allows us to keep our Java engine for the backend while giving us a new lease of life in our UI design.
Comparison between them
https://www.javacodegeeks.com/2015/08/hello-world-in-vaadin-dukescript.html
To me, I want to vote for Dukescript since I can readily use javascript to build UI and thus having easy access to all the goodies out there.
https://vaadin.com/features
See https://vaadin.com/license :
Vaadin Framework is distributed for free as Open Source, and it is licensed under permissive Apache License, version 2.0. This means for example that you may freely download and use Vaadin Framework for commercial, personal and non-commercial purposes. You may also modify the framework, use it as part of your own software and distribute it.
Comparison between Vaadin with others:
https://vaadin.com/comparison
From https://vaadin.com/faq
"The browser side rendering engine of Vaadin is implemented using GWT. At runtime there are no GWT dependencies in the server side, and the browser side is pure Javascript with no dependencies to Google. The javascript code shipping with Vaadin is precompiled, and no GWT compilation is required when using them. "
https://dukescript.com/
http://www.i-programmer.info/news/80-java/7828-dukescript-java-everywhere-again.html
https://www.javacodegeeks.com/2015/08/java-everywhere-write-once-run-anywhere-with-dukescript.html
"Currently, DukeScript supports the various desktop platforms, as well as iOS, Android, and the browser. The available Maven archetypes create, for each supported platform, a separate subproject."
I'm no web UI expert, my day job is about creating large enterprise solutions using cooperating services or micro services as they are called these days. I think its important to have a stable and well documented service that any client could use. So i feel if we do want a client server answer the client side should use a technology that allows rapid development in that sphere and independent of the back end.
However my personal opinion is a JavaScript framework like Ember or React is the best choice; reuse what is already out there. These are widely used and certianly on the rise it terms of popularity. Previously I have played with AngularJS which I really like but the new version appears very different and has a lot of doubters out there. Apache Condova also looks impressive. But i dont have the knowledge to create such a web UI from scratch currently.
I think what i am saying it would be good to know someone who is an expert in this field and get them to write it :-)
I intent to develop the back end service further over the next few weeks and see how that works out delivering the complex entities like Settlements over JSON?
Sure. it's better we take advantage of a javascript framework.
Can we refine our ideas further and lay out a list of objectives as we are going to fork our development to create a mars-sim version for web ?
Also, will it simply be carrying the multiplayer feature (in addition to the single player/standalone) OR becoming a Massively Multiplayer Online (MMO) type of game/simulation ?
I have pushed a later version of the web service code to my BitBucket repo over the weekend. it covers some more of the Settlement endpoint which is probably the largest as it has the largest object graph.
I plan to add the container concept so it can show what is stored in a Unit.
I can see you created 3 folders, namely, \controllers, \mappers and \model in \src\main\java\org\mars_sim\restws and with a bunch of new java classes in them.
I'm reading tutorials such as http://rest.elkstein.org/2008/02/using-rest-in-java.html
It'll take a while for me to comprehend how everything works and how you implement REST in mars-sim.
btw, I've seen how people build a backend server app in node.js (or if just a very simple web page they use sail.js). Then they use angular to build the front end Web UI.
So, in our case, how easy/difficult is it for us to build a server app (aka mars-sim server) that can be run in any PC?
This way, any users can deploy their own mars-sim on their own network cuz we should probably first make mars-sim multiplayer capable.
As it stands right now, even after we've converted everything in mars-sim-core to a web service, It still lack multiplayer-specific features.
Besides having to buid a Web UI from the ground up, we need to define different levels of interactions. Say, if multiple users log in, how do we assign what they can and cannot do to make the co-op experience to execute in a smooth way?
So, I'm thinking we should first lay out a list of multiplayer features we want to see and implement in the next few releases.
After we test out these new multiplayer features for a year or two, mars-sim will be matured enough to move to the next level of a becoming MMO.
By then we would have laid out a full plan for marketing mars-sim and get support to be funded and host it in AWS.
I see the eventual solution should have 3 repos:
1. existing mars sim code base for the core simulator and maybe a desktop version
2. the REST WS code base that pligs on to the core
3. a seperate web client using just the published interface
Keeping the client seperate allows multiple clients to be developed
The readme in the BitBucket repo explains how to use the maven spring boot plugin to run the server locally.
How do you want to capture the potential multi user features? A page on the Wiki in Source Forge. Or use something like Trello to create a board to manage some cards.
I have now added some items and resources end points on Persons and Settlements.
I'll fresh out some multiplayer features I have in mind in a wiki page.
Yes I've also just set up a team called mars-sim in trello. see https://trello.com/marssim. Are you still in mars-sim developer's mailing list? Let me know your name in trello and/or email address so that I can add you as a member and we can add more cards.
Meanwhile, I'll add that maven spring boot plugin into mars-sim's pom.xml.
Based on http://docs.spring.io/spring-boot/docs/1.3.3.RELEASE/maven-plugin/usage.html , it will automatically repackages the jar that is built during the package phase of the Maven lifecycle for spring boot.
Do you have the latest development environment for 3.8-snapshot set up up running in eclipse ?
We're coming very close to wrapping up everything for the final release. May be we can start working off the 3.8-snapshot source code, rather than 3.7.
We could create two additional maven modules such as called mars-sim-server and mars-sim-webui. If we can keep everything under one single repo, it'll be nice. Having one repo rather than multiple repos simplifies how to sync up the new changes.
Can you DEMONSTRATE THE EXACT STEPS it would take for me to integerate spring tool into mars-sim so that I may load it up as a server app on my PC (not in AWS) ?
I've read your readme at https://bitbucket.org/bevans2000/mars-sim-service a few times but I don't know how to connect all the pieces together yet. Any links and/or examples would appreciate !
If we are fast, we could quickly build a web ui to run in a broswer with just a few elements/things to look up. It will be like creating an "observer mode" to just observe how things are going in a running settlement.
My account on Trello is @barryevans4. I forgot that SourceForge uses SVN; haven't used it for years. I only use Git these days. Can you my sourceforge account back to the access list.
I will add some ore notes to the readme at weekend when I have some time. But in short:
1. clone the Git repo from BitBucket
2. run mvn spring-boot:run
3. this will build everything needed and start the server on port 8080
4. go to http://localhost:8080/swagger-ui.html and you get that to access the WS endpoints or go directly.
So you are thinking to add new top level folders for a service and web-UI
Hi Barry,
Can you put together quickly a working app (say with just a couple of java class) and run on spring and deploy it locally and run in with just a few basic web ui controls? This way, I can observe how different pieces works together.
I've sent you some mails on your SF account under your name. Do you by any chance receive them?
What's your github username? I'd like to add you onto our site.
Manny
Yes I got the emails and sent you a reply. I'm not a lover of SourceForge and would much prefer to use GitHub.
In the dev mailing list yesterday I said I would try and get something out together at weekend.
Barry
Sent from my iPad
Can you resend ? sorry I don't see your reply. it must have been deleted accidentally.
We have a github site at https://github.com/mars-sim for backup but SF is the main site.
I can add you to our core team if you give me your github name in email.
We're sticking to svn for now since git is not that perfect and arguably svn is more dependable and work well for us.
SF is not bad at all. It just needs modernization of its UI and be more transparent.
On the other hand, github has over 1M+ issues to fix:
https://news.ycombinator.com/item?id=11578191
Last edit: manny kung 2016-04-27