#3 Scalable Solution for Student Progress Feed

[Erin Twamley]

Estimate is 160 hours and 2-3 weeks of full-time effort. This is not a bug, but a long-term solution may need to be developed.

Some options include:

Researching different PHP XML generators or XML generation techniques. There are a number of ways to create an XML document, and we may not be using the most efficient tools or techniques.
Researching writing to the socket immediately after getting the request, instead of after the full XML feed has been generated. This might effectively nullify the hub time-out issue we believe we are seeing.
Possible Apache or MySQL optimizations. We've already spent some time on this, so we have already handled the low hanging fruit. But it's possible there are other optimizations we can consider.

If we want to look at ways of creating smaller feeds that are easier for people (NTER and the mobile app most likely) then we have a different set of options:

Create an archived feed that is able to support paging. This is defined in RFC 5005. This change will require updates to both NWTP and NTER to handle the new paging links.
    Positives: Creates a smaller feed for NTER to process. Should be natively supported by PuSH hubs, but this will have to be tested.
    Negatives: Creates a much more complicated feed for NWTP to create and possible means we won't be able to use our caching table. So far, everything points to NWTP taking too long to create a feed as the root problem, so creating a more complicated feed will probably cause more problems.
Create a per-student feed. This feed would only contain the record for a give student. This change will be limited to NWTP, but will in reality, probably only be used by the MobileSync project.
    Positives: Creates a much smaller feed for mobile devices
    Negatives: Requires an update to NWTP's feed generation. However, this added complexity will not be as time constrained as the regular feed since it does not involve a PuSH server.
Move the entire feed generation out of NWTP and into a separate, stand-alone web app on a dedicated server.
    Positives: If we isolate feed generation, then slow performance won't affect students.
    Negatives: A lot. This option should be considered as a last resort. It would force us to create a new, stand-alone app from the ground up, that to truly not affect student performance, will most likely have to be running on a separate server. This will drastically increase maintenance and configuration requirements. I only mention it here to point out how hard it would be and to recommend avoiding it.
Deploy a larger, more scalable NWTP installation.
    Positives: This has already been vetted on other projects, so we know that it improves performance of the entire system. The performance of feed generation included. Also, we wouldn't have to do anything on our end. The code and installation guides are already written for a load-balanced, multi-server architecture.
    Negatives: Cost to the customer. Moving away from a single, medium sized server to a more robust infrastructure will be more expensive.

[Erin Twamley]

We are documenting pushsubhub server and progress, course feed function currently.