SDCFramework
The SDCF Service Application
The SDCF service application package is composed of the service itself and a related control activity.
The Service
The SDCF service must be started explicitly and runs sticky in the background until it is stopped. Other applications can bind to the service to receive the collected samples via broadcasts.
Sensor Data Collection
A core feature of the service is the ability to collect device specific data from various sensors.
Sensor Categories
Beside real hardware sensors, like wifi, bluetooth, gsm, gps, accelerometer or proximity, some virtual sensor types are known. The audio, tag or twitter sensors have been added as abstract concepts, to collect and transport high level data through the service. Those virtual sensor types are passive data sources receiving the specific sensor data through a corresponding data sink (realised by content providers). To transport tags, tweets or recorded audio files through the service, it's necessary to write an Android application which does generate the particular sensor data and stores it in the data sink.
[ln_scan_behaviour]:
Another distinction between sensor types is the scan behaviour. Active scanned sensor types are polled in a constant frequency for new samples. Some devices, like all virtual sensor devices, the light and the proximity sensor, will rarely change it's state. An active polling in a constant frequency would just increase the system load without further information gain. Thus the related passive device scanners do take samples in the event of sensor information changes signalled by the observed device.
Known Sensor Types & Configuration
For each known sensor type there are three parameters to configure:
- an enabled flag, to activate/deactivate data collection for the sensor.
- a priority level, which does determine the importance of the collected sensor information. Per definition, sensor samples with a higher priority will be preferred for transmission.
- a scan frequency (in milliseconds) for the active scanned devices. (For passive observed devices, this parameter is available as well but ignored at runtime).
| Sensor Type | Active Scanned | Virtual Type | Configuration File ID |
|---|---|---|---|
| Magnetic Field | Yes | No | MagneticField |
| Proximity | No | No | Proximity |
| Light | No | No | Light |
| Accelerometer | Yes | No | Accelerometer |
| Bluetooth | Yes | No | Bluetooth |
| Gsm | Yes | No | GSM |
| Gps | Yes | No | GPS |
| Network Location | Yes | No | NetworkLocation |
| Wifi | Yes | No | Wifi |
| Orientation | Yes | No | Orientation |
| Audio | No | Yes | Audio |
| Tags | No | Yes | Tags |
| No | Yes | ||
| Time Sync State | No | Yes | TimeSyncStateChanges |
Priorities of Samples
The known five priority levels are numbered top down: From Level0 to Level4. This means the lower the level number, the higher is the priority.
Collected Data
Common Data
Independent of the device specific sensor information, sensor samples have some data in common:
- an unique identifier for the sensor type,
- a time stamp (UTC),
- a priority (resp. the configured sensor priority at the time when the sample is generated),
- Optional (can be configured): the last known network location (longitude & latitude) where the sample was taken.
- Since version 1.3.2: a flag for the NTP synchronisation state of the internal time provider at the moment the time stamp was taken.
Sensor Specific Data
The following table will list the collected sensor information per device type:
| Sensor Type | Device specific data |
|---|---|
| Magnetic Field | ambient magnetic fields in the x, y and z axis (in micro-Tesla) |
| Proximity | proximity sensor distance (in centimeters), resp. the near and far state values for binary sensors. |
| Ligh | ambient light level (in SI lux unit) |
| Accelerometer | accelerations in x, y and z direction (in m/s^2) |
| Bluetooth | received signal strength indicator (rssi), hardware address and bluetooth class and name |
| Gsm | operator, cell id, location area code, signal strength and neighbor cell information (cell id & signal strength), if provided |
| Gps | longitude, latitude, altitude, speed and accuracy |
| Network Location | longitude, latitude and accuracy |
| Wifi | ssid, bsssid, capabilities, frequency (in MHz), signal level (in dBm) and a flag whether the device is right now connected to this network or not |
| Orientation | heading (ambient angle between magnetic north and the y-axis), pitch (rotation around x-axis) and roll (rotation around y-axis) |
| Audio | the file name of the recording, which will be packed in the archive together with the serialized sensor data |
| Tags | a series of tags as string |
| the twitter message as string | |
| Time Sync State | a synced to NTP time flag indicating the new state after a change event |
Sensor Availability State Awareness
For each supported sensor device type the overall availability is detected during service startup. Only available types will be supported at runtime.
If available devices are temporarily unaccessable, e.g. in flight mode, the service can recognize the state changes and dynamically stop or restart the related device scanner to preserve system resources.
Local Intermediate Storage
Collected sensor samples are often provided in a high frequency, they will be cached in memory first and then stored in a local database asynchronously.
Database Size Limit Exceedance
As there is no way to guarantee a continuous transmission of stored samples (e.g. in case of connectivity problems), it is possible, that the rate of new collected samples is higher as the delivery rate of the transfer service. In that case the database will grow until the size limit is reached, and a strategy is needed, to respond to the databse full event.
Actually two strategies are supported:
(1) WAIT_DELETE_NOTIFY (Deletion of older samples, preferred those with a lower priority, if configured).
(2) WAIT_NOTIFY_STOPSERVICE (Automatic service shutdown).
Both strategies do first wait for a configurable time span, just for the case the transmission problem is of short duration. User notifications will be sent to the notification area in either case.
Available Configuration Options
General options:
- maximum database size (in KB).
- database full strategy selection.
- inital time to wait in case of database size limit exceedence.
Parameters for the deletion strategy:
- the count of records to delete.
- flag for the preference of samples with a lower priorities.
Periodical Data Transfer to a Remote Host
Sample Selection
The transfer happens periodically for a configurable amount of samples, which are selected by age and priority from the database. By definition the samples with the oldest time stamp and highest priority will be removed first from the database.
Archive Creation
To reduce the amount of data for transmission, the samples are serialized to XML (1) and compressed into an archive (zip or jar). Optionally the archive can be encrypted with RSA. It is temporarily stored in the local file system for a delayed transmission. In case of transmission errors or a lost connection the archive will be processed in the next execution cycle. Thus its guaranteed, that samples can not be lost in the transfer step.
(1) Serialisation is realised with the Simple library.
Identification of the Origin of a Transmitted Archive
Together with the data file some device specific information is transmitted with each archive. A generated unique device identifier allows the association of the samples with a specifc device. It is displayed in a sub screen of the service preferences.
 |
Unique Device IdentiferOn its first start, the service generates a UUID as unique device indentifier. This identifier is unique per device and installation only. More precisely, if you uninstall the service and reinstall it later, the generated new identifier will be different from the previous one. Additional Device InformationBeside the generated UUID some further information about the concrete device is transmitted, e.g. the manufacturer and the device name. Figure 1 shows the device information screen, displaying all the device specific data transferred with each archive. |
Archive Content
Each archive contains:
- devinfo.xml (the serialized device information)
- samples.xml (the serialized samples)
- optional: binary audio files received through the content provider of the audio sensor.
Configurable Connection Strategy
Mobile connectivity may cause additional cost, thus a connection strategy can be selected to configure the allowed network types for internet connections:
- any_available,
- wlan (uses available wlan connections only),
- mobile_connection (uses mobile connections only, e.g. GPRS or UMTS),
- wlan_else_mobile (prefers available wlan connections over mobile ones).
Transfer Protocol
The protocol is extracted from the configured URL. An invalid or unsupported scheme in the URL will raise an error. For the moment, just http is implemented.
Transfer Settings
The following settings are available to configure the transfer module:
- host URL,
- optional: username and password for an http basic authentication (the password is transferred as MD5 hash),
- a lower limit for the transfer interval length in seconds,
- a lower and an upper limit for the count of samples to transfer,
- the archive type (jar or zip),
- the network connection strategy,
- optional, special build required: RSA archive encryption (requires a provided RSA public key file).
At runtime the next optimal execution time for a file transfer is calculated dynamically. It depends on the available samples in database, the current overall sample rate and the configured lower time window length. E.g. for a configured time interval of 900 seconds it is guaranteed, that a sample transfer will not be triggered more often than every quarter of an hour and the count of transmitted samples fullfills the configured limits.
Logging
The framework uses it's own logging mechanism. Any log statement is redirected to the Android Log (for debug purpose), notified to registered observers (e.g. the control activity) and saved to a logfile. Logfile rotation is done on a daily basis. While in version 1.3.1 the service was broadcasting the log information as well, this feature was removed in revision 1.3.2 to reduce the overall system load.
Log level
The log level can be configured in the service settings. Known log levels are:
- ERROR (just errors are logged),
- WARNING (errors and warnings),
- INFO (additional information, errors and warnings),
- DEBUG (like INFO level, but with special debug information).
The default log level is INFO.
Logfile storage
The SDCF Logfiles are stored to the shared external storage in a subdirectory with name sdcfframework.
Logfile transfer
It is possible to setup a remote host as receiver for logfiles. If configured, old logfiles will be transferred at service startup or whenever a log rotation occurs. The files are transferred by http upload (method post). Http basic authentication is supported.
Internal Time Provider
The main purpose of the SDCF is data provision for data mining processes. The comparability of sample time stamps from different devices is a core requirement in this context. The SDCF service has its own internal time provider, which uses one or more configurable NTP servers to provide reliable UTC time stamps. For this reason the service requires a network connection at startup, but also at runtime whenever the device time is updated. If the synchronisation attempt fails at startup, the service will shutdown itself and notify the user.
Up to version 1.3.1 the result of a failed attempt for re-synchronisation at runtime was a service shutdown as well. With version 1.3.2 a time provider out-of-sync strategy was introduced. Any sample is tagged with an additional time sync state flag now, indicating whether the time provider was synchronized to NTP time the moment the sample was taken or not. The user can now select between the old strict shutdown strategy and a new one, allowing the service to continue with sample gathering while out-of-sync. For a later time stamp validation the time provider is observed for synchronisation state changes and acts like a virtual sensor now. State changes are collected and transferred like other sensor samples.
The Control Activity
 |
Service ControlWith the control activity the service can be started, stopped and configured. While it is active and the service is running, it does display the current log information of the service. NotificationWhen the service is started, a permanent notification Service Preferences and Maintenance OptionsThe context menu of the control activity provides access to the service configuration (Preferences). The second option (Clear Log) does clear the displayed log information and deletes all locally stored log files. In case of a configured remote host, the files will be transferred to the log file server before deletion. If the service is not running, it is possible to erase the database (Reset Database). This option is for maintenance purpose and does drop the whole database with all it's content. Since version 1.3.3 the possibility for configuration by external files exists. With version 1.3.4 the option to trigger a file transfer manually was added. |
Configuration by External Files
With SDCF Version 1.3.3 the possibility to reconfigure the service by external files is introduced.
 |
If the user selects the entry Configuration by Files in the option menu (figure 2 ) of the controller activity, he can choose between two actions (figure 3):
|
 |
Load Service Settings From File
Up to version 1.3.2 only a simple default configuration for the service was provided in the projects asset folder, while a concrete configuration, e.g. the remote host settings, is done manually in the service preference dialog. To deploy specific test settings to a large set of devices was a rather laborious task.
With version 1.3.3 it is possible to create an external XML configuration file for the SDCF service and to override the local settings of concrete device installations with it.
Step-by-Step Guidance:
- First Create an external configuration file. Your file must have the extension ".xml"!
- Copy the file to the root of the external storage of your device, e.g. via USB-connection with a PC.
- Assure, that the external storage media is accessable by SDCF (disconnect your device from the PC properly).
- Start the SDCF service controller activity (but not the service itself!).
- Open the context menu and select Configuration by Files. Now select Load settings from File.
- In the file browser activity
- navigate to your configuration file (only files with the extension "xml" will be displayed here) and select it,
- commit your selection by activating the Choose-Button.
- Your service settings should be updated by now. Otherwise you will see an alert dialog, with an error message.
If a loaded external configuration file is corrupt in any way, the service settings will remain unchanged.
Create an External XML Configuration File
The best way to genrate an external XML configuration file for the SDCF service is to modify an existing one.
You can download a sample file with useful hints and comments here!
RSA Public Key File
Up to version 1.3.2 the public key file must be provided in the projects asset folder. Thus a special build of the service is required for RSA encryption support .
With version 1.3.3 the possibility to load the public RSA key from an external file into an existing SDCF service installation is introduced. The corresponding private key for decryption must be provided to the configured remote host.
Step-by-Step Guidance:
- First Generate your RSA keys.
- Copy the public key file to the root of the external storage of your device, e.g. via USB-connection with a PC.
- Assure, that the external storage media is accessable by SDCF (disconnect your device from the PC properly).
- Start the SDCF service controller activity (but not the service itself!).
- Open the context menu and select Configuration by Files. Now select Load RSA public key.
- In the file browser activity navigate to your public key file, select it, and commit your selection by activating the Choose-Button.
Attention: There's no validation for an uploaded public key file. If you activate RSA encryption in the file transfer settings of the service, the key will be used for archive encryption with the next file transfer. Best way to verify if it is working: Increase the log level to "DEBUG" and check the log information for possible errors.
RSA Key File Generation
To generate a 2048 bit RSA key pair for archive encryption wirh SDCF, you can use the provided java program.
Usage: Unzip the archive in a local folder and follow the instructions in the text file.
Weekly runtime schedule
Since version 1.3.4 a runtime scheduler exists to schedule service start and stops for a weekly pattern.
 |
 |
 |
The service scheduler activity for weekly pattern can be accessed from the preference menu in the controller activity.
-
To add a new entry, press the plus symbol in the lower right corner. A new window will open where you can select the day, the kind of activity (service start or stop) and the time for the new schedule entry. Press the little disk symbol to save it.
-
To edit or delete an existing entry in the schedule, select it on the screen and wait a moment for the context menu to open.
-
To save your schedule, press the button with the disk symbol in the lower right corner. It will only be enabled if your schedule is valid. Otherwise check for red entries in the schedule.
-
To delete a whole schedule, open the context menu of the activity and select the "clear schedule" entry.
