I am trying to get my mind around how I might be able to implement a remote software download scheme for my remote field monitoring equipment (to save having to go to site and swap out parts).
My current thinking is to basically break my code into 2 separate parts.
The part in the bottom of memory will contain a minimum set of low level drivers, libraries, etc. Only those required to implement some sort of rudimentary boot code. This boot code will need to be able to check the integrity of the system and communicate with some sort of remote download server (under my control) via TCP/IP. I do not expect this part of the code to change all that frequently. So essentially this will be fixed firmware (only changeable by way of a site visit and direct programming).
The second part of the code will be everything else (application firmware).
The system needs to be able to boot cleanly whether there is any application firmware present or not. This will be achieved with a combination of checksums, signature checks, code size checks, etc.
If the boot code can not start the application code it will ask the download server to provide the latest code and then reboot with default settings.
All operations must be able to be achieved totally without any user intervention!!!!
From a GCB point of view, I think I need to be able to create 2 separate modules of compiled code (Section A & Section B, with Section A being the minimal boot code).
Section A code would be typically loaded into the devices at production time.
Section B code could be loaded at production time also but could just as easily be placed on the download server and only be loaded on request from the device in the field on first power up.
To make life easy, I am thinking about making all the entry points to Section A accessible via some sort of boot table. This boot table will sit at a fixed location (in Section A) so that minor code modifications within Section A can be easily made with absolutely no impact on the separately compiled applications code in Section B. Section B would need some sort of include file to reference the jump table entries for the section A code.
In principle this all sounds fine but I am not sure how GCB would handle the 2 separate compile instances.
It should be relatively easy to force the Section B code to start at a high address (via the boot code features I believe).
I do not know how to handle the issue where a number of the standard GCB library files need to be in the Section A (boot code) area and I do not want them repeated in the Section B area. If I create a special library file for all the calls I need (all in one big include file to point to the jump table in Section A) will the GCB compiler recognise these duplicate entries and not also try to include the standard library code again?????
Essentially I need to turn off the automatic inclusion of some of the referenced standard library files for the Section B compile and include my references to the fixed jump table entries instead.
As an example, the h/w serial port routines and most of the string handling routines will all be needed in Section A. I will also add a jump table at a fixed location in section A to all these library routines. For section B, I need to define a separate include file to redefine all of the calls to the h/w serial routines and the string handling routines so that they go via my jump table. So any call to these library routines made from section B will first call the jump table location which will then jump to the actual real routine in Section A. The return will go straight back to the original caller. If GCB enables library entry points to be redefined at compile time, then I suspect it should all work.
Any thoughts on all of this would be greatly appreciated.
If anyone has a better idea, I am open to suggestions.
Peter.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks.
The complication I face is that the firmware needs to be loaded over a 3G modem using a TCP/IP connection.
There is a bit of code required to set up and utilise the TCP/IP connection which I desire to share between the modules for efficiency.
I already have all the TCP/IP comms s/w coded and working which I just want to relocate into the boot area.
I am sure that I can set this all up with jump tables (just a table of goto instructions in the boot code area).
The problem I need to resolve is how to get GCB to use a different library set (other than the standard one) at compile time.
Is there some compiler switch that enables you to specify where the folder for the main library files live? If so, this would allow me to compile the boot code section with the standard library and then compile my main application code with the modified libraries. The modified libraries would have particular (common) sections just replaced with #define entries. eg a #define for the HSerSend routine that would point to an entry in the jump table in the boot section.
Any further thoughts on how I set up the separate library arrangement?
Peter.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Using a different library set. Yes. Using a define you could redirect all the standard library methods to your library methods. So, you would have the following.
~~~~~
#define Use_My_Libraries
'lots of code
#ifdef Use_My_Libraries
'a list of the all methods you need to adapt
#define HSERPRINT my_HSERPRINT
#define HSERSEND my_HSERSEND
'etc etc
#endif
~~~~
You would need to create all your methods, of course. These would be called only in Use_My_Libraries is defined leaving all the standard methods as is.
But, you may need to adapt the 'lowlevel.dat' file - you may choose to remove all entries and manually add the standard include files in your solution. Why? The initialisation of the standard include files may mess up the redirection to your replacement methods.
Still sounds doable. I use a lot of redirection to develop drivers etc. Work ok. :-)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for the suggestion.
My concern with this however, is that I would have to maintain an independent set of "standard" library routines that would need hand modifying whenever new library updates were released.
Is there some way that the existing overlaying feature could be used? This would technically allow me to define a new entry point for a particular function (that I would include in a special library file) that would simply be used by the compiler in preference to the original routine. Maybe the compiler throws up an error message (to alert unsuspecting users that they have replaced a standard library function) just to be safe. When I want to compile with the original libraries I just delete the #include reference to my special library file.
I think this would be much easier to maintain in the long term.
Failing all this, a simple switch within the compiler so that you can specify where all the library files are located (other than the default locations) would also work reasonably well. This would allow me to simply copy all the library files to a new location and delete the entry points that I want to redefine in a separate library file.
The original library files would remain intact (in their default location) and could be easily updated as changes were released.
Just thinking a bit more about this.
It appears that the library reference used by the compiler is a relative one. include folder is in the same folder as the compiler.
So if I make a totally separate copy of GCB in another area (call this GCB-B or something else), then I could modify the include files for this version of GCB to implement my jump table idea.
I would then simply compile all normal code (including my bootstrap code) under the original GCB folder. I would then switch to the special GCB-B version when I needed to compile the code that I wish to place in the higher memory area with the jump table references to the code in the bootstrap area.
A little messy but it might work and could simplify library maintenance but would mean maintaining 2 separate copies of the compiler etc.
Any further, more rational thoughts???
Peter.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Following this thread, I have to say I'm confused about the goal of your questions. At first, I thought you were trying to built a a run-time auto-updater for the firmware in your remote device. Now however, it seems like you're looking for build-time options to build various versions of the code prior to loading into the unit. Can you clarify?
Joe
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sorry for the confusion.
The primary objective is to end up with a scheme that enables the remote download and update of code within the PIC chip.
From my current understanding, this will require a bootloader installed in the devices at production time that has the capability to communicate with my cloud server via TCP/IP.
It has to pass some basic access security tests and then negotiate with the server to acquire the latest application firmware appropriate for that particular installation.
Since I will be using much of the serial communications code and TCP/IP code (that I have already written) in both the bootloader and application code areas (as well as requiring access to common variables) there is a desire to share as much code as possible.
I need to be able to separate the bootloader code from the main application code at compile time, since I do not intend to update the boot code during a download. The boot code will be in an area of EPROM that is write protected to ensure that a download failure can not cause the system to end up in an unrecoverable state. These field units are sometimes located in remote, inaccessible locations and must be very reliable.
So what I need to be able to do is compile my application code independently of the bootloader code so that I can store this on my server as a download image. There will most likely be multiple versions of the application code for different clients, different product configurations and even different locations. The application code sets need to be able to access subroutines and functions within the bootloader and access common variables (for transferring details etc). The interrupt service routines for the system (currently 5) will also most likely need to be located in the common bootloader code area (as some of these will at least be needed for serial port and RTC support).
I hope that this clarifies what I am trying to achieve.
Do you have any additional thoughts or implementation ideas for my project?
Thanks for your interest.
Peter.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
This is a very ambitious but interesting project for a PIC or AVR, especially under GCBasic. I have more questions:
1) I assume that the download paradigm will be “pull”, meaning the remote microcontroller requests the object file from a network server somewhere out in the internet cloud, not “push” which is the more common method where a hard-wired host forces code into the microcontroller.
2) What chip are you planning to use and can it support flash-writes, a TCP/IP stack as well as your application + loader code?
3) Do you have firewall & security issues to deal with, like blocked ports/protocols (TFTP/FTP) on the remote device’s local network?
4) How do you propose to get your device onto the TCP/IP network...via an external terminal-server module such as an XPort or with a built-in TCP/IP stack?
5) If it’s commercial product you’re developing, consider that many enterprises are now requiring IPV6 addressing for any device attached to their networks.
6) Have you ever developed an application like a bootloader that has to write to program Flash? It’s tedious and unanticipated results can be catastrophic.
There are numerous articles about this topic on the net. I suspect your head will be spinning once you consider all the challenges, but if you get something running on a PIC or AVR, it would be interesting to everyone.
Joe
From: Peter Stone
Sent: Monday, July 06, 2015 9:34 PM
To: [gcbasic:discussion]
Subject: [gcbasic:discussion] Remote firmware download scheme
Hi Joe,
Sorry for the confusion.
The primary objective is to end up with a scheme that enables the remote download and update of code within the PIC chip.
From my current understanding, this will require a bootloader installed in the devices at production time that has the capability to communicate with my cloud server via TCP/IP.
It has to pass some basic access security tests and then negotiate with the server to acquire the latest application firmware appropriate for that particular installation.
Since I will be using much of the serial communications code and TCP/IP code (that I have already written) in both the bootloader and application code areas (as well as requiring access to common variables) there is a desire to share as much code as possible.
I need to be able to separate the bootloader code from the main application code at compile time, since I do not intend to update the boot code during a download. The boot code will be in an area of EPROM that is write protected to ensure that a download failure can not cause the system to end up in an unrecoverable state. These field units are sometimes located in remote, inaccessible locations and must be very reliable.
So what I need to be able to do is compile my application code independently of the bootloader code so that I can store this on my server as a download image. There will most likely be multiple versions of the application code for different clients, different product configurations and even different locations. The application code sets need to be able to access subroutines and functions within the bootloader and access common variables (for transferring details etc). The interrupt service routines for the system (currently 5) will also most likely need to be located in the common bootloader code area (as some of these will at least be needed for serial port and RTC support).
I hope that this clarifies what I am trying to achieve.
Do you have any additional thoughts or implementation ideas for my project?
Joe,
Yes,
I am aware this this going to be a fairly complex task, the outcome of which I am sure will be of considerable interest to others.
Some brief answers to some of the questions asked:
1)I have full control of the server. I have already written custom code that is running under Linux and it communicates with my existing PIC based products over TCP/IP using a custom communications protocol running on dedicated ports.
Here is a link to what the existing remote systems can deliver via a PIC running GCB code. http://dipstik.info/DipStik0001-Demo-System.jpg
Images and overlay information is updated hourly.
The server comms protocol will be modified to negotiate with the PIC based device when it initiates remote connection. Based on information passed from the remote device the server will push the latest version of application code down to the device.
2) I am using the 18F26K22 currently, but the code should work with other chips.
3) I only have limited programs running on my server at the far end and they respond to a very strict comms protocol. The protocol includes some security check measures for the field devices before data transfer can proceed.
4) Network connection is via my 3G modems designed for M2M communications.
5) I currently use IPV4 but the system will support IPv6. The devices currently only connect to my server.
6) I have done numerous bits of coding with embedded systems over the last 30 years, but I have never implemented bootloader code from scratch. As for PIC stuff, I have done EEPROM stuff with GCB and I am intending to base the bootloader code around the EPROM read/write features within GCB. I have also done some work using GCB to write directly to SD cards. Based on what I have achieved so far, this does not seem to be particularly difficult.
I have worked on other multitasking embedded ROM based systems in the past that have had similar inter-module/task operability requirements.
The hard part for me is trying to nut out how to get the GCB compiler to handle code generated across a number of compile sessions so that it all can be uploaded into the PIC micros in the field reliably (even if a download session fails).
Peter.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Take a look at TinyBootLoader for a peeled-back version of the bare essentials that a bootloader has to accomplish. It all fits in 100 bytes of high PIC memory. The most challenging part is writing to the Flash memory, as this has to be done in very well-controlled block sizes and boundaries. Also note that GCB, though it handles EEPROM programming well, doesn’t handle Flash programming very well (or not at all) on most parts, so you’ll be somewhat on your own in this area.
I’m encouraged to hear that you’ll be using a “push” approach. With this, you should be able to ‘burn’ a bootloader at manufacturing time that never gets changed in the field. The operating paradigm is generally that the bootloader code runs whenever the part resets, loads the application code if it’s being sent right after reset, then changes the startup vectors so that the newly loaded application can run. You never have both programs available to run at the same time, as they’re separately compiled programs that don’t share memory resources. If you can find a way for the firmware server to put the remote device in reset, then you can almost ‘borrow’ an existing bootloader as is. Of course, your firmware server will need an application that decides when a device needs an update and pushes the code to it. I’ve done this in the past by having the remote devices periodically ‘broadcast’ their firmware version and current IP address to a server app. The server app decides if the code needs to be updated and pushes new code to it if necessary. This has the added advantage of being able to keep track of each unit’s current IP address if dynamic IP addresses are used, as well as keeping a record of which firmware version is in each remote unit. Of course, this is all based on the ‘push’ paradigm. If you choose to let the remote device ‘pull’ its code, things are very different.
Keep us informed.
Joe
From: Peter Stone
Sent: Tuesday, July 07, 2015 10:30 AM
To: [gcbasic:discussion]
Subject: [gcbasic:discussion] Remote firmware download scheme
Joe,
Yes,
I am aware this this going to be a fairly complex task, the outcome of which I am sure will be of considerable interest to others.
Some brief answers to some of the questions asked:
1)I have full control of the server. I have already written custom code that is running under Linux and it communicates with my existing PIC based products over TCP/IP using a custom communications protocol running on dedicated ports.
Here is a link to what the existing remote systems can deliver via a PIC running GCB code. http://dipstik.info/DipStik0001-Demo-System.jpg
Images and overlay information is updated hourly.
The server comms protocol will be modified to negotiate with the PIC based device when it initiates remote connection. Based on information passed from the remote device the server will push the latest version of application code down to the device.
2) I am using the 18F26K22 currently, but the code should work with other chips.
3) I only have limited programs running on my server at the far end and they respond to a very strict comms protocol. The protocol includes some security check measures for the field devices before data transfer can proceed.
4) Network connection is via my 3G modems designed for M2M communications.
5) I currently use IPV4 but the system will support IPv6. The devices currently only connect to my server.
6) I have done numerous bits of coding with embedded systems over the last 30 years, but I have never implemented bootloader code from scratch. As for PIC stuff, I have done EEPROM stuff with GCB and I am intending to base the bootloader code around the EPROM read/write features within GCB. I have also done some work using GCB to write directly to SD cards. Based on what I have achieved so far, this does not seem to be particularly difficult.
I have worked on other multitasking embedded ROM based systems in the past that have had similar inter-module/task operability requirements.
The hard part for me is trying to nut out how to get the GCB compiler to handle code generated across a number of compile sessions so that it all can be uploaded into the PIC micros in the field reliably (even if a download session fails).
I will give some thought to how I might be able to get a simple solution working using RESET as the means of kicking off the update. On the 18F26K22 the RESET instruction might be able to be used effectively.
So on power up, the bootloader gets initial control. The bootloader needs to first get the version number for the current loaded code (must be at a fixed location). It then needs to connect to the server and ask for the latest applicable update. If there is an update, it loads it. The bootloader performs a check sum check on the main code and boots it if all is ok. If the check sum fails, it needs to repeat the process (maybe after a delay).
The main application code, when running, will periodically ask the server to do a version check. If there is a new version of code available the application just simply reboots, which kicks off the bootloader which should update the system to the latest version of code.
All sounds simple enough.
At least for now.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
From: Peter Stone [mailto:pwstone@users.sf.net]
Sent: Thursday, July 09, 2015 4:28 AM
To: [gcbasic:discussion]
Subject: [gcbasic:discussion] Remote firmware download scheme
Thanks.
I will give some thought to how I might be able to get a simple solution
working using RESET as the means of kicking off the update. On the 18F26K22
the RESET instruction might be able to be used effectively.
So on power up, the bootloader gets initial control. The bootloader needs to
first get the version number for the current loaded code (must be at a fixed
location). It then needs to connect to the server and ask for the latest
applicable update. If there is an update, it loads it. The bootloader
performs a check sum check on the main code and boots it if all is ok. If
the check sum fails, it needs to repeat the process (maybe after a delay).
The main application code, when running, will periodically ask the server to
do a version check. If there is a new version of code available the
application just simply reboots, which kicks off the bootloader which should
update the system to the latest version of code.
a brief read of the sell sheet indicates a user gui to "help" create your bootloader source code to your needs and generates an .asm file, im not really one for doing much with .asm but maybe worth a go, its got me interested as well now for some ideas lol
I am trying to get my mind around how I might be able to implement a remote software download scheme for my remote field monitoring equipment (to save having to go to site and swap out parts).
My current thinking is to basically break my code into 2 separate parts.
The part in the bottom of memory will contain a minimum set of low level drivers, libraries, etc. Only those required to implement some sort of rudimentary boot code. This boot code will need to be able to check the integrity of the system and communicate with some sort of remote download server (under my control) via TCP/IP. I do not expect this part of the code to change all that frequently. So essentially this will be fixed firmware (only changeable by way of a site visit and direct programming).
The second part of the code will be everything else (application firmware).
The system needs to be able to boot cleanly whether there is any application firmware present or not. This will be achieved with a combination of checksums, signature checks, code size checks, etc.
If the boot code can not start the application code it will ask the download server to provide the latest code and then reboot with default settings.
All operations must be able to be achieved totally without any user intervention!!!!
From a GCB point of view, I think I need to be able to create 2 separate modules of compiled code (Section A & Section B, with Section A being the minimal boot code).
Section A code would be typically loaded into the devices at production time.
Section B code could be loaded at production time also but could just as easily be placed on the download server and only be loaded on request from the device in the field on first power up.
To make life easy, I am thinking about making all the entry points to Section A accessible via some sort of boot table. This boot table will sit at a fixed location (in Section A) so that minor code modifications within Section A can be easily made with absolutely no impact on the separately compiled applications code in Section B. Section B would need some sort of include file to reference the jump table entries for the section A code.
In principle this all sounds fine but I am not sure how GCB would handle the 2 separate compile instances.
It should be relatively easy to force the Section B code to start at a high address (via the boot code features I believe).
I do not know how to handle the issue where a number of the standard GCB library files need to be in the Section A (boot code) area and I do not want them repeated in the Section B area. If I create a special library file for all the calls I need (all in one big include file to point to the jump table in Section A) will the GCB compiler recognise these duplicate entries and not also try to include the standard library code again?????
Essentially I need to turn off the automatic inclusion of some of the referenced standard library files for the Section B compile and include my references to the fixed jump table entries instead.
As an example, the h/w serial port routines and most of the string handling routines will all be needed in Section A. I will also add a jump table at a fixed location in section A to all these library routines. For section B, I need to define a separate include file to redefine all of the calls to the h/w serial routines and the string handling routines so that they go via my jump table. So any call to these library routines made from section B will first call the jump table location which will then jump to the actual real routine in Section A. The return will go straight back to the original caller. If GCB enables library entry points to be redefined at compile time, then I suspect it should all work.
Any thoughts on all of this would be greatly appreciated.
If anyone has a better idea, I am open to suggestions.
Peter.
My thoughts.
I would break the code into three parts.
I think the rest of the approach is doable. There will be challenges but we are here to help.
Thanks.
The complication I face is that the firmware needs to be loaded over a 3G modem using a TCP/IP connection.
There is a bit of code required to set up and utilise the TCP/IP connection which I desire to share between the modules for efficiency.
I already have all the TCP/IP comms s/w coded and working which I just want to relocate into the boot area.
I am sure that I can set this all up with jump tables (just a table of goto instructions in the boot code area).
The problem I need to resolve is how to get GCB to use a different library set (other than the standard one) at compile time.
Is there some compiler switch that enables you to specify where the folder for the main library files live? If so, this would allow me to compile the boot code section with the standard library and then compile my main application code with the modified libraries. The modified libraries would have particular (common) sections just replaced with #define entries. eg a #define for the HSerSend routine that would point to an entry in the jump table in the boot section.
Any further thoughts on how I set up the separate library arrangement?
Peter.
Using a different library set. Yes. Using a define you could redirect all the standard library methods to your library methods. So, you would have the following.
~~~~~
#define Use_My_Libraries
'lots of code
#ifdef Use_My_Libraries
'a list of the all methods you need to adapt
#define HSERPRINT my_HSERPRINT
#define HSERSEND my_HSERSEND
'etc etc
#endif
~~~~
You would need to create all your methods, of course. These would be called only in Use_My_Libraries is defined leaving all the standard methods as is.
But, you may need to adapt the 'lowlevel.dat' file - you may choose to remove all entries and manually add the standard include files in your solution. Why? The initialisation of the standard include files may mess up the redirection to your replacement methods.
Still sounds doable. I use a lot of redirection to develop drivers etc. Work ok. :-)
Thanks for the suggestion.
My concern with this however, is that I would have to maintain an independent set of "standard" library routines that would need hand modifying whenever new library updates were released.
Is there some way that the existing overlaying feature could be used? This would technically allow me to define a new entry point for a particular function (that I would include in a special library file) that would simply be used by the compiler in preference to the original routine. Maybe the compiler throws up an error message (to alert unsuspecting users that they have replaced a standard library function) just to be safe. When I want to compile with the original libraries I just delete the #include reference to my special library file.
I think this would be much easier to maintain in the long term.
Failing all this, a simple switch within the compiler so that you can specify where all the library files are located (other than the default locations) would also work reasonably well. This would allow me to simply copy all the library files to a new location and delete the entry points that I want to redefine in a separate library file.
The original library files would remain intact (in their default location) and could be easily updated as changes were released.
Just thinking a bit more about this.
It appears that the library reference used by the compiler is a relative one. include folder is in the same folder as the compiler.
So if I make a totally separate copy of GCB in another area (call this GCB-B or something else), then I could modify the include files for this version of GCB to implement my jump table idea.
I would then simply compile all normal code (including my bootstrap code) under the original GCB folder. I would then switch to the special GCB-B version when I needed to compile the code that I wish to place in the higher memory area with the jump table references to the code in the bootstrap area.
A little messy but it might work and could simplify library maintenance but would mean maintaining 2 separate copies of the compiler etc.
Any further, more rational thoughts???
Peter.
Peter
Following this thread, I have to say I'm confused about the goal of your questions. At first, I thought you were trying to built a a run-time auto-updater for the firmware in your remote device. Now however, it seems like you're looking for build-time options to build various versions of the code prior to loading into the unit. Can you clarify?
Joe
Hi Joe,
Sorry for the confusion.
The primary objective is to end up with a scheme that enables the remote download and update of code within the PIC chip.
From my current understanding, this will require a bootloader installed in the devices at production time that has the capability to communicate with my cloud server via TCP/IP.
It has to pass some basic access security tests and then negotiate with the server to acquire the latest application firmware appropriate for that particular installation.
Since I will be using much of the serial communications code and TCP/IP code (that I have already written) in both the bootloader and application code areas (as well as requiring access to common variables) there is a desire to share as much code as possible.
I need to be able to separate the bootloader code from the main application code at compile time, since I do not intend to update the boot code during a download. The boot code will be in an area of EPROM that is write protected to ensure that a download failure can not cause the system to end up in an unrecoverable state. These field units are sometimes located in remote, inaccessible locations and must be very reliable.
So what I need to be able to do is compile my application code independently of the bootloader code so that I can store this on my server as a download image. There will most likely be multiple versions of the application code for different clients, different product configurations and even different locations. The application code sets need to be able to access subroutines and functions within the bootloader and access common variables (for transferring details etc). The interrupt service routines for the system (currently 5) will also most likely need to be located in the common bootloader code area (as some of these will at least be needed for serial port and RTC support).
I hope that this clarifies what I am trying to achieve.
Do you have any additional thoughts or implementation ideas for my project?
Thanks for your interest.
Peter.
Peter
This is a very ambitious but interesting project for a PIC or AVR, especially under GCBasic. I have more questions:
1) I assume that the download paradigm will be “pull”, meaning the remote microcontroller requests the object file from a network server somewhere out in the internet cloud, not “push” which is the more common method where a hard-wired host forces code into the microcontroller.
2) What chip are you planning to use and can it support flash-writes, a TCP/IP stack as well as your application + loader code?
3) Do you have firewall & security issues to deal with, like blocked ports/protocols (TFTP/FTP) on the remote device’s local network?
4) How do you propose to get your device onto the TCP/IP network...via an external terminal-server module such as an XPort or with a built-in TCP/IP stack?
5) If it’s commercial product you’re developing, consider that many enterprises are now requiring IPV6 addressing for any device attached to their networks.
6) Have you ever developed an application like a bootloader that has to write to program Flash? It’s tedious and unanticipated results can be catastrophic.
There are numerous articles about this topic on the net. I suspect your head will be spinning once you consider all the challenges, but if you get something running on a PIC or AVR, it would be interesting to everyone.
Joe
From: Peter Stone
Sent: Monday, July 06, 2015 9:34 PM
To: [gcbasic:discussion]
Subject: [gcbasic:discussion] Remote firmware download scheme
Hi Joe,
Sorry for the confusion.
The primary objective is to end up with a scheme that enables the remote download and update of code within the PIC chip.
From my current understanding, this will require a bootloader installed in the devices at production time that has the capability to communicate with my cloud server via TCP/IP.
It has to pass some basic access security tests and then negotiate with the server to acquire the latest application firmware appropriate for that particular installation.
Since I will be using much of the serial communications code and TCP/IP code (that I have already written) in both the bootloader and application code areas (as well as requiring access to common variables) there is a desire to share as much code as possible.
I need to be able to separate the bootloader code from the main application code at compile time, since I do not intend to update the boot code during a download. The boot code will be in an area of EPROM that is write protected to ensure that a download failure can not cause the system to end up in an unrecoverable state. These field units are sometimes located in remote, inaccessible locations and must be very reliable.
So what I need to be able to do is compile my application code independently of the bootloader code so that I can store this on my server as a download image. There will most likely be multiple versions of the application code for different clients, different product configurations and even different locations. The application code sets need to be able to access subroutines and functions within the bootloader and access common variables (for transferring details etc). The interrupt service routines for the system (currently 5) will also most likely need to be located in the common bootloader code area (as some of these will at least be needed for serial port and RTC support).
I hope that this clarifies what I am trying to achieve.
Do you have any additional thoughts or implementation ideas for my project?
Thanks for your interest.
Peter.
Remote firmware download scheme
Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/gcbasic/discussion/579125/
To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2015.0.6037 / Virus Database: 4365/10177 - Release Date: 07/06/15
Joe,
Yes,
I am aware this this going to be a fairly complex task, the outcome of which I am sure will be of considerable interest to others.
Some brief answers to some of the questions asked:
1)I have full control of the server. I have already written custom code that is running under Linux and it communicates with my existing PIC based products over TCP/IP using a custom communications protocol running on dedicated ports.
Here is a link to what the existing remote systems can deliver via a PIC running GCB code.
http://dipstik.info/DipStik0001-Demo-System.jpg
Images and overlay information is updated hourly.
The server comms protocol will be modified to negotiate with the PIC based device when it initiates remote connection. Based on information passed from the remote device the server will push the latest version of application code down to the device.
2) I am using the 18F26K22 currently, but the code should work with other chips.
3) I only have limited programs running on my server at the far end and they respond to a very strict comms protocol. The protocol includes some security check measures for the field devices before data transfer can proceed.
4) Network connection is via my 3G modems designed for M2M communications.
5) I currently use IPV4 but the system will support IPv6. The devices currently only connect to my server.
6) I have done numerous bits of coding with embedded systems over the last 30 years, but I have never implemented bootloader code from scratch. As for PIC stuff, I have done EEPROM stuff with GCB and I am intending to base the bootloader code around the EPROM read/write features within GCB. I have also done some work using GCB to write directly to SD cards. Based on what I have achieved so far, this does not seem to be particularly difficult.
I have worked on other multitasking embedded ROM based systems in the past that have had similar inter-module/task operability requirements.
The hard part for me is trying to nut out how to get the GCB compiler to handle code generated across a number of compile sessions so that it all can be uploaded into the PIC micros in the field reliably (even if a download session fails).
Peter.
Peter
Take a look at TinyBootLoader for a peeled-back version of the bare essentials that a bootloader has to accomplish. It all fits in 100 bytes of high PIC memory. The most challenging part is writing to the Flash memory, as this has to be done in very well-controlled block sizes and boundaries. Also note that GCB, though it handles EEPROM programming well, doesn’t handle Flash programming very well (or not at all) on most parts, so you’ll be somewhat on your own in this area.
I’m encouraged to hear that you’ll be using a “push” approach. With this, you should be able to ‘burn’ a bootloader at manufacturing time that never gets changed in the field. The operating paradigm is generally that the bootloader code runs whenever the part resets, loads the application code if it’s being sent right after reset, then changes the startup vectors so that the newly loaded application can run. You never have both programs available to run at the same time, as they’re separately compiled programs that don’t share memory resources. If you can find a way for the firmware server to put the remote device in reset, then you can almost ‘borrow’ an existing bootloader as is. Of course, your firmware server will need an application that decides when a device needs an update and pushes the code to it. I’ve done this in the past by having the remote devices periodically ‘broadcast’ their firmware version and current IP address to a server app. The server app decides if the code needs to be updated and pushes new code to it if necessary. This has the added advantage of being able to keep track of each unit’s current IP address if dynamic IP addresses are used, as well as keeping a record of which firmware version is in each remote unit. Of course, this is all based on the ‘push’ paradigm. If you choose to let the remote device ‘pull’ its code, things are very different.
Keep us informed.
Joe
From: Peter Stone
Sent: Tuesday, July 07, 2015 10:30 AM
To: [gcbasic:discussion]
Subject: [gcbasic:discussion] Remote firmware download scheme
Joe,
Yes,
I am aware this this going to be a fairly complex task, the outcome of which I am sure will be of considerable interest to others.
Some brief answers to some of the questions asked:
1)I have full control of the server. I have already written custom code that is running under Linux and it communicates with my existing PIC based products over TCP/IP using a custom communications protocol running on dedicated ports.
Here is a link to what the existing remote systems can deliver via a PIC running GCB code.
http://dipstik.info/DipStik0001-Demo-System.jpg
Images and overlay information is updated hourly.
The server comms protocol will be modified to negotiate with the PIC based device when it initiates remote connection. Based on information passed from the remote device the server will push the latest version of application code down to the device.
2) I am using the 18F26K22 currently, but the code should work with other chips.
3) I only have limited programs running on my server at the far end and they respond to a very strict comms protocol. The protocol includes some security check measures for the field devices before data transfer can proceed.
4) Network connection is via my 3G modems designed for M2M communications.
5) I currently use IPV4 but the system will support IPv6. The devices currently only connect to my server.
6) I have done numerous bits of coding with embedded systems over the last 30 years, but I have never implemented bootloader code from scratch. As for PIC stuff, I have done EEPROM stuff with GCB and I am intending to base the bootloader code around the EPROM read/write features within GCB. I have also done some work using GCB to write directly to SD cards. Based on what I have achieved so far, this does not seem to be particularly difficult.
I have worked on other multitasking embedded ROM based systems in the past that have had similar inter-module/task operability requirements.
The hard part for me is trying to nut out how to get the GCB compiler to handle code generated across a number of compile sessions so that it all can be uploaded into the PIC micros in the field reliably (even if a download session fails).
Peter.
Remote firmware download scheme
Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/gcbasic/discussion/579125/
To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2015.0.6037 / Virus Database: 4365/10180 - Release Date: 07/07/15
Thanks.
I will give some thought to how I might be able to get a simple solution working using RESET as the means of kicking off the update. On the 18F26K22 the RESET instruction might be able to be used effectively.
So on power up, the bootloader gets initial control. The bootloader needs to first get the version number for the current loaded code (must be at a fixed location). It then needs to connect to the server and ask for the latest applicable update. If there is an update, it loads it. The bootloader performs a check sum check on the main code and boots it if all is ok. If the check sum fails, it needs to repeat the process (maybe after a delay).
The main application code, when running, will periodically ask the server to do a version check. If there is a new version of code available the application just simply reboots, which kicks off the bootloader which should update the system to the latest version of code.
All sounds simple enough.
At least for now.
Peter
Yes, it all sounds simple enough.
Please keep us informed how you make out.
From: Peter Stone [mailto:pwstone@users.sf.net]
Sent: Thursday, July 09, 2015 4:28 AM
To: [gcbasic:discussion]
Subject: [gcbasic:discussion] Remote firmware download scheme
Thanks.
I will give some thought to how I might be able to get a simple solution
working using RESET as the means of kicking off the update. On the 18F26K22
the RESET instruction might be able to be used effectively.
So on power up, the bootloader gets initial control. The bootloader needs to
first get the version number for the current loaded code (must be at a fixed
location). It then needs to connect to the server and ask for the latest
applicable update. If there is an update, it loads it. The bootloader
performs a check sum check on the main code and boots it if all is ok. If
the check sum fails, it needs to repeat the process (maybe after a delay).
The main application code, when running, will periodically ask the server to
do a version check. If there is a new version of code available the
application just simply reboots, which kicks off the bootloader which should
update the system to the latest version of code.
All sounds simple enough.
At least for now.
Remote firmware download scheme
https://sourceforge.net/p/gcbasic/discussion/579125/thread/15ac72f0/?limit= 25#e027
Sent from sourceforge.net because you indicated interest in
https://sourceforge.net/p/gcbasic/discussion/579125/
https://sourceforge.net/p/gcbasic/discussion/579125
To unsubscribe from further messages, please visit
https://sourceforge.net/auth/subscriptions/
https://sourceforge.net/auth/subscriptions
i just stumbled upon this whilst looking through the additonal docs on microchips page,its for the 16f/18f family of chips
http://ww1.microchip.com/downloads/en/DeviceDoc/30010103A.pdf
a brief read of the sell sheet indicates a user gui to "help" create your bootloader source code to your needs and generates an .asm file, im not really one for doing much with .asm but maybe worth a go, its got me interested as well now for some ideas lol
the bootloader page
http://www.microchip.com/wwwAppNotes/AppNotes.aspx?appnote=en546974
tony
Last edit: tony_g 2015-08-26