astlinux-devel — The AstLinux development environment, compilation, etc

Re: [Astlinux-devel] [Astlinux-users] Stopping logging of Crontab

[Lonnie A. <li...@lo...>]

Hi Michael,

Yes, "building our own image" is the correct way to tweak this. :-)

> • Is it ok to do this?

Not in general, but in this specific case it is probably fine.

If you do the 'sed -i ...' as you stated, you are no longer using the read-only base image '/mnt/asturo/etc/init.d/crond' file, instead you are creating an edited writable '/mnt/asturw/etc/init.d/crond' overlay file.  This will work as long as any future base image /etc/init.d/crond file does not have important changes your edited copy would not contain.

The good news for this specific file, it was last changed on "Nov 2, 2015" and is probably not likely to change in the near future.  As such, your 'sed -i ...' may be a useable fix until you are building our own image.

Understand that when the day comes and you build an image with the fix, you would need to remove the '/mnt/asturw/etc/init.d/crond' file to see the read-only base image version.

> • Will the changes disappear after an upgrade?

The newly created/edited '/mnt/asturw/etc/init.d/crond' file will be used after any upgrade.

Lonnie





> On Sep 29, 2023, at 7:44 PM, Michael Knill <mic...@ip...> wrote:
> 
> Hi Lonnie
>  
> Moving this to the developer list.
> So as part of our upgrade we will run ‘sed -i 's/^  crond$/  crond -L \/dev\/null/g' /etc/init.d/crond’ which seems to work fine.
> So my questions are:
> 	• Is it ok to do this?
> 	• Will the changes disappear after an upgrade?
>  
> We have decided that moving forward (timeframe unknown) we will be forking the repository and building our own image (finally you say!).
> We had toyed with the idea of completely rebuilding the system on standard platforms but now want to continue with Astlinux because of its reliability and small footprint.
> As such, the above will be unnecessary eventually.
>  
> Regards
> Michael Knill
>  
>  
> From: Lonnie Abelbeck <li...@lo...>
> Date: Friday, 29 September 2023 at 4:43 am
> To: AstLinux Users Mailing List <ast...@li...>
> Subject: Re: [Astlinux-users] Stopping logging of Crontab
> 
> Hi Michael,
> 
> Looking at the /etc/init.d/crond init script, here [1]
> 
> If the line "crond" was changed to "crond -L /var/log/crond.log" it would disable syslog and use that file ... but may need rotating if it gets large.
> 
> If the line "crond" was changed to "crond -L /dev/null" it would disable syslog and disable logging (ie. to /dev/null).
> 
> BTW, I manually tested both cases to be certain.
> 
> Lonnie
> 
> [1] https://github.com/astlinux-project/astlinux/blob/09e87eff8bca82bf4afab8dbe09560737dd80d5c/project/astlinux/target_skeleton/etc/init.d/crond#L38
> 
> 
> 
> > On Sep 27, 2023, at 8:01 PM, Michael Knill <mic...@ip...> wrote:
> > 
> > Hi group
> >  
> > Replying to this email again. I do understand below but just wondering if there is any way to turn off Cron logging totally or send to a separate log file?
> >  
> > Regards
> > Michael Knill
> >  
> >  
> > From: Lonnie Abelbeck <li...@lo...>
> > Date: Friday, 31 March 2023 at 1:01 am
> > To: AstLinux Users Mailing List <ast...@li...>
> > Subject: Re: [Astlinux-users] Stopping logging of Crontab
> > 
> > Hi Michael,
> > 
> > The (busybox) crond daemon has a syslog level setting which defaults to 8, the least verbose log level.  So no help there.
> > 
> > Using the filter for the Status Tab, is a reasonable idea.
> > 
> > 
> > Personally, when executing shell commands on a regular interval of seconds/minutes, I prefer to use a bash shell script and the sleep builtin. (Using the sleep builtin keeps from spawning a new process whenever 'sleep' is called).
> > 
> > The simplest example of this is the 'msmtpqueue' bash script [1]
> > 
> > Basic code setup and loop:
> > --
> > #!/bin/bash
> > 
> > LOCKFILE="/var/lock/foobar.lock"
> > 
> > # Robust 'bash' method of creating/testing for a lockfile
> > if ! ( set -o noclobber; echo "$$" > "$LOCKFILE" ) 2>/dev/null; then
> >   echo "foobar: already running, lockfile \"$LOCKFILE\" exists, process id: $(cat "$LOCKFILE")."
> >   return 9
> > fi
> > 
> > # Load 'sleep' builtin if it exists
> > if [ -f /usr/lib/bash/sleep ]; then
> >   enable -f /usr/lib/bash/sleep sleep
> > fi
> > 
> > #seconds to wait
> > wait=300
> > 
> > trap 'rm -f "$LOCKFILE"; exit $?' INT TERM EXIT
> > 
> > while true; do
> >   # do stuff
> > 
> >   sleep $wait
> > done
> > 
> > rm -f "$LOCKFILE"
> > trap - INT TERM EXIT
> > --
> > 
> > Look at the actual code [1] for finer details.  Another fairly simple example, asterisk-sip-monitor [2] which adds a PID file that can be removed to exit the script.
> > 
> > Lonnie
> > 
> > [1] https://github.com/astlinux-project/astlinux/blob/master/package/msmtp/msmtpqueue.sh
> > 
> > [2] https://github.com/astlinux-project/astlinux/blob/master/package/asterisk/asterisk-sip-monitor
> > 
> > 
> > 
> > 
> > 
> > > On Mar 29, 2023, at 11:39 PM, Michael Knill <mic...@ip...> wrote:
> > > 
> > > Short of putting in a filter for the Status Tab, is there any way to stop Crontab logging to Syslog.
> > > I now have a process that is run every 10 minutes and its annoying that it logs to Syslog each time.
> > >  
> > > Regards
> > >  
> > > Michael Knill
> > > Managing Director
> > >  
> > > D: +61 2 6189 1360
> > > P: +61 2 6140 4656
> > > E: mic...@ip...
> > > W: ipcsolutions.com.au
> > > 
> > 
> > 
> > 
> > _______________________________________________
> > Astlinux-users mailing list
> > Ast...@li...
> > https://lists.sourceforge.net/lists/listinfo/astlinux-users
> > 
> > Donations to support AstLinux are graciously accepted via PayPal to pa...@kr....
> > _______________________________________________
> > Astlinux-users mailing list
> > Ast...@li...
> > https://lists.sourceforge.net/lists/listinfo/astlinux-users
> > 
> > Donations to support AstLinux are graciously accepted via PayPal to pa...@kr....
> 
> 
> 
> _______________________________________________
> Astlinux-users mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-users
> 
> Donations to support AstLinux are graciously accepted via PayPal to pa...@kr....
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel

Re: [Astlinux-devel] [Astlinux-users] Stopping logging of Crontab

[Michael K. <mic...@ip...>]

Hi Lonnie

Moving this to the developer list.
So as part of our upgrade we will run ‘sed -i 's/^  crond$/  crond -L \/dev\/null/g' /etc/init.d/crond’ which seems to work fine.
So my questions are:

  1.  Is it ok to do this?
  2.  Will the changes disappear after an upgrade?

We have decided that moving forward (timeframe unknown) we will be forking the repository and building our own image (finally you say!).
We had toyed with the idea of completely rebuilding the system on standard platforms but now want to continue with Astlinux because of its reliability and small footprint.
As such, the above will be unnecessary eventually.

Regards
Michael Knill


From: Lonnie Abelbeck <li...@lo...>
Date: Friday, 29 September 2023 at 4:43 am
To: AstLinux Users Mailing List <ast...@li...>
Subject: Re: [Astlinux-users] Stopping logging of Crontab
Hi Michael,

Looking at the /etc/init.d/crond init script, here [1]

If the line "crond" was changed to "crond -L /var/log/crond.log" it would disable syslog and use that file ... but may need rotating if it gets large.

If the line "crond" was changed to "crond -L /dev/null" it would disable syslog and disable logging (ie. to /dev/null).

BTW, I manually tested both cases to be certain.

Lonnie

[1] https://github.com/astlinux-project/astlinux/blob/09e87eff8bca82bf4afab8dbe09560737dd80d5c/project/astlinux/target_skeleton/etc/init.d/crond#L38



> On Sep 27, 2023, at 8:01 PM, Michael Knill <mic...@ip...> wrote:
>
> Hi group
>
> Replying to this email again. I do understand below but just wondering if there is any way to turn off Cron logging totally or send to a separate log file?
>
> Regards
> Michael Knill
>
>
> From: Lonnie Abelbeck <li...@lo...>
> Date: Friday, 31 March 2023 at 1:01 am
> To: AstLinux Users Mailing List <ast...@li...>
> Subject: Re: [Astlinux-users] Stopping logging of Crontab
>
> Hi Michael,
>
> The (busybox) crond daemon has a syslog level setting which defaults to 8, the least verbose log level.  So no help there.
>
> Using the filter for the Status Tab, is a reasonable idea.
>
>
> Personally, when executing shell commands on a regular interval of seconds/minutes, I prefer to use a bash shell script and the sleep builtin. (Using the sleep builtin keeps from spawning a new process whenever 'sleep' is called).
>
> The simplest example of this is the 'msmtpqueue' bash script [1]
>
> Basic code setup and loop:
> --
> #!/bin/bash
>
> LOCKFILE="/var/lock/foobar.lock"
>
> # Robust 'bash' method of creating/testing for a lockfile
> if ! ( set -o noclobber; echo "$$" > "$LOCKFILE" ) 2>/dev/null; then
>   echo "foobar: already running, lockfile \"$LOCKFILE\" exists, process id: $(cat "$LOCKFILE")."
>   return 9
> fi
>
> # Load 'sleep' builtin if it exists
> if [ -f /usr/lib/bash/sleep ]; then
>   enable -f /usr/lib/bash/sleep sleep
> fi
>
> #seconds to wait
> wait=300
>
> trap 'rm -f "$LOCKFILE"; exit $?' INT TERM EXIT
>
> while true; do
>   # do stuff
>
>   sleep $wait
> done
>
> rm -f "$LOCKFILE"
> trap - INT TERM EXIT
> --
>
> Look at the actual code [1] for finer details.  Another fairly simple example, asterisk-sip-monitor [2] which adds a PID file that can be removed to exit the script.
>
> Lonnie
>
> [1] https://github.com/astlinux-project/astlinux/blob/master/package/msmtp/msmtpqueue.sh
>
> [2] https://github.com/astlinux-project/astlinux/blob/master/package/asterisk/asterisk-sip-monitor
>
>
>
>
>
> > On Mar 29, 2023, at 11:39 PM, Michael Knill <mic...@ip...> wrote:
> >
> > Short of putting in a filter for the Status Tab, is there any way to stop Crontab logging to Syslog.
> > I now have a process that is run every 10 minutes and its annoying that it logs to Syslog each time.
> >
> > Regards
> >
> > Michael Knill
> > Managing Director
> >
> > D: +61 2 6189 1360
> > P: +61 2 6140 4656
> > E: mic...@ip...
> > W: ipcsolutions.com.au
> >
>
>
>
> _______________________________________________
> Astlinux-users mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>
> Donations to support AstLinux are graciously accepted via PayPal to pa...@kr....
> _______________________________________________
> Astlinux-users mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-users
>
> Donations to support AstLinux are graciously accepted via PayPal to pa...@kr....



_______________________________________________
Astlinux-users mailing list
Ast...@li...
https://lists.sourceforge.net/lists/listinfo/astlinux-users

Donations to support AstLinux are graciously accepted via PayPal to pa...@kr....

Re: [Astlinux-devel] /etc/hosts file after internet outage

[David K. <da...@ke...>]

Lonnie,
  Yes indeed.  Lapse of memory on my part, this is indeed something I
changed.  Rather than use a statically defined ULA I query the internal
interface to see what IPv6 address(s) is/are assigned... which could be a
ULA or a GUA or both and set them into /etc/hosts

pbx ~ # interface=br0
pbx ~ # ip_list=$(ip -6 -o addr show dev $interface scope global
2>/dev/null | sed -n -r -e 's/.*inet6 *([0-9a-fA-F:]+).*$/\1/p')

I need to fix this in my own branch.

Thanks
David


On Tue, Sep 26, 2023 at 11:09 AM Lonnie Abelbeck <li...@lo...>
wrote:

> Hi David,
>
> I looked at my setup to refresh my memory, the only IPv6 address in my
> /etc/hosts (not manually defined under Network -> {Configure DNS Hosts} )
> is INTIPV6 which is defined in the Network tab.
>
> In my case all internal IPv6 addresses are 'fd' ULA addresses, mapped with
> the "Network Prefix Translation" firewall plugin.
>
> I don't see any case where /etc/hosts entries are derived from realtime
> 'ip addr ...' values.
>
> Possibly some of your IPv6 enhancements may have changed this in your
> "develop" branch.
>
> Bottom line, if a GUA IPv6 address in entered in the Network -> Internal
> Interfaces or Network -> {Configure DNS Hosts} that would cause issues if
> the GUA prefix changes.
>
> I'm a big fan of internal ULA IPv6 addresses, mapped with the "Network
> Prefix Translation". [1]
>
> Lonnie
>
> [1] https://doc.astlinux-project.org/userdoc:tt_ipv6_ula_nptv6_config
>
>
>
> > On Sep 26, 2023, at 8:31 AM, David Kerr <Da...@Ke...> wrote:
> >
> > Comcast went down on me again overnight.  It did not recover on its own,
> I had to reset the cable modem.  Astlinux handled it well with one
> exception, on recovery I noticed that DHCPv6 had delegated me a new /60
> subnet... different from before.  It is in fact the first time comcast has
> changed the IPv6 delegated subnet for me ever.  So in abundance of caution
> I went and grep'd to see if I had hardcoded the old /60 subnet somewhere.
> >
> > All I found was the /etc/hosts file that was not updated so the IPv6
> address for "pbx" on the local LAN was set using the old subnet.
> >
> > This is set by /etc/init.d/network and also /etc/init.d/dnsmasq
> >
> > It feels like dnsmasq should be restarted if the external network
> interface goes down and then comes back up.  This may be done already (have
> not researched that) but the setting of /etc/hosts is done in the "init"
> part of the code, not the "start" part of the code, so a "restart" is not
> good enough.  I manually ran a service dnsmasq stop followed by service
> dnsmasq init and that rebuilt the hosts file.  But could this be done
> automatically?
> >
> > Thanks
> > David.
> > _______________________________________________
> > Astlinux-devel mailing list
> > Ast...@li...
> > https://lists.sourceforge.net/lists/listinfo/astlinux-devel
>
>
>
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel
>

Re: [Astlinux-devel] /etc/hosts file after internet outage

[Lonnie A. <li...@lo...>]

Hi David,

I looked at my setup to refresh my memory, the only IPv6 address in my /etc/hosts (not manually defined under Network -> {Configure DNS Hosts} ) is INTIPV6 which is defined in the Network tab.

In my case all internal IPv6 addresses are 'fd' ULA addresses, mapped with the "Network Prefix Translation" firewall plugin.

I don't see any case where /etc/hosts entries are derived from realtime 'ip addr ...' values.

Possibly some of your IPv6 enhancements may have changed this in your "develop" branch.

Bottom line, if a GUA IPv6 address in entered in the Network -> Internal Interfaces or Network -> {Configure DNS Hosts} that would cause issues if the GUA prefix changes.

I'm a big fan of internal ULA IPv6 addresses, mapped with the "Network Prefix Translation". [1]

Lonnie

[1] https://doc.astlinux-project.org/userdoc:tt_ipv6_ula_nptv6_config



> On Sep 26, 2023, at 8:31 AM, David Kerr <Da...@Ke...> wrote:
> 
> Comcast went down on me again overnight.  It did not recover on its own, I had to reset the cable modem.  Astlinux handled it well with one exception, on recovery I noticed that DHCPv6 had delegated me a new /60 subnet... different from before.  It is in fact the first time comcast has changed the IPv6 delegated subnet for me ever.  So in abundance of caution I went and grep'd to see if I had hardcoded the old /60 subnet somewhere.
> 
> All I found was the /etc/hosts file that was not updated so the IPv6 address for "pbx" on the local LAN was set using the old subnet.
> 
> This is set by /etc/init.d/network and also /etc/init.d/dnsmasq
> 
> It feels like dnsmasq should be restarted if the external network interface goes down and then comes back up.  This may be done already (have not researched that) but the setting of /etc/hosts is done in the "init" part of the code, not the "start" part of the code, so a "restart" is not good enough.  I manually ran a service dnsmasq stop followed by service dnsmasq init and that rebuilt the hosts file.  But could this be done automatically?
> 
> Thanks
> David.
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel

[Astlinux-devel] /etc/hosts file after internet outage

[David K. <da...@ke...>]

Comcast went down on me again overnight.  It did not recover on its own, I
had to reset the cable modem.  Astlinux handled it well with one exception,
on recovery I noticed that DHCPv6 had delegated me a new /60 subnet...
different from before.  It is in fact the first time comcast has changed
the IPv6 delegated subnet for me ever.  So in abundance of caution I went
and grep'd to see if I had hardcoded the old /60 subnet somewhere.

All I found was the /etc/hosts file that was not updated so the IPv6
address for "pbx" on the local LAN was set using the old subnet.

This is set by /etc/init.d/network and also /etc/init.d/dnsmasq

It feels like dnsmasq should be restarted if the external network interface
goes down and then comes back up.  This may be done already (have not
researched that) but the setting of /etc/hosts is done in the "init" part
of the code, not the "start" part of the code, so a "restart" is not good
enough.  I manually ran a service dnsmasq stop followed by service dnsmasq
init and that rebuilt the hosts file.  But could this be done automatically?

Thanks
David.

[Astlinux-devel] GitHub will soon remove support for Subversion

[Lonnie A. <li...@lo...>]

Hi Devs,

GitHub will soon remove support for Subversion [1]

If you are already using a 'git' repository to build AstLinux, you can ignore this note.

As such, with this commit [2], you will need to use a 'git' repository to build AstLinux.

The developer docs have been tweaked [3] to represent the change to git-only repository support.

Lonnie

[1] https://github.blog/2023-01-20-sunsetting-subversion-support/

[2] https://github.com/astlinux-project/astlinux/commit/e87f72a0efa3b640796278d77bfe5207697e09f3

[3] https://doc.astlinux-project.org/devdoc:documentation

Re: [Astlinux-devel] Reducing memory usage as much as possible

[Lonnie A. <li...@lo...>]

Nice graph. Looks like not much change in iowait except for your extra testing.

Something to keep in mind with your Zabbix RAM graphs, when data is read off the squashfs image, the data is cached, which reduces the "free" RAM but not the "available" RAM as the cached data will be dropped when needed by the kernel. (see FYI data below)

Great to hear 'noram' may be a good solution for you.

Lonnie




> On Aug 27, 2023, at 6:02 PM, Michael Knill <mic...@ip...> wrote:
> 
> Whoops Im stupid. Zabbix is recording iowait time. See the graph below. I made the change on 08-16.
> <image001.png>
>  
> I think there is a small increase but nothing of significance. PS the peaks are in the evening when it does routine maintenance e.g. backups and log rotation etc.
>  
> Regards
> Michael Knill
>  
>  
> From: Lonnie Abelbeck <li...@lo...>
> Date: Sunday, 27 August 2023 at 1:43 am
> To: AstLinux Developers Mailing List <ast...@li...>
> Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
> 
> Hi Michael,
> 
> My guess is the 'noram' squashfs loop mount does not add any significant IO/CPU usage.
> 
> In particular, our squashfs compression is the simplest (gzip) and the kernel caches any accessed files.  The storage (virtio_scsi) is quite fast in a VM.
> 
> I can't suggest any good test metrics, as most everything of interest (squashfs, virtio_scsi and cache) runs in the kernel.
> 
> 
> FYI, on my 512 MB RAM, 2-Core test VM using a 'noram' squashfs loop '/' mount, I performed the following after a fresh reboot. Look over the memory use and how little the IO use is...
> 
> # free
> --
>               total        used        free      shared  buff/cache   available
> Mem:         500948       82228      334844         280       83876      411136
> Swap:             0           0           0
> --
> 
> # top -d1
> --
> Mem: 165776K used, 335172K free, 280K shrd, 4648K buff, 79232K cached
> CPU:  0.0% usr  0.5% sys  0.0% nic 98.5% idle  0.0% io  0.0% irq  0.0% sirq
> --
> 
> ## As a test, copy about 100 MB of files (/usr/) off squashfs to tmpfs
> 
> # mkdir /tmp/disk
> # mount -t tmpfs -o size=150m none /tmp/disk
> # time ( cp -a /usr/. /tmp/disk/ )
> --
> real    0m0.783s
> 
> (top) 2.4% io, 4.5% io (two 1 second samples)
> --
> 
> # top -d1
> --
> Mem: 359280K used, 141668K free, 103400K shrd, 4648K buff, 265492K cached
> CPU:  0.4% usr  0.4% sys  0.0% nic 98.5% idle  0.0% io  0.0% irq  0.0% sirq
> --
> 
> # free
> --
>               total        used        free      shared  buff/cache   available
> Mem:         500948       89140      141668      103400      270140      303776
> Swap:             0           0           0
> --
> 
> ## Remove the tmpfs files
> 
> # rm -rf /tmp/disk/*
> 
> # top -d1
> --
> Mem: 254220K used, 246728K free, 280K shrd, 4648K buff, 162372K cached
> CPU:  0.4% usr  0.4% sys  0.0% nic 99.0% idle  0.0% io  0.0% irq  0.0% sirq
> --
> 
> # free
> --
>               total        used        free      shared  buff/cache   available
> Mem:         500948       87356      246572         280      167020      408472
> Swap:             0           0           0
> --
> 
> Additionally, the Asterisk sound files are on the ext2/ext4 FUSE unionfs overlay.
> 
> My guess is your VM's 250 IOPS setting is fine if that was working well before.
> 
> 
> > The results were actually unexpected. The io% was actually LOWER on the system without tmpfs and noticeably higher on ones with tmpfs with 1.3.10 being the worst.
> > Now we are not talking high here e.g. occasionally seeing a reading up to 0.2% vs regularly seeing higher, up to 1.6%, but certainly a noticeable difference.
> 
> I suspect the newer kernels have better performance, regardless of tmpfs vs squashfs (noram) '/' mount.  The Asterisk version could make a difference as well.
> 
> 
> Lonnie
> 
> 
> 
> 
> > On Aug 25, 2023, at 8:48 PM, Michael Knill <mic...@ip...> wrote:
> > 
> > Hmm here is an interesting one.
> > 
> > I did some testing using ‘top’ and looking at the ‘io’ percentage as I assumed that disk access would be the most likely to produce an io wait.
> > 
> > While making continuous calls into the system, I watched the io% also trying it on different systems in the same vDC (so basically identical) and different Astlinux versions (1.3.10 & 1.4.7).
> > 
> >  
> > 
> > The results were actually unexpected. The io% was actually LOWER on the system without tmpfs and noticeably higher on ones with tmpfs with 1.3.10 being the worst.
> > 
> > Now we are not talking high here e.g. occasionally seeing a reading up to 0.2% vs regularly seeing higher, up to 1.6%, but certainly a noticeable difference.
> > 
> > I also tested on KVM’s in another DC on 1.3.10 with tmpfs for interest and the results were similar to the system without tmpfs e.g. 0.0%-0.2%, but Im pretty sure they are using faster storage.
> > 
> >  
> > 
> > Is this a valid test? If so, any reason you can think of why it is better?
> > 
> >  
> > 
> > Regards
> > 
> > Michael Knill
> > 
> >  
> > 
> >  
> > 
> > From: Michael Knill <mic...@ip...>
> > Date: Saturday, 26 August 2023 at 10:26 am
> > To: AstLinux Developers Mailing List <ast...@li...>
> > Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
> > 
> > Just some feedback on this change. There have been no issues detected with very minor variations in memory usage:
> > 
> > <image001.png>
> >  
> > 
> > My concern at this stage is that I am using fairly slow storage at 250 IOPS and as there is no tmpfs whether this will impact anything?
> > 
> > How would I test this?
> > 
> >  
> > 
> > I can push it up to 500 IOPS with relatively minimal expense (much less so than RAM) but I wont bother if I don’t need to.
> > 
> >  
> > 
> > Your thoughts would be greatly appreciated.
> > 
> >  
> > 
> > Regards
> > 
> > Michael Knill
> > 
> >  
> > 
> >  
> > 
> > From: Michael Knill <mic...@ip...>
> > Date: Friday, 18 August 2023 at 8:04 am
> > To: AstLinux Developers Mailing List <ast...@li...>
> > Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
> > 
> > Yes that is amazing. I started playing with Astlinux sometime in 2005/6 I believe. Left it for a bit and came back to it and now have over 100 systems out there and looking to expand to much more!
> > 
> >  
> > 
> > Thanks for the fix. Pretty sure we have never configured the /etc/inittab file so shouldn’t be an issue.
> > 
> >  
> > 
> > So I made some changes to our office system (always ends up being the guinea pig) using noram, dropping out ethernet drivers (I assumed I needed vmxnet3 also as I use VMWare) and took out a few Asterisk modules from modules.conf.
> > 
> >  
> > 
> > Its now running with 281 MB, Available 147 MB so around 135M usage. Looking at my monitoring over the last 3 months I only saw a 50M variation so I think the current setup should be fine. We get notified by Zabbix if it reduces below 50M. Lets see how it goes. Even though its only a single processor, I suspect an Intel Xeon Gold 5118 (1x) @ 2294 MHz will be fine.
> > 
> >  
> > 
> > PS I have also noticed that it boots up faster. In my build system on Vultr I rebooted the system and could log back in around 45 seconds later. Nice!
> > 
> >  
> > 
> > Regards
> > 
> > Michael Knill
> > 
> >  
> > 
> >  
> > 
> > From: Lonnie Abelbeck <li...@lo...>
> > Date: Friday, 18 August 2023 at 1:50 am
> > To: AstLinux Developers Mailing List <ast...@li...>
> > Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
> > 
> > Cool how after almost 20 years, old AstLinux features become new again :-)
> > 
> > A fix for the "Very Important" warning has been added to master [1].  This would be difficult to fix at the web interface level, other than disabling "Revert to Previous".
> > 
> > I did discover a very minor issue with a "noram" configuration.  If the /etc/inittab file was edited (rarely needed) the edited /etc/inittab is applied to the filesystem after /usr/sbin/chroot is called in the initrd, as such the default base image /etc/inittab is always used.  A custom build for genx86_64-vm could customize the /etc/inittab here [2].  For the tmpfs root filesystem case, the initrd copies-forward the /etc/inittab file to the uncompressed tmpfs image before calling /usr/sbin/chroot.  Again, should not be a big deal for a "noram" configuration.  BTW, before this was fixed for the tmpfs case, AstLinux 1.3.10 through 1.4.2 ignored any user edits to the /etc/inittab file and nobody noticed :-)
> > 
> > Additionally, if you wanted to build a custom genx86_64-vm image with "noram" enabled by default, you would add noram after astlive here [3].
> > 
> > Lonnie
> > 
> > [1] https://github.com/astlinux-project/astlinux/commit/09eed53ddea8f45414195507aeee6d0f8edb9e92
> > 
> > [2] https://github.com/astlinux-project/astlinux/blob/master/project/astlinux/board/genx86_64-vm/target_skeleton/etc/inittab
> > 
> > [3] https://github.com/astlinux-project/astlinux/blob/master/project/astlinux/board/genx86_64-vm/runnix.conf
> > 
> > 
> > 
> > > On Aug 17, 2023, at 12:04 AM, Michael Knill <mic...@ip...> wrote:
> > > 
> > > Wow thanks Lonnie that is fantastic and we will test it out. Gosh this will change things considerably. Will be even less once we pull out other modules we don’t use. Having such a lean image will mean that we don’t really need to explore difficult to manage containerisation options such as Docker.
> > > Will update the System Tab so that you cannot perform the revert and upgrade as you mentioned.
> > >  
> > > Regards
> > > Michael Knill
> > >  
> > >  
> > > From: Lonnie Abelbeck <li...@lo...>
> > > Date: Thursday, 17 August 2023 at 1:59 am
> > > To: AstLinux Developers Mailing List <ast...@li...>
> > > Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
> > > 
> > > 
> > > > On Aug 16, 2023, at 3:41 AM, Michael Keuter <li...@mk...> wrote:
> > > > 
> > > > 
> > > >> Am 16.08.2023 um 05:36 schrieb Michael Knill <mic...@ip...>:
> > > >> 
> > > >> Hi Devs
> > > >> Im looking to expand considerably and it will be all in the cloud.
> > > >> Our current hosting provider allows us to overcommit CPU resources which works well for us as Astlinux uses nothing however memory is something that cannot be overcommitted and is actually the most expensive of all the resources. As such, I want to have a build that uses the minimal amount of memory I can possibly get away with safely for my small systems. Note this is only for the small systems which have very low usage and its very easy to add memory if required.
> > > >> I believe have turned off as much as I can:<image002.png> So is there anything else I can turn off, no matter how small?
> > > >> Can I get rid of tmpfs somehow since a read only file system is not necessary?
> > > >> My ultimate goal is to have a VM template that I just push a button and its all built and ready to go and a minimal image will make this far more competitive for the mentioned high end hosting. Of course we could just use Vultr which we love but you get what you pay for ☹
> > > >> Regards
> > > >> Michael Knill
> > > >> Managing Director
> > > >> D: +61 2 6189 1360
> > > >> P: +61 2 6140 4656
> > > >> E: mic...@ip...
> > > >> W: ipcsolutions.com.au
> > > >>  <image001.png>Smarter Business Communications
> > > > 
> > > > Hi Michael,
> > > > 
> > > > I have here an older list of packages, which I disabled (from the defaults) for Alix boards (at that time):
> > > > 
> > > > Asterisk: dahdi_echocan_oslec, prosody off
> > > > Asterisk: dahdi_no_card_firmware on
> > > > Hardware handling: monit, zabbix_proxy, zabbix off
> > > > Libraries/Networking: pjsip, 'Asterisk PRI Support' off
> > > > Networking apps: keepalived, netsnmp, openldap-server, strongswan off
> > > > 
> > > > These were all packages that were not needed on those boxes.
> > > > 
> > > > Michael
> > > 
> > > Micheal Keuter brings up a good point.  Though disabling packages in the build system may not be required.
> > > 
> > > In AstLinux, memory usage has two elements:
> > > 
> > > 1) Standard application and kernel RAM usage.
> > > 1a) Reducing memory usage can be achieved by only starting needed services and tuning the /etc/rc.modules to comment-out unused NIC modules.
> > > 
> > > 2) The Read-Only root filesystem "/" is by default mounted uncompressed using tmpfs.
> > > 2a) Reducing memory usage can be achieved by adding "noram" to the KCMD of the /oldroot/cdrom/os/*run.conf file(s).  In the "noram" case the root filesystem "/" is mounted compressed using squashfs, via /dev/loop0.
> > > 
> > > First, a little history.
> > > 
> > > Back in the day when CompactFlash (CF) storage was the norm and boards contained 256 MB or more RAM, the Read-Only root filesystem "/" was copied to RAM (tmpfs).  This served two purposes,
> > > 1) no realtime uncompressing of the squashfs run image noticeably reduced file access latency and speed on single core 500 Mhz (or less) CPUs
> > > 2) root filesystem access was much faster using RAM (tmpfs) rather than a loop mount directly off CF storage.
> > > 
> > > In later years, while the advantages of the tmpfs root filesystem were less pronounced as SATA storage appeared and multi-core CPUs, there are still advantages today to make a tmpfs root filesystem still the default.  Eliminating the loop mount directly off the storage is probably the main advantage today (KISS), and systems with 1 GB RAM or more there is no practical RAM disadvantage.
> > > 
> > > As per Michael Knill's use case above, there may be a legitimate case for adding "noram" in the KCMD of the /oldroot/cdrom/os/*run.conf file(s).
> > > 
> > > Demonstrate via an example.
> > > 
> > > Using the latest pre-release [1] Guest VM x86-64bit ISO [2] with Proxmox 8, standard build using Asterisk 20.4.0 with pjsip loaded.
> > > 
> > > The Proxmox VM is configured with 2-cores and 512 MB RAM
> > > 
> > > Default case:
> > > Proxmox: Memory usage 38.13% (195.24 MiB of 512.00 MiB)
> > > --
> > >       Host Name: pbx.astlinux
> > >           Linux: 5.10.179-astlinux x86_64
> > >             CPU: Common KVM processor (2x) @ 2096 MHz
> > >             RAM: 489 MB, Available 253 MB
> > >      Board Type: genx86_64-vm
> > >        Hardware: KVM Hypervisor Guest VM
> > > --
> > > none on / type tmpfs (ro,relatime,size=134128k)
> > > none                    143.4M    143.4M         0 100% /
> > > 
> > > Note that 143.4 MB of RAM is used for the root filesystem.
> > > 
> > > Next, add "noram" to the KCMD= line ...
> > > --
> > > # mount -o rw,remount /oldroot/cdrom
> > > 
> > > Edit the KCMD= line by adding noram after astlive
> > > # vi /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
> > > After the change, it should show...
> > > # grep '^KCMD=' /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
> > > KCMD="root=/dev/ram0 rw init=/linuxrc astlinux=genx86_64-vm astimg=astlinux-1.5-5855-4d7596.run astkd=auto asturw=auto astlive noram libata.dma=3"
> > > 
> > > # mount -o ro,remount /oldroot/cdrom
> > > --
> > > 
> > > Very Important ->  With "noram" in the KCMD, never perform a firmware "Revert to Previous" followed by an "Upgrade with New" without rebooting immediately after the firmware revert.
> > > 
> > > "noram" case:
> > > Proxmox: Memory usage 19.81% (101.42 MiB of 512.00 MiB)
> > > --
> > >       Host Name: pbx.astlinux
> > >           Linux: 5.10.179-astlinux x86_64
> > >             CPU: Common KVM processor (2x) @ 2096 MHz
> > >             RAM: 489 MB, Available 400 MB
> > >      Board Type: genx86_64-vm
> > >        Hardware: KVM Hypervisor Guest VM
> > > --
> > > /dev/loop0 on / type squashfs (ro,relatime)
> > > /dev/loop0               51.1M     51.1M         0 100% /
> > > 
> > > Note that no extra RAM is used, the 51.1 MB is the compressed squashfs run image directly mounted off storage.
> > > 
> > > Interestingly, the AstLinux 400 MB vs. 253 MB Available RAM closely matches the tmpfs root filesystem size, but the Proxmox Memory usage only dropped to 101 MB vs. 195 MB.
> > > 
> > > Additional RAM (albeit minor) RAM reduction
> > > Disable unused NIC modules (Proxmox example)
> > > --
> > > pbx ~ # cat /etc/rc.modules 
> > > ## These modules get modprobe'd when the system starts up.
> > > ##
> > > ## Comment out any modules you don't need.
> > > ##
> > > ## Ethernet support
> > > ## Virtual first, then Emulated
> > > virtio_net
> > > #vmxnet3
> > > #pcnet32
> > > #e1000
> > > #e1000e
> > > #8139too
> > > --
> > > 
> > > In summary, setting "noram" in the /oldroot/cdrom/os/*run.conf KCMD should be a production-worthy option for special low-RAM VM situations, but since it has not been used by default, testing should be performed.  Don't forget the "Very Important" notice above.
> > > 
> > > Also note that once "noram" in the /oldroot/cdrom/os/*run.conf KCMD is defined, it is automatically carried forward on future upgrades.
> > > 
> > > Lonnie
> > > 
> > > [1] https://www.astlinux-project.org/dev.html
> > > 
> > > [2] https://astlinux-project.org/beta/iso/astlinux-beta-genx86_64-vm.iso
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > _______________________________________________
> > > Astlinux-devel mailing list
> > > Ast...@li...
> > > https://lists.sourceforge.net/lists/listinfo/astlinux-devel
> > > _______________________________________________
> > > Astlinux-devel mailing list
> > > Ast...@li...
> > > https://lists.sourceforge.net/lists/listinfo/astlinux-devel
> > 
> > 
> > 
> > _______________________________________________
> > Astlinux-devel mailing list
> > Ast...@li...
> > https://lists.sourceforge.net/lists/listinfo/astlinux-devel
> > 
> > _______________________________________________
> > Astlinux-devel mailing list
> > Ast...@li...
> > https://lists.sourceforge.net/lists/listinfo/astlinux-devel
> 
> 
> 
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel

Re: [Astlinux-devel] Reducing memory usage as much as possible

[Michael K. <mic...@ip...>]

Whoops Im stupid. Zabbix is recording iowait time. See the graph below. I made the change on 08-16.
[cid:image001.png@01D9D98C.99D285E0]

I think there is a small increase but nothing of significance. PS the peaks are in the evening when it does routine maintenance e.g. backups and log rotation etc.

Regards
Michael Knill


From: Lonnie Abelbeck <li...@lo...>
Date: Sunday, 27 August 2023 at 1:43 am
To: AstLinux Developers Mailing List <ast...@li...>
Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
Hi Michael,

My guess is the 'noram' squashfs loop mount does not add any significant IO/CPU usage.

In particular, our squashfs compression is the simplest (gzip) and the kernel caches any accessed files.  The storage (virtio_scsi) is quite fast in a VM.

I can't suggest any good test metrics, as most everything of interest (squashfs, virtio_scsi and cache) runs in the kernel.


FYI, on my 512 MB RAM, 2-Core test VM using a 'noram' squashfs loop '/' mount, I performed the following after a fresh reboot. Look over the memory use and how little the IO use is...

# free
--
              total        used        free      shared  buff/cache   available
Mem:         500948       82228      334844         280       83876      411136
Swap:             0           0           0
--

# top -d1
--
Mem: 165776K used, 335172K free, 280K shrd, 4648K buff, 79232K cached
CPU:  0.0% usr  0.5% sys  0.0% nic 98.5% idle  0.0% io  0.0% irq  0.0% sirq
--

## As a test, copy about 100 MB of files (/usr/) off squashfs to tmpfs

# mkdir /tmp/disk
# mount -t tmpfs -o size=150m none /tmp/disk
# time ( cp -a /usr/. /tmp/disk/ )
--
real    0m0.783s

(top) 2.4% io, 4.5% io (two 1 second samples)
--

# top -d1
--
Mem: 359280K used, 141668K free, 103400K shrd, 4648K buff, 265492K cached
CPU:  0.4% usr  0.4% sys  0.0% nic 98.5% idle  0.0% io  0.0% irq  0.0% sirq
--

# free
--
              total        used        free      shared  buff/cache   available
Mem:         500948       89140      141668      103400      270140      303776
Swap:             0           0           0
--

## Remove the tmpfs files

# rm -rf /tmp/disk/*

# top -d1
--
Mem: 254220K used, 246728K free, 280K shrd, 4648K buff, 162372K cached
CPU:  0.4% usr  0.4% sys  0.0% nic 99.0% idle  0.0% io  0.0% irq  0.0% sirq
--

# free
--
              total        used        free      shared  buff/cache   available
Mem:         500948       87356      246572         280      167020      408472
Swap:             0           0           0
--

Additionally, the Asterisk sound files are on the ext2/ext4 FUSE unionfs overlay.

My guess is your VM's 250 IOPS setting is fine if that was working well before.


> The results were actually unexpected. The io% was actually LOWER on the system without tmpfs and noticeably higher on ones with tmpfs with 1.3.10 being the worst.
> Now we are not talking high here e.g. occasionally seeing a reading up to 0.2% vs regularly seeing higher, up to 1.6%, but certainly a noticeable difference.

I suspect the newer kernels have better performance, regardless of tmpfs vs squashfs (noram) '/' mount.  The Asterisk version could make a difference as well.


Lonnie




> On Aug 25, 2023, at 8:48 PM, Michael Knill <mic...@ip...> wrote:
>
> Hmm here is an interesting one.
>
> I did some testing using ‘top’ and looking at the ‘io’ percentage as I assumed that disk access would be the most likely to produce an io wait.
>
> While making continuous calls into the system, I watched the io% also trying it on different systems in the same vDC (so basically identical) and different Astlinux versions (1.3.10 & 1.4.7).
>
>
>
> The results were actually unexpected. The io% was actually LOWER on the system without tmpfs and noticeably higher on ones with tmpfs with 1.3.10 being the worst.
>
> Now we are not talking high here e.g. occasionally seeing a reading up to 0.2% vs regularly seeing higher, up to 1.6%, but certainly a noticeable difference.
>
> I also tested on KVM’s in another DC on 1.3.10 with tmpfs for interest and the results were similar to the system without tmpfs e.g. 0.0%-0.2%, but Im pretty sure they are using faster storage.
>
>
>
> Is this a valid test? If so, any reason you can think of why it is better?
>
>
>
> Regards
>
> Michael Knill
>
>
>
>
>
> From: Michael Knill <mic...@ip...>
> Date: Saturday, 26 August 2023 at 10:26 am
> To: AstLinux Developers Mailing List <ast...@li...>
> Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
>
> Just some feedback on this change. There have been no issues detected with very minor variations in memory usage:
>
> <image001.png>
>
>
> My concern at this stage is that I am using fairly slow storage at 250 IOPS and as there is no tmpfs whether this will impact anything?
>
> How would I test this?
>
>
>
> I can push it up to 500 IOPS with relatively minimal expense (much less so than RAM) but I wont bother if I don’t need to.
>
>
>
> Your thoughts would be greatly appreciated.
>
>
>
> Regards
>
> Michael Knill
>
>
>
>
>
> From: Michael Knill <mic...@ip...>
> Date: Friday, 18 August 2023 at 8:04 am
> To: AstLinux Developers Mailing List <ast...@li...>
> Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
>
> Yes that is amazing. I started playing with Astlinux sometime in 2005/6 I believe. Left it for a bit and came back to it and now have over 100 systems out there and looking to expand to much more!
>
>
>
> Thanks for the fix. Pretty sure we have never configured the /etc/inittab file so shouldn’t be an issue.
>
>
>
> So I made some changes to our office system (always ends up being the guinea pig) using noram, dropping out ethernet drivers (I assumed I needed vmxnet3 also as I use VMWare) and took out a few Asterisk modules from modules.conf.
>
>
>
> Its now running with 281 MB, Available 147 MB so around 135M usage. Looking at my monitoring over the last 3 months I only saw a 50M variation so I think the current setup should be fine. We get notified by Zabbix if it reduces below 50M. Lets see how it goes. Even though its only a single processor, I suspect an Intel Xeon Gold 5118 (1x) @ 2294 MHz will be fine.
>
>
>
> PS I have also noticed that it boots up faster. In my build system on Vultr I rebooted the system and could log back in around 45 seconds later. Nice!
>
>
>
> Regards
>
> Michael Knill
>
>
>
>
>
> From: Lonnie Abelbeck <li...@lo...>
> Date: Friday, 18 August 2023 at 1:50 am
> To: AstLinux Developers Mailing List <ast...@li...>
> Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
>
> Cool how after almost 20 years, old AstLinux features become new again :-)
>
> A fix for the "Very Important" warning has been added to master [1].  This would be difficult to fix at the web interface level, other than disabling "Revert to Previous".
>
> I did discover a very minor issue with a "noram" configuration.  If the /etc/inittab file was edited (rarely needed) the edited /etc/inittab is applied to the filesystem after /usr/sbin/chroot is called in the initrd, as such the default base image /etc/inittab is always used.  A custom build for genx86_64-vm could customize the /etc/inittab here [2].  For the tmpfs root filesystem case, the initrd copies-forward the /etc/inittab file to the uncompressed tmpfs image before calling /usr/sbin/chroot.  Again, should not be a big deal for a "noram" configuration.  BTW, before this was fixed for the tmpfs case, AstLinux 1.3.10 through 1.4.2 ignored any user edits to the /etc/inittab file and nobody noticed :-)
>
> Additionally, if you wanted to build a custom genx86_64-vm image with "noram" enabled by default, you would add noram after astlive here [3].
>
> Lonnie
>
> [1] https://github.com/astlinux-project/astlinux/commit/09eed53ddea8f45414195507aeee6d0f8edb9e92
>
> [2] https://github.com/astlinux-project/astlinux/blob/master/project/astlinux/board/genx86_64-vm/target_skeleton/etc/inittab
>
> [3] https://github.com/astlinux-project/astlinux/blob/master/project/astlinux/board/genx86_64-vm/runnix.conf
>
>
>
> > On Aug 17, 2023, at 12:04 AM, Michael Knill <mic...@ip...> wrote:
> >
> > Wow thanks Lonnie that is fantastic and we will test it out. Gosh this will change things considerably. Will be even less once we pull out other modules we don’t use. Having such a lean image will mean that we don’t really need to explore difficult to manage containerisation options such as Docker.
> > Will update the System Tab so that you cannot perform the revert and upgrade as you mentioned.
> >
> > Regards
> > Michael Knill
> >
> >
> > From: Lonnie Abelbeck <li...@lo...>
> > Date: Thursday, 17 August 2023 at 1:59 am
> > To: AstLinux Developers Mailing List <ast...@li...>
> > Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
> >
> >
> > > On Aug 16, 2023, at 3:41 AM, Michael Keuter <li...@mk...> wrote:
> > >
> > >
> > >> Am 16.08.2023 um 05:36 schrieb Michael Knill <mic...@ip...>:
> > >>
> > >> Hi Devs
> > >> Im looking to expand considerably and it will be all in the cloud.
> > >> Our current hosting provider allows us to overcommit CPU resources which works well for us as Astlinux uses nothing however memory is something that cannot be overcommitted and is actually the most expensive of all the resources. As such, I want to have a build that uses the minimal amount of memory I can possibly get away with safely for my small systems. Note this is only for the small systems which have very low usage and its very easy to add memory if required.
> > >> I believe have turned off as much as I can:<image002.png> So is there anything else I can turn off, no matter how small?
> > >> Can I get rid of tmpfs somehow since a read only file system is not necessary?
> > >> My ultimate goal is to have a VM template that I just push a button and its all built and ready to go and a minimal image will make this far more competitive for the mentioned high end hosting. Of course we could just use Vultr which we love but you get what you pay for ☹
> > >> Regards
> > >> Michael Knill
> > >> Managing Director
> > >> D: +61 2 6189 1360
> > >> P: +61 2 6140 4656
> > >> E: mic...@ip...
> > >> W: ipcsolutions.com.au
> > >>  <image001.png>Smarter Business Communications
> > >
> > > Hi Michael,
> > >
> > > I have here an older list of packages, which I disabled (from the defaults) for Alix boards (at that time):
> > >
> > > Asterisk: dahdi_echocan_oslec, prosody off
> > > Asterisk: dahdi_no_card_firmware on
> > > Hardware handling: monit, zabbix_proxy, zabbix off
> > > Libraries/Networking: pjsip, 'Asterisk PRI Support' off
> > > Networking apps: keepalived, netsnmp, openldap-server, strongswan off
> > >
> > > These were all packages that were not needed on those boxes.
> > >
> > > Michael
> >
> > Micheal Keuter brings up a good point.  Though disabling packages in the build system may not be required.
> >
> > In AstLinux, memory usage has two elements:
> >
> > 1) Standard application and kernel RAM usage.
> > 1a) Reducing memory usage can be achieved by only starting needed services and tuning the /etc/rc.modules to comment-out unused NIC modules.
> >
> > 2) The Read-Only root filesystem "/" is by default mounted uncompressed using tmpfs.
> > 2a) Reducing memory usage can be achieved by adding "noram" to the KCMD of the /oldroot/cdrom/os/*run.conf file(s).  In the "noram" case the root filesystem "/" is mounted compressed using squashfs, via /dev/loop0.
> >
> > First, a little history.
> >
> > Back in the day when CompactFlash (CF) storage was the norm and boards contained 256 MB or more RAM, the Read-Only root filesystem "/" was copied to RAM (tmpfs).  This served two purposes,
> > 1) no realtime uncompressing of the squashfs run image noticeably reduced file access latency and speed on single core 500 Mhz (or less) CPUs
> > 2) root filesystem access was much faster using RAM (tmpfs) rather than a loop mount directly off CF storage.
> >
> > In later years, while the advantages of the tmpfs root filesystem were less pronounced as SATA storage appeared and multi-core CPUs, there are still advantages today to make a tmpfs root filesystem still the default.  Eliminating the loop mount directly off the storage is probably the main advantage today (KISS), and systems with 1 GB RAM or more there is no practical RAM disadvantage.
> >
> > As per Michael Knill's use case above, there may be a legitimate case for adding "noram" in the KCMD of the /oldroot/cdrom/os/*run.conf file(s).
> >
> > Demonstrate via an example.
> >
> > Using the latest pre-release [1] Guest VM x86-64bit ISO [2] with Proxmox 8, standard build using Asterisk 20.4.0 with pjsip loaded.
> >
> > The Proxmox VM is configured with 2-cores and 512 MB RAM
> >
> > Default case:
> > Proxmox: Memory usage 38.13% (195.24 MiB of 512.00 MiB)
> > --
> >       Host Name: pbx.astlinux
> >           Linux: 5.10.179-astlinux x86_64
> >             CPU: Common KVM processor (2x) @ 2096 MHz
> >             RAM: 489 MB, Available 253 MB
> >      Board Type: genx86_64-vm
> >        Hardware: KVM Hypervisor Guest VM
> > --
> > none on / type tmpfs (ro,relatime,size=134128k)
> > none                    143.4M    143.4M         0 100% /
> >
> > Note that 143.4 MB of RAM is used for the root filesystem.
> >
> > Next, add "noram" to the KCMD= line ...
> > --
> > # mount -o rw,remount /oldroot/cdrom
> >
> > Edit the KCMD= line by adding noram after astlive
> > # vi /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
> > After the change, it should show...
> > # grep '^KCMD=' /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
> > KCMD="root=/dev/ram0 rw init=/linuxrc astlinux=genx86_64-vm astimg=astlinux-1.5-5855-4d7596.run astkd=auto asturw=auto astlive noram libata.dma=3"
> >
> > # mount -o ro,remount /oldroot/cdrom
> > --
> >
> > Very Important ->  With "noram" in the KCMD, never perform a firmware "Revert to Previous" followed by an "Upgrade with New" without rebooting immediately after the firmware revert.
> >
> > "noram" case:
> > Proxmox: Memory usage 19.81% (101.42 MiB of 512.00 MiB)
> > --
> >       Host Name: pbx.astlinux
> >           Linux: 5.10.179-astlinux x86_64
> >             CPU: Common KVM processor (2x) @ 2096 MHz
> >             RAM: 489 MB, Available 400 MB
> >      Board Type: genx86_64-vm
> >        Hardware: KVM Hypervisor Guest VM
> > --
> > /dev/loop0 on / type squashfs (ro,relatime)
> > /dev/loop0               51.1M     51.1M         0 100% /
> >
> > Note that no extra RAM is used, the 51.1 MB is the compressed squashfs run image directly mounted off storage.
> >
> > Interestingly, the AstLinux 400 MB vs. 253 MB Available RAM closely matches the tmpfs root filesystem size, but the Proxmox Memory usage only dropped to 101 MB vs. 195 MB.
> >
> > Additional RAM (albeit minor) RAM reduction
> > Disable unused NIC modules (Proxmox example)
> > --
> > pbx ~ # cat /etc/rc.modules
> > ## These modules get modprobe'd when the system starts up.
> > ##
> > ## Comment out any modules you don't need.
> > ##
> > ## Ethernet support
> > ## Virtual first, then Emulated
> > virtio_net
> > #vmxnet3
> > #pcnet32
> > #e1000
> > #e1000e
> > #8139too
> > --
> >
> > In summary, setting "noram" in the /oldroot/cdrom/os/*run.conf KCMD should be a production-worthy option for special low-RAM VM situations, but since it has not been used by default, testing should be performed.  Don't forget the "Very Important" notice above.
> >
> > Also note that once "noram" in the /oldroot/cdrom/os/*run.conf KCMD is defined, it is automatically carried forward on future upgrades.
> >
> > Lonnie
> >
> > [1] https://www.astlinux-project.org/dev.html
> >
> > [2] https://astlinux-project.org/beta/iso/astlinux-beta-genx86_64-vm.iso
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > Astlinux-devel mailing list
> > Ast...@li...
> > https://lists.sourceforge.net/lists/listinfo/astlinux-devel
> > _______________________________________________
> > Astlinux-devel mailing list
> > Ast...@li...
> > https://lists.sourceforge.net/lists/listinfo/astlinux-devel
>
>
>
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel
>
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel



_______________________________________________
Astlinux-devel mailing list
Ast...@li...
https://lists.sourceforge.net/lists/listinfo/astlinux-devel

Re: [Astlinux-devel] Reducing memory usage as much as possible

[Michael K. <mic...@ip...>]

Thanks Lonnie

Yes I forgot that I am using a different Asterisk version between 1.3.10 & 1.4.7 (13 => 16) so that could also be changing it. But even the differences between exactly the same versions in the same vDC was noticeable.

Anyway I think I have a new standard for my VM’s. This new low memory footprint is going to change the way we do things for the better.

Thanks so much for your help once again.

Regards
Michael Knill


From: Lonnie Abelbeck <li...@lo...>
Date: Sunday, 27 August 2023 at 1:43 am
To: AstLinux Developers Mailing List <ast...@li...>
Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
Hi Michael,

My guess is the 'noram' squashfs loop mount does not add any significant IO/CPU usage.

In particular, our squashfs compression is the simplest (gzip) and the kernel caches any accessed files.  The storage (virtio_scsi) is quite fast in a VM.

I can't suggest any good test metrics, as most everything of interest (squashfs, virtio_scsi and cache) runs in the kernel.


FYI, on my 512 MB RAM, 2-Core test VM using a 'noram' squashfs loop '/' mount, I performed the following after a fresh reboot. Look over the memory use and how little the IO use is...

# free
--
              total        used        free      shared  buff/cache   available
Mem:         500948       82228      334844         280       83876      411136
Swap:             0           0           0
--

# top -d1
--
Mem: 165776K used, 335172K free, 280K shrd, 4648K buff, 79232K cached
CPU:  0.0% usr  0.5% sys  0.0% nic 98.5% idle  0.0% io  0.0% irq  0.0% sirq
--

## As a test, copy about 100 MB of files (/usr/) off squashfs to tmpfs

# mkdir /tmp/disk
# mount -t tmpfs -o size=150m none /tmp/disk
# time ( cp -a /usr/. /tmp/disk/ )
--
real    0m0.783s

(top) 2.4% io, 4.5% io (two 1 second samples)
--

# top -d1
--
Mem: 359280K used, 141668K free, 103400K shrd, 4648K buff, 265492K cached
CPU:  0.4% usr  0.4% sys  0.0% nic 98.5% idle  0.0% io  0.0% irq  0.0% sirq
--

# free
--
              total        used        free      shared  buff/cache   available
Mem:         500948       89140      141668      103400      270140      303776
Swap:             0           0           0
--

## Remove the tmpfs files

# rm -rf /tmp/disk/*

# top -d1
--
Mem: 254220K used, 246728K free, 280K shrd, 4648K buff, 162372K cached
CPU:  0.4% usr  0.4% sys  0.0% nic 99.0% idle  0.0% io  0.0% irq  0.0% sirq
--

# free
--
              total        used        free      shared  buff/cache   available
Mem:         500948       87356      246572         280      167020      408472
Swap:             0           0           0
--

Additionally, the Asterisk sound files are on the ext2/ext4 FUSE unionfs overlay.

My guess is your VM's 250 IOPS setting is fine if that was working well before.


> The results were actually unexpected. The io% was actually LOWER on the system without tmpfs and noticeably higher on ones with tmpfs with 1.3.10 being the worst.
> Now we are not talking high here e.g. occasionally seeing a reading up to 0.2% vs regularly seeing higher, up to 1.6%, but certainly a noticeable difference.

I suspect the newer kernels have better performance, regardless of tmpfs vs squashfs (noram) '/' mount.  The Asterisk version could make a difference as well.


Lonnie




> On Aug 25, 2023, at 8:48 PM, Michael Knill <mic...@ip...> wrote:
>
> Hmm here is an interesting one.
>
> I did some testing using ‘top’ and looking at the ‘io’ percentage as I assumed that disk access would be the most likely to produce an io wait.
>
> While making continuous calls into the system, I watched the io% also trying it on different systems in the same vDC (so basically identical) and different Astlinux versions (1.3.10 & 1.4.7).
>
>
>
> The results were actually unexpected. The io% was actually LOWER on the system without tmpfs and noticeably higher on ones with tmpfs with 1.3.10 being the worst.
>
> Now we are not talking high here e.g. occasionally seeing a reading up to 0.2% vs regularly seeing higher, up to 1.6%, but certainly a noticeable difference.
>
> I also tested on KVM’s in another DC on 1.3.10 with tmpfs for interest and the results were similar to the system without tmpfs e.g. 0.0%-0.2%, but Im pretty sure they are using faster storage.
>
>
>
> Is this a valid test? If so, any reason you can think of why it is better?
>
>
>
> Regards
>
> Michael Knill
>
>
>
>
>
> From: Michael Knill <mic...@ip...>
> Date: Saturday, 26 August 2023 at 10:26 am
> To: AstLinux Developers Mailing List <ast...@li...>
> Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
>
> Just some feedback on this change. There have been no issues detected with very minor variations in memory usage:
>
> <image001.png>
>
>
> My concern at this stage is that I am using fairly slow storage at 250 IOPS and as there is no tmpfs whether this will impact anything?
>
> How would I test this?
>
>
>
> I can push it up to 500 IOPS with relatively minimal expense (much less so than RAM) but I wont bother if I don’t need to.
>
>
>
> Your thoughts would be greatly appreciated.
>
>
>
> Regards
>
> Michael Knill
>
>
>
>
>
> From: Michael Knill <mic...@ip...>
> Date: Friday, 18 August 2023 at 8:04 am
> To: AstLinux Developers Mailing List <ast...@li...>
> Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
>
> Yes that is amazing. I started playing with Astlinux sometime in 2005/6 I believe. Left it for a bit and came back to it and now have over 100 systems out there and looking to expand to much more!
>
>
>
> Thanks for the fix. Pretty sure we have never configured the /etc/inittab file so shouldn’t be an issue.
>
>
>
> So I made some changes to our office system (always ends up being the guinea pig) using noram, dropping out ethernet drivers (I assumed I needed vmxnet3 also as I use VMWare) and took out a few Asterisk modules from modules.conf.
>
>
>
> Its now running with 281 MB, Available 147 MB so around 135M usage. Looking at my monitoring over the last 3 months I only saw a 50M variation so I think the current setup should be fine. We get notified by Zabbix if it reduces below 50M. Lets see how it goes. Even though its only a single processor, I suspect an Intel Xeon Gold 5118 (1x) @ 2294 MHz will be fine.
>
>
>
> PS I have also noticed that it boots up faster. In my build system on Vultr I rebooted the system and could log back in around 45 seconds later. Nice!
>
>
>
> Regards
>
> Michael Knill
>
>
>
>
>
> From: Lonnie Abelbeck <li...@lo...>
> Date: Friday, 18 August 2023 at 1:50 am
> To: AstLinux Developers Mailing List <ast...@li...>
> Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
>
> Cool how after almost 20 years, old AstLinux features become new again :-)
>
> A fix for the "Very Important" warning has been added to master [1].  This would be difficult to fix at the web interface level, other than disabling "Revert to Previous".
>
> I did discover a very minor issue with a "noram" configuration.  If the /etc/inittab file was edited (rarely needed) the edited /etc/inittab is applied to the filesystem after /usr/sbin/chroot is called in the initrd, as such the default base image /etc/inittab is always used.  A custom build for genx86_64-vm could customize the /etc/inittab here [2].  For the tmpfs root filesystem case, the initrd copies-forward the /etc/inittab file to the uncompressed tmpfs image before calling /usr/sbin/chroot.  Again, should not be a big deal for a "noram" configuration.  BTW, before this was fixed for the tmpfs case, AstLinux 1.3.10 through 1.4.2 ignored any user edits to the /etc/inittab file and nobody noticed :-)
>
> Additionally, if you wanted to build a custom genx86_64-vm image with "noram" enabled by default, you would add noram after astlive here [3].
>
> Lonnie
>
> [1] https://github.com/astlinux-project/astlinux/commit/09eed53ddea8f45414195507aeee6d0f8edb9e92
>
> [2] https://github.com/astlinux-project/astlinux/blob/master/project/astlinux/board/genx86_64-vm/target_skeleton/etc/inittab
>
> [3] https://github.com/astlinux-project/astlinux/blob/master/project/astlinux/board/genx86_64-vm/runnix.conf
>
>
>
> > On Aug 17, 2023, at 12:04 AM, Michael Knill <mic...@ip...> wrote:
> >
> > Wow thanks Lonnie that is fantastic and we will test it out. Gosh this will change things considerably. Will be even less once we pull out other modules we don’t use. Having such a lean image will mean that we don’t really need to explore difficult to manage containerisation options such as Docker.
> > Will update the System Tab so that you cannot perform the revert and upgrade as you mentioned.
> >
> > Regards
> > Michael Knill
> >
> >
> > From: Lonnie Abelbeck <li...@lo...>
> > Date: Thursday, 17 August 2023 at 1:59 am
> > To: AstLinux Developers Mailing List <ast...@li...>
> > Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
> >
> >
> > > On Aug 16, 2023, at 3:41 AM, Michael Keuter <li...@mk...> wrote:
> > >
> > >
> > >> Am 16.08.2023 um 05:36 schrieb Michael Knill <mic...@ip...>:
> > >>
> > >> Hi Devs
> > >> Im looking to expand considerably and it will be all in the cloud.
> > >> Our current hosting provider allows us to overcommit CPU resources which works well for us as Astlinux uses nothing however memory is something that cannot be overcommitted and is actually the most expensive of all the resources. As such, I want to have a build that uses the minimal amount of memory I can possibly get away with safely for my small systems. Note this is only for the small systems which have very low usage and its very easy to add memory if required.
> > >> I believe have turned off as much as I can:<image002.png> So is there anything else I can turn off, no matter how small?
> > >> Can I get rid of tmpfs somehow since a read only file system is not necessary?
> > >> My ultimate goal is to have a VM template that I just push a button and its all built and ready to go and a minimal image will make this far more competitive for the mentioned high end hosting. Of course we could just use Vultr which we love but you get what you pay for ☹
> > >> Regards
> > >> Michael Knill
> > >> Managing Director
> > >> D: +61 2 6189 1360
> > >> P: +61 2 6140 4656
> > >> E: mic...@ip...
> > >> W: ipcsolutions.com.au
> > >>  <image001.png>Smarter Business Communications
> > >
> > > Hi Michael,
> > >
> > > I have here an older list of packages, which I disabled (from the defaults) for Alix boards (at that time):
> > >
> > > Asterisk: dahdi_echocan_oslec, prosody off
> > > Asterisk: dahdi_no_card_firmware on
> > > Hardware handling: monit, zabbix_proxy, zabbix off
> > > Libraries/Networking: pjsip, 'Asterisk PRI Support' off
> > > Networking apps: keepalived, netsnmp, openldap-server, strongswan off
> > >
> > > These were all packages that were not needed on those boxes.
> > >
> > > Michael
> >
> > Micheal Keuter brings up a good point.  Though disabling packages in the build system may not be required.
> >
> > In AstLinux, memory usage has two elements:
> >
> > 1) Standard application and kernel RAM usage.
> > 1a) Reducing memory usage can be achieved by only starting needed services and tuning the /etc/rc.modules to comment-out unused NIC modules.
> >
> > 2) The Read-Only root filesystem "/" is by default mounted uncompressed using tmpfs.
> > 2a) Reducing memory usage can be achieved by adding "noram" to the KCMD of the /oldroot/cdrom/os/*run.conf file(s).  In the "noram" case the root filesystem "/" is mounted compressed using squashfs, via /dev/loop0.
> >
> > First, a little history.
> >
> > Back in the day when CompactFlash (CF) storage was the norm and boards contained 256 MB or more RAM, the Read-Only root filesystem "/" was copied to RAM (tmpfs).  This served two purposes,
> > 1) no realtime uncompressing of the squashfs run image noticeably reduced file access latency and speed on single core 500 Mhz (or less) CPUs
> > 2) root filesystem access was much faster using RAM (tmpfs) rather than a loop mount directly off CF storage.
> >
> > In later years, while the advantages of the tmpfs root filesystem were less pronounced as SATA storage appeared and multi-core CPUs, there are still advantages today to make a tmpfs root filesystem still the default.  Eliminating the loop mount directly off the storage is probably the main advantage today (KISS), and systems with 1 GB RAM or more there is no practical RAM disadvantage.
> >
> > As per Michael Knill's use case above, there may be a legitimate case for adding "noram" in the KCMD of the /oldroot/cdrom/os/*run.conf file(s).
> >
> > Demonstrate via an example.
> >
> > Using the latest pre-release [1] Guest VM x86-64bit ISO [2] with Proxmox 8, standard build using Asterisk 20.4.0 with pjsip loaded.
> >
> > The Proxmox VM is configured with 2-cores and 512 MB RAM
> >
> > Default case:
> > Proxmox: Memory usage 38.13% (195.24 MiB of 512.00 MiB)
> > --
> >       Host Name: pbx.astlinux
> >           Linux: 5.10.179-astlinux x86_64
> >             CPU: Common KVM processor (2x) @ 2096 MHz
> >             RAM: 489 MB, Available 253 MB
> >      Board Type: genx86_64-vm
> >        Hardware: KVM Hypervisor Guest VM
> > --
> > none on / type tmpfs (ro,relatime,size=134128k)
> > none                    143.4M    143.4M         0 100% /
> >
> > Note that 143.4 MB of RAM is used for the root filesystem.
> >
> > Next, add "noram" to the KCMD= line ...
> > --
> > # mount -o rw,remount /oldroot/cdrom
> >
> > Edit the KCMD= line by adding noram after astlive
> > # vi /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
> > After the change, it should show...
> > # grep '^KCMD=' /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
> > KCMD="root=/dev/ram0 rw init=/linuxrc astlinux=genx86_64-vm astimg=astlinux-1.5-5855-4d7596.run astkd=auto asturw=auto astlive noram libata.dma=3"
> >
> > # mount -o ro,remount /oldroot/cdrom
> > --
> >
> > Very Important ->  With "noram" in the KCMD, never perform a firmware "Revert to Previous" followed by an "Upgrade with New" without rebooting immediately after the firmware revert.
> >
> > "noram" case:
> > Proxmox: Memory usage 19.81% (101.42 MiB of 512.00 MiB)
> > --
> >       Host Name: pbx.astlinux
> >           Linux: 5.10.179-astlinux x86_64
> >             CPU: Common KVM processor (2x) @ 2096 MHz
> >             RAM: 489 MB, Available 400 MB
> >      Board Type: genx86_64-vm
> >        Hardware: KVM Hypervisor Guest VM
> > --
> > /dev/loop0 on / type squashfs (ro,relatime)
> > /dev/loop0               51.1M     51.1M         0 100% /
> >
> > Note that no extra RAM is used, the 51.1 MB is the compressed squashfs run image directly mounted off storage.
> >
> > Interestingly, the AstLinux 400 MB vs. 253 MB Available RAM closely matches the tmpfs root filesystem size, but the Proxmox Memory usage only dropped to 101 MB vs. 195 MB.
> >
> > Additional RAM (albeit minor) RAM reduction
> > Disable unused NIC modules (Proxmox example)
> > --
> > pbx ~ # cat /etc/rc.modules
> > ## These modules get modprobe'd when the system starts up.
> > ##
> > ## Comment out any modules you don't need.
> > ##
> > ## Ethernet support
> > ## Virtual first, then Emulated
> > virtio_net
> > #vmxnet3
> > #pcnet32
> > #e1000
> > #e1000e
> > #8139too
> > --
> >
> > In summary, setting "noram" in the /oldroot/cdrom/os/*run.conf KCMD should be a production-worthy option for special low-RAM VM situations, but since it has not been used by default, testing should be performed.  Don't forget the "Very Important" notice above.
> >
> > Also note that once "noram" in the /oldroot/cdrom/os/*run.conf KCMD is defined, it is automatically carried forward on future upgrades.
> >
> > Lonnie
> >
> > [1] https://www.astlinux-project.org/dev.html
> >
> > [2] https://astlinux-project.org/beta/iso/astlinux-beta-genx86_64-vm.iso
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > Astlinux-devel mailing list
> > Ast...@li...
> > https://lists.sourceforge.net/lists/listinfo/astlinux-devel
> > _______________________________________________
> > Astlinux-devel mailing list
> > Ast...@li...
> > https://lists.sourceforge.net/lists/listinfo/astlinux-devel
>
>
>
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel
>
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel



_______________________________________________
Astlinux-devel mailing list
Ast...@li...
https://lists.sourceforge.net/lists/listinfo/astlinux-devel

Re: [Astlinux-devel] Reducing memory usage as much as possible

[Lonnie A. <li...@lo...>]

Hi Michael,

My guess is the 'noram' squashfs loop mount does not add any significant IO/CPU usage.

In particular, our squashfs compression is the simplest (gzip) and the kernel caches any accessed files.  The storage (virtio_scsi) is quite fast in a VM.

I can't suggest any good test metrics, as most everything of interest (squashfs, virtio_scsi and cache) runs in the kernel.


FYI, on my 512 MB RAM, 2-Core test VM using a 'noram' squashfs loop '/' mount, I performed the following after a fresh reboot. Look over the memory use and how little the IO use is...

# free
--
              total        used        free      shared  buff/cache   available
Mem:         500948       82228      334844         280       83876      411136
Swap:             0           0           0
--

# top -d1
--
Mem: 165776K used, 335172K free, 280K shrd, 4648K buff, 79232K cached
CPU:  0.0% usr  0.5% sys  0.0% nic 98.5% idle  0.0% io  0.0% irq  0.0% sirq
--

## As a test, copy about 100 MB of files (/usr/) off squashfs to tmpfs

# mkdir /tmp/disk
# mount -t tmpfs -o size=150m none /tmp/disk
# time ( cp -a /usr/. /tmp/disk/ )
--
real	0m0.783s

(top) 2.4% io, 4.5% io (two 1 second samples)
--

# top -d1
--
Mem: 359280K used, 141668K free, 103400K shrd, 4648K buff, 265492K cached
CPU:  0.4% usr  0.4% sys  0.0% nic 98.5% idle  0.0% io  0.0% irq  0.0% sirq
--

# free
--
              total        used        free      shared  buff/cache   available
Mem:         500948       89140      141668      103400      270140      303776
Swap:             0           0           0
--

## Remove the tmpfs files

# rm -rf /tmp/disk/*

# top -d1
--
Mem: 254220K used, 246728K free, 280K shrd, 4648K buff, 162372K cached
CPU:  0.4% usr  0.4% sys  0.0% nic 99.0% idle  0.0% io  0.0% irq  0.0% sirq
--

# free
--
              total        used        free      shared  buff/cache   available
Mem:         500948       87356      246572         280      167020      408472
Swap:             0           0           0
--

Additionally, the Asterisk sound files are on the ext2/ext4 FUSE unionfs overlay.

My guess is your VM's 250 IOPS setting is fine if that was working well before.


> The results were actually unexpected. The io% was actually LOWER on the system without tmpfs and noticeably higher on ones with tmpfs with 1.3.10 being the worst.
> Now we are not talking high here e.g. occasionally seeing a reading up to 0.2% vs regularly seeing higher, up to 1.6%, but certainly a noticeable difference.

I suspect the newer kernels have better performance, regardless of tmpfs vs squashfs (noram) '/' mount.  The Asterisk version could make a difference as well.


Lonnie




> On Aug 25, 2023, at 8:48 PM, Michael Knill <mic...@ip...> wrote:
> 
> Hmm here is an interesting one.
> 
> I did some testing using ‘top’ and looking at the ‘io’ percentage as I assumed that disk access would be the most likely to produce an io wait.
> 
> While making continuous calls into the system, I watched the io% also trying it on different systems in the same vDC (so basically identical) and different Astlinux versions (1.3.10 & 1.4.7).
> 
>  
> 
> The results were actually unexpected. The io% was actually LOWER on the system without tmpfs and noticeably higher on ones with tmpfs with 1.3.10 being the worst.
> 
> Now we are not talking high here e.g. occasionally seeing a reading up to 0.2% vs regularly seeing higher, up to 1.6%, but certainly a noticeable difference.
> 
> I also tested on KVM’s in another DC on 1.3.10 with tmpfs for interest and the results were similar to the system without tmpfs e.g. 0.0%-0.2%, but Im pretty sure they are using faster storage.
> 
>  
> 
> Is this a valid test? If so, any reason you can think of why it is better?
> 
>  
> 
> Regards
> 
> Michael Knill
> 
>  
> 
>  
> 
> From: Michael Knill <mic...@ip...>
> Date: Saturday, 26 August 2023 at 10:26 am
> To: AstLinux Developers Mailing List <ast...@li...>
> Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
> 
> Just some feedback on this change. There have been no issues detected with very minor variations in memory usage:
> 
> <image001.png>
>  
> 
> My concern at this stage is that I am using fairly slow storage at 250 IOPS and as there is no tmpfs whether this will impact anything?
> 
> How would I test this?
> 
>  
> 
> I can push it up to 500 IOPS with relatively minimal expense (much less so than RAM) but I wont bother if I don’t need to.
> 
>  
> 
> Your thoughts would be greatly appreciated.
> 
>  
> 
> Regards
> 
> Michael Knill
> 
>  
> 
>  
> 
> From: Michael Knill <mic...@ip...>
> Date: Friday, 18 August 2023 at 8:04 am
> To: AstLinux Developers Mailing List <ast...@li...>
> Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
> 
> Yes that is amazing. I started playing with Astlinux sometime in 2005/6 I believe. Left it for a bit and came back to it and now have over 100 systems out there and looking to expand to much more!
> 
>  
> 
> Thanks for the fix. Pretty sure we have never configured the /etc/inittab file so shouldn’t be an issue.
> 
>  
> 
> So I made some changes to our office system (always ends up being the guinea pig) using noram, dropping out ethernet drivers (I assumed I needed vmxnet3 also as I use VMWare) and took out a few Asterisk modules from modules.conf.
> 
>  
> 
> Its now running with 281 MB, Available 147 MB so around 135M usage. Looking at my monitoring over the last 3 months I only saw a 50M variation so I think the current setup should be fine. We get notified by Zabbix if it reduces below 50M. Lets see how it goes. Even though its only a single processor, I suspect an Intel Xeon Gold 5118 (1x) @ 2294 MHz will be fine.
> 
>  
> 
> PS I have also noticed that it boots up faster. In my build system on Vultr I rebooted the system and could log back in around 45 seconds later. Nice!
> 
>  
> 
> Regards
> 
> Michael Knill
> 
>  
> 
>  
> 
> From: Lonnie Abelbeck <li...@lo...>
> Date: Friday, 18 August 2023 at 1:50 am
> To: AstLinux Developers Mailing List <ast...@li...>
> Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
> 
> Cool how after almost 20 years, old AstLinux features become new again :-)
> 
> A fix for the "Very Important" warning has been added to master [1].  This would be difficult to fix at the web interface level, other than disabling "Revert to Previous".
> 
> I did discover a very minor issue with a "noram" configuration.  If the /etc/inittab file was edited (rarely needed) the edited /etc/inittab is applied to the filesystem after /usr/sbin/chroot is called in the initrd, as such the default base image /etc/inittab is always used.  A custom build for genx86_64-vm could customize the /etc/inittab here [2].  For the tmpfs root filesystem case, the initrd copies-forward the /etc/inittab file to the uncompressed tmpfs image before calling /usr/sbin/chroot.  Again, should not be a big deal for a "noram" configuration.  BTW, before this was fixed for the tmpfs case, AstLinux 1.3.10 through 1.4.2 ignored any user edits to the /etc/inittab file and nobody noticed :-)
> 
> Additionally, if you wanted to build a custom genx86_64-vm image with "noram" enabled by default, you would add noram after astlive here [3].
> 
> Lonnie
> 
> [1] https://github.com/astlinux-project/astlinux/commit/09eed53ddea8f45414195507aeee6d0f8edb9e92
> 
> [2] https://github.com/astlinux-project/astlinux/blob/master/project/astlinux/board/genx86_64-vm/target_skeleton/etc/inittab
> 
> [3] https://github.com/astlinux-project/astlinux/blob/master/project/astlinux/board/genx86_64-vm/runnix.conf
> 
> 
> 
> > On Aug 17, 2023, at 12:04 AM, Michael Knill <mic...@ip...> wrote:
> > 
> > Wow thanks Lonnie that is fantastic and we will test it out. Gosh this will change things considerably. Will be even less once we pull out other modules we don’t use. Having such a lean image will mean that we don’t really need to explore difficult to manage containerisation options such as Docker.
> > Will update the System Tab so that you cannot perform the revert and upgrade as you mentioned.
> >  
> > Regards
> > Michael Knill
> >  
> >  
> > From: Lonnie Abelbeck <li...@lo...>
> > Date: Thursday, 17 August 2023 at 1:59 am
> > To: AstLinux Developers Mailing List <ast...@li...>
> > Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
> > 
> > 
> > > On Aug 16, 2023, at 3:41 AM, Michael Keuter <li...@mk...> wrote:
> > > 
> > > 
> > >> Am 16.08.2023 um 05:36 schrieb Michael Knill <mic...@ip...>:
> > >> 
> > >> Hi Devs
> > >> Im looking to expand considerably and it will be all in the cloud.
> > >> Our current hosting provider allows us to overcommit CPU resources which works well for us as Astlinux uses nothing however memory is something that cannot be overcommitted and is actually the most expensive of all the resources. As such, I want to have a build that uses the minimal amount of memory I can possibly get away with safely for my small systems. Note this is only for the small systems which have very low usage and its very easy to add memory if required.
> > >> I believe have turned off as much as I can:<image002.png> So is there anything else I can turn off, no matter how small?
> > >> Can I get rid of tmpfs somehow since a read only file system is not necessary?
> > >> My ultimate goal is to have a VM template that I just push a button and its all built and ready to go and a minimal image will make this far more competitive for the mentioned high end hosting. Of course we could just use Vultr which we love but you get what you pay for ☹
> > >> Regards
> > >> Michael Knill
> > >> Managing Director
> > >> D: +61 2 6189 1360
> > >> P: +61 2 6140 4656
> > >> E: mic...@ip...
> > >> W: ipcsolutions.com.au
> > >>  <image001.png>Smarter Business Communications
> > > 
> > > Hi Michael,
> > > 
> > > I have here an older list of packages, which I disabled (from the defaults) for Alix boards (at that time):
> > > 
> > > Asterisk: dahdi_echocan_oslec, prosody off
> > > Asterisk: dahdi_no_card_firmware on
> > > Hardware handling: monit, zabbix_proxy, zabbix off
> > > Libraries/Networking: pjsip, 'Asterisk PRI Support' off
> > > Networking apps: keepalived, netsnmp, openldap-server, strongswan off
> > > 
> > > These were all packages that were not needed on those boxes.
> > > 
> > > Michael
> > 
> > Micheal Keuter brings up a good point.  Though disabling packages in the build system may not be required.
> > 
> > In AstLinux, memory usage has two elements:
> > 
> > 1) Standard application and kernel RAM usage.
> > 1a) Reducing memory usage can be achieved by only starting needed services and tuning the /etc/rc.modules to comment-out unused NIC modules.
> > 
> > 2) The Read-Only root filesystem "/" is by default mounted uncompressed using tmpfs.
> > 2a) Reducing memory usage can be achieved by adding "noram" to the KCMD of the /oldroot/cdrom/os/*run.conf file(s).  In the "noram" case the root filesystem "/" is mounted compressed using squashfs, via /dev/loop0.
> > 
> > First, a little history.
> > 
> > Back in the day when CompactFlash (CF) storage was the norm and boards contained 256 MB or more RAM, the Read-Only root filesystem "/" was copied to RAM (tmpfs).  This served two purposes,
> > 1) no realtime uncompressing of the squashfs run image noticeably reduced file access latency and speed on single core 500 Mhz (or less) CPUs
> > 2) root filesystem access was much faster using RAM (tmpfs) rather than a loop mount directly off CF storage.
> > 
> > In later years, while the advantages of the tmpfs root filesystem were less pronounced as SATA storage appeared and multi-core CPUs, there are still advantages today to make a tmpfs root filesystem still the default.  Eliminating the loop mount directly off the storage is probably the main advantage today (KISS), and systems with 1 GB RAM or more there is no practical RAM disadvantage.
> > 
> > As per Michael Knill's use case above, there may be a legitimate case for adding "noram" in the KCMD of the /oldroot/cdrom/os/*run.conf file(s).
> > 
> > Demonstrate via an example.
> > 
> > Using the latest pre-release [1] Guest VM x86-64bit ISO [2] with Proxmox 8, standard build using Asterisk 20.4.0 with pjsip loaded.
> > 
> > The Proxmox VM is configured with 2-cores and 512 MB RAM
> > 
> > Default case:
> > Proxmox: Memory usage 38.13% (195.24 MiB of 512.00 MiB)
> > --
> >       Host Name: pbx.astlinux
> >           Linux: 5.10.179-astlinux x86_64
> >             CPU: Common KVM processor (2x) @ 2096 MHz
> >             RAM: 489 MB, Available 253 MB
> >      Board Type: genx86_64-vm
> >        Hardware: KVM Hypervisor Guest VM
> > --
> > none on / type tmpfs (ro,relatime,size=134128k)
> > none                    143.4M    143.4M         0 100% /
> > 
> > Note that 143.4 MB of RAM is used for the root filesystem.
> > 
> > Next, add "noram" to the KCMD= line ...
> > --
> > # mount -o rw,remount /oldroot/cdrom
> > 
> > Edit the KCMD= line by adding noram after astlive
> > # vi /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
> > After the change, it should show...
> > # grep '^KCMD=' /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
> > KCMD="root=/dev/ram0 rw init=/linuxrc astlinux=genx86_64-vm astimg=astlinux-1.5-5855-4d7596.run astkd=auto asturw=auto astlive noram libata.dma=3"
> > 
> > # mount -o ro,remount /oldroot/cdrom
> > --
> > 
> > Very Important ->  With "noram" in the KCMD, never perform a firmware "Revert to Previous" followed by an "Upgrade with New" without rebooting immediately after the firmware revert.
> > 
> > "noram" case:
> > Proxmox: Memory usage 19.81% (101.42 MiB of 512.00 MiB)
> > --
> >       Host Name: pbx.astlinux
> >           Linux: 5.10.179-astlinux x86_64
> >             CPU: Common KVM processor (2x) @ 2096 MHz
> >             RAM: 489 MB, Available 400 MB
> >      Board Type: genx86_64-vm
> >        Hardware: KVM Hypervisor Guest VM
> > --
> > /dev/loop0 on / type squashfs (ro,relatime)
> > /dev/loop0               51.1M     51.1M         0 100% /
> > 
> > Note that no extra RAM is used, the 51.1 MB is the compressed squashfs run image directly mounted off storage.
> > 
> > Interestingly, the AstLinux 400 MB vs. 253 MB Available RAM closely matches the tmpfs root filesystem size, but the Proxmox Memory usage only dropped to 101 MB vs. 195 MB.
> > 
> > Additional RAM (albeit minor) RAM reduction
> > Disable unused NIC modules (Proxmox example)
> > --
> > pbx ~ # cat /etc/rc.modules 
> > ## These modules get modprobe'd when the system starts up.
> > ##
> > ## Comment out any modules you don't need.
> > ##
> > ## Ethernet support
> > ## Virtual first, then Emulated
> > virtio_net
> > #vmxnet3
> > #pcnet32
> > #e1000
> > #e1000e
> > #8139too
> > --
> > 
> > In summary, setting "noram" in the /oldroot/cdrom/os/*run.conf KCMD should be a production-worthy option for special low-RAM VM situations, but since it has not been used by default, testing should be performed.  Don't forget the "Very Important" notice above.
> > 
> > Also note that once "noram" in the /oldroot/cdrom/os/*run.conf KCMD is defined, it is automatically carried forward on future upgrades.
> > 
> > Lonnie
> > 
> > [1] https://www.astlinux-project.org/dev.html
> > 
> > [2] https://astlinux-project.org/beta/iso/astlinux-beta-genx86_64-vm.iso
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > _______________________________________________
> > Astlinux-devel mailing list
> > Ast...@li...
> > https://lists.sourceforge.net/lists/listinfo/astlinux-devel
> > _______________________________________________
> > Astlinux-devel mailing list
> > Ast...@li...
> > https://lists.sourceforge.net/lists/listinfo/astlinux-devel
> 
> 
> 
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel
> 
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel

Re: [Astlinux-devel] Reducing memory usage as much as possible

[Michael K. <mic...@ip...>]

Hmm here is an interesting one.
I did some testing using ‘top’ and looking at the ‘io’ percentage as I assumed that disk access would be the most likely to produce an io wait.
While making continuous calls into the system, I watched the io% also trying it on different systems in the same vDC (so basically identical) and different Astlinux versions (1.3.10 & 1.4.7).

The results were actually unexpected. The io% was actually LOWER on the system without tmpfs and noticeably higher on ones with tmpfs with 1.3.10 being the worst.
Now we are not talking high here e.g. occasionally seeing a reading up to 0.2% vs regularly seeing higher, up to 1.6%, but certainly a noticeable difference.
I also tested on KVM’s in another DC on 1.3.10 with tmpfs for interest and the results were similar to the system without tmpfs e.g. 0.0%-0.2%, but Im pretty sure they are using faster storage.

Is this a valid test? If so, any reason you can think of why it is better?

Regards
Michael Knill


From: Michael Knill <mic...@ip...>
Date: Saturday, 26 August 2023 at 10:26 am
To: AstLinux Developers Mailing List <ast...@li...>
Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
Just some feedback on this change. There have been no issues detected with very minor variations in memory usage:
[cid:image001.png@01D9D7FC.4F657580]

My concern at this stage is that I am using fairly slow storage at 250 IOPS and as there is no tmpfs whether this will impact anything?
How would I test this?

I can push it up to 500 IOPS with relatively minimal expense (much less so than RAM) but I wont bother if I don’t need to.

Your thoughts would be greatly appreciated.

Regards
Michael Knill


From: Michael Knill <mic...@ip...>
Date: Friday, 18 August 2023 at 8:04 am
To: AstLinux Developers Mailing List <ast...@li...>
Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
Yes that is amazing. I started playing with Astlinux sometime in 2005/6 I believe. Left it for a bit and came back to it and now have over 100 systems out there and looking to expand to much more!

Thanks for the fix. Pretty sure we have never configured the /etc/inittab file so shouldn’t be an issue.

So I made some changes to our office system (always ends up being the guinea pig) using noram, dropping out ethernet drivers (I assumed I needed vmxnet3 also as I use VMWare) and took out a few Asterisk modules from modules.conf.

Its now running with 281 MB, Available 147 MB so around 135M usage. Looking at my monitoring over the last 3 months I only saw a 50M variation so I think the current setup should be fine. We get notified by Zabbix if it reduces below 50M. Lets see how it goes. Even though its only a single processor, I suspect an Intel Xeon Gold 5118 (1x) @ 2294 MHz will be fine.

PS I have also noticed that it boots up faster. In my build system on Vultr I rebooted the system and could log back in around 45 seconds later. Nice!

Regards
Michael Knill


From: Lonnie Abelbeck <li...@lo...>
Date: Friday, 18 August 2023 at 1:50 am
To: AstLinux Developers Mailing List <ast...@li...>
Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
Cool how after almost 20 years, old AstLinux features become new again :-)

A fix for the "Very Important" warning has been added to master [1].  This would be difficult to fix at the web interface level, other than disabling "Revert to Previous".

I did discover a very minor issue with a "noram" configuration.  If the /etc/inittab file was edited (rarely needed) the edited /etc/inittab is applied to the filesystem after /usr/sbin/chroot is called in the initrd, as such the default base image /etc/inittab is always used.  A custom build for genx86_64-vm could customize the /etc/inittab here [2].  For the tmpfs root filesystem case, the initrd copies-forward the /etc/inittab file to the uncompressed tmpfs image before calling /usr/sbin/chroot.  Again, should not be a big deal for a "noram" configuration.  BTW, before this was fixed for the tmpfs case, AstLinux 1.3.10 through 1.4.2 ignored any user edits to the /etc/inittab file and nobody noticed :-)

Additionally, if you wanted to build a custom genx86_64-vm image with "noram" enabled by default, you would add noram after astlive here [3].

Lonnie

[1] https://github.com/astlinux-project/astlinux/commit/09eed53ddea8f45414195507aeee6d0f8edb9e92

[2] https://github.com/astlinux-project/astlinux/blob/master/project/astlinux/board/genx86_64-vm/target_skeleton/etc/inittab

[3] https://github.com/astlinux-project/astlinux/blob/master/project/astlinux/board/genx86_64-vm/runnix.conf



> On Aug 17, 2023, at 12:04 AM, Michael Knill <mic...@ip...> wrote:
>
> Wow thanks Lonnie that is fantastic and we will test it out. Gosh this will change things considerably. Will be even less once we pull out other modules we don’t use. Having such a lean image will mean that we don’t really need to explore difficult to manage containerisation options such as Docker.
> Will update the System Tab so that you cannot perform the revert and upgrade as you mentioned.
>
> Regards
> Michael Knill
>
>
> From: Lonnie Abelbeck <li...@lo...>
> Date: Thursday, 17 August 2023 at 1:59 am
> To: AstLinux Developers Mailing List <ast...@li...>
> Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
>
>
> > On Aug 16, 2023, at 3:41 AM, Michael Keuter <li...@mk...> wrote:
> >
> >
> >> Am 16.08.2023 um 05:36 schrieb Michael Knill <mic...@ip...>:
> >>
> >> Hi Devs
> >> Im looking to expand considerably and it will be all in the cloud.
> >> Our current hosting provider allows us to overcommit CPU resources which works well for us as Astlinux uses nothing however memory is something that cannot be overcommitted and is actually the most expensive of all the resources. As such, I want to have a build that uses the minimal amount of memory I can possibly get away with safely for my small systems. Note this is only for the small systems which have very low usage and its very easy to add memory if required.
> >> I believe have turned off as much as I can:<image002.png> So is there anything else I can turn off, no matter how small?
> >> Can I get rid of tmpfs somehow since a read only file system is not necessary?
> >> My ultimate goal is to have a VM template that I just push a button and its all built and ready to go and a minimal image will make this far more competitive for the mentioned high end hosting. Of course we could just use Vultr which we love but you get what you pay for ☹
> >> Regards
> >> Michael Knill
> >> Managing Director
> >> D: +61 2 6189 1360
> >> P: +61 2 6140 4656
> >> E: mic...@ip...
> >> W: ipcsolutions.com.au
> >>  <image001.png>Smarter Business Communications
> >
> > Hi Michael,
> >
> > I have here an older list of packages, which I disabled (from the defaults) for Alix boards (at that time):
> >
> > Asterisk: dahdi_echocan_oslec, prosody off
> > Asterisk: dahdi_no_card_firmware on
> > Hardware handling: monit, zabbix_proxy, zabbix off
> > Libraries/Networking: pjsip, 'Asterisk PRI Support' off
> > Networking apps: keepalived, netsnmp, openldap-server, strongswan off
> >
> > These were all packages that were not needed on those boxes.
> >
> > Michael
>
> Micheal Keuter brings up a good point.  Though disabling packages in the build system may not be required.
>
> In AstLinux, memory usage has two elements:
>
> 1) Standard application and kernel RAM usage.
> 1a) Reducing memory usage can be achieved by only starting needed services and tuning the /etc/rc.modules to comment-out unused NIC modules.
>
> 2) The Read-Only root filesystem "/" is by default mounted uncompressed using tmpfs.
> 2a) Reducing memory usage can be achieved by adding "noram" to the KCMD of the /oldroot/cdrom/os/*run.conf file(s).  In the "noram" case the root filesystem "/" is mounted compressed using squashfs, via /dev/loop0.
>
> First, a little history.
>
> Back in the day when CompactFlash (CF) storage was the norm and boards contained 256 MB or more RAM, the Read-Only root filesystem "/" was copied to RAM (tmpfs).  This served two purposes,
> 1) no realtime uncompressing of the squashfs run image noticeably reduced file access latency and speed on single core 500 Mhz (or less) CPUs
> 2) root filesystem access was much faster using RAM (tmpfs) rather than a loop mount directly off CF storage.
>
> In later years, while the advantages of the tmpfs root filesystem were less pronounced as SATA storage appeared and multi-core CPUs, there are still advantages today to make a tmpfs root filesystem still the default.  Eliminating the loop mount directly off the storage is probably the main advantage today (KISS), and systems with 1 GB RAM or more there is no practical RAM disadvantage.
>
> As per Michael Knill's use case above, there may be a legitimate case for adding "noram" in the KCMD of the /oldroot/cdrom/os/*run.conf file(s).
>
> Demonstrate via an example.
>
> Using the latest pre-release [1] Guest VM x86-64bit ISO [2] with Proxmox 8, standard build using Asterisk 20.4.0 with pjsip loaded.
>
> The Proxmox VM is configured with 2-cores and 512 MB RAM
>
> Default case:
> Proxmox: Memory usage 38.13% (195.24 MiB of 512.00 MiB)
> --
>       Host Name: pbx.astlinux
>           Linux: 5.10.179-astlinux x86_64
>             CPU: Common KVM processor (2x) @ 2096 MHz
>             RAM: 489 MB, Available 253 MB
>      Board Type: genx86_64-vm
>        Hardware: KVM Hypervisor Guest VM
> --
> none on / type tmpfs (ro,relatime,size=134128k)
> none                    143.4M    143.4M         0 100% /
>
> Note that 143.4 MB of RAM is used for the root filesystem.
>
> Next, add "noram" to the KCMD= line ...
> --
> # mount -o rw,remount /oldroot/cdrom
>
> Edit the KCMD= line by adding noram after astlive
> # vi /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
> After the change, it should show...
> # grep '^KCMD=' /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
> KCMD="root=/dev/ram0 rw init=/linuxrc astlinux=genx86_64-vm astimg=astlinux-1.5-5855-4d7596.run astkd=auto asturw=auto astlive noram libata.dma=3"
>
> # mount -o ro,remount /oldroot/cdrom
> --
>
> Very Important ->  With "noram" in the KCMD, never perform a firmware "Revert to Previous" followed by an "Upgrade with New" without rebooting immediately after the firmware revert.
>
> "noram" case:
> Proxmox: Memory usage 19.81% (101.42 MiB of 512.00 MiB)
> --
>       Host Name: pbx.astlinux
>           Linux: 5.10.179-astlinux x86_64
>             CPU: Common KVM processor (2x) @ 2096 MHz
>             RAM: 489 MB, Available 400 MB
>      Board Type: genx86_64-vm
>        Hardware: KVM Hypervisor Guest VM
> --
> /dev/loop0 on / type squashfs (ro,relatime)
> /dev/loop0               51.1M     51.1M         0 100% /
>
> Note that no extra RAM is used, the 51.1 MB is the compressed squashfs run image directly mounted off storage.
>
> Interestingly, the AstLinux 400 MB vs. 253 MB Available RAM closely matches the tmpfs root filesystem size, but the Proxmox Memory usage only dropped to 101 MB vs. 195 MB.
>
> Additional RAM (albeit minor) RAM reduction
> Disable unused NIC modules (Proxmox example)
> --
> pbx ~ # cat /etc/rc.modules
> ## These modules get modprobe'd when the system starts up.
> ##
> ## Comment out any modules you don't need.
> ##
> ## Ethernet support
> ## Virtual first, then Emulated
> virtio_net
> #vmxnet3
> #pcnet32
> #e1000
> #e1000e
> #8139too
> --
>
> In summary, setting "noram" in the /oldroot/cdrom/os/*run.conf KCMD should be a production-worthy option for special low-RAM VM situations, but since it has not been used by default, testing should be performed.  Don't forget the "Very Important" notice above.
>
> Also note that once "noram" in the /oldroot/cdrom/os/*run.conf KCMD is defined, it is automatically carried forward on future upgrades.
>
> Lonnie
>
> [1] https://www.astlinux-project.org/dev.html
>
> [2] https://astlinux-project.org/beta/iso/astlinux-beta-genx86_64-vm.iso
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel



_______________________________________________
Astlinux-devel mailing list
Ast...@li...
https://lists.sourceforge.net/lists/listinfo/astlinux-devel

Re: [Astlinux-devel] Reducing memory usage as much as possible

[Michael K. <mic...@ip...>]

Just some feedback on this change. There have been no issues detected with very minor variations in memory usage:
[cid:image001.png@01D9D7FC.4F657580]

My concern at this stage is that I am using fairly slow storage at 250 IOPS and as there is no tmpfs whether this will impact anything?
How would I test this?

I can push it up to 500 IOPS with relatively minimal expense (much less so than RAM) but I wont bother if I don’t need to.

Your thoughts would be greatly appreciated.

Regards
Michael Knill


From: Michael Knill <mic...@ip...>
Date: Friday, 18 August 2023 at 8:04 am
To: AstLinux Developers Mailing List <ast...@li...>
Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
Yes that is amazing. I started playing with Astlinux sometime in 2005/6 I believe. Left it for a bit and came back to it and now have over 100 systems out there and looking to expand to much more!

Thanks for the fix. Pretty sure we have never configured the /etc/inittab file so shouldn’t be an issue.

So I made some changes to our office system (always ends up being the guinea pig) using noram, dropping out ethernet drivers (I assumed I needed vmxnet3 also as I use VMWare) and took out a few Asterisk modules from modules.conf.

Its now running with 281 MB, Available 147 MB so around 135M usage. Looking at my monitoring over the last 3 months I only saw a 50M variation so I think the current setup should be fine. We get notified by Zabbix if it reduces below 50M. Lets see how it goes. Even though its only a single processor, I suspect an Intel Xeon Gold 5118 (1x) @ 2294 MHz will be fine.

PS I have also noticed that it boots up faster. In my build system on Vultr I rebooted the system and could log back in around 45 seconds later. Nice!

Regards
Michael Knill


From: Lonnie Abelbeck <li...@lo...>
Date: Friday, 18 August 2023 at 1:50 am
To: AstLinux Developers Mailing List <ast...@li...>
Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
Cool how after almost 20 years, old AstLinux features become new again :-)

A fix for the "Very Important" warning has been added to master [1].  This would be difficult to fix at the web interface level, other than disabling "Revert to Previous".

I did discover a very minor issue with a "noram" configuration.  If the /etc/inittab file was edited (rarely needed) the edited /etc/inittab is applied to the filesystem after /usr/sbin/chroot is called in the initrd, as such the default base image /etc/inittab is always used.  A custom build for genx86_64-vm could customize the /etc/inittab here [2].  For the tmpfs root filesystem case, the initrd copies-forward the /etc/inittab file to the uncompressed tmpfs image before calling /usr/sbin/chroot.  Again, should not be a big deal for a "noram" configuration.  BTW, before this was fixed for the tmpfs case, AstLinux 1.3.10 through 1.4.2 ignored any user edits to the /etc/inittab file and nobody noticed :-)

Additionally, if you wanted to build a custom genx86_64-vm image with "noram" enabled by default, you would add noram after astlive here [3].

Lonnie

[1] https://github.com/astlinux-project/astlinux/commit/09eed53ddea8f45414195507aeee6d0f8edb9e92

[2] https://github.com/astlinux-project/astlinux/blob/master/project/astlinux/board/genx86_64-vm/target_skeleton/etc/inittab

[3] https://github.com/astlinux-project/astlinux/blob/master/project/astlinux/board/genx86_64-vm/runnix.conf



> On Aug 17, 2023, at 12:04 AM, Michael Knill <mic...@ip...> wrote:
>
> Wow thanks Lonnie that is fantastic and we will test it out. Gosh this will change things considerably. Will be even less once we pull out other modules we don’t use. Having such a lean image will mean that we don’t really need to explore difficult to manage containerisation options such as Docker.
> Will update the System Tab so that you cannot perform the revert and upgrade as you mentioned.
>
> Regards
> Michael Knill
>
>
> From: Lonnie Abelbeck <li...@lo...>
> Date: Thursday, 17 August 2023 at 1:59 am
> To: AstLinux Developers Mailing List <ast...@li...>
> Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
>
>
> > On Aug 16, 2023, at 3:41 AM, Michael Keuter <li...@mk...> wrote:
> >
> >
> >> Am 16.08.2023 um 05:36 schrieb Michael Knill <mic...@ip...>:
> >>
> >> Hi Devs
> >> Im looking to expand considerably and it will be all in the cloud.
> >> Our current hosting provider allows us to overcommit CPU resources which works well for us as Astlinux uses nothing however memory is something that cannot be overcommitted and is actually the most expensive of all the resources. As such, I want to have a build that uses the minimal amount of memory I can possibly get away with safely for my small systems. Note this is only for the small systems which have very low usage and its very easy to add memory if required.
> >> I believe have turned off as much as I can:<image002.png> So is there anything else I can turn off, no matter how small?
> >> Can I get rid of tmpfs somehow since a read only file system is not necessary?
> >> My ultimate goal is to have a VM template that I just push a button and its all built and ready to go and a minimal image will make this far more competitive for the mentioned high end hosting. Of course we could just use Vultr which we love but you get what you pay for ☹
> >> Regards
> >> Michael Knill
> >> Managing Director
> >> D: +61 2 6189 1360
> >> P: +61 2 6140 4656
> >> E: mic...@ip...
> >> W: ipcsolutions.com.au
> >>  <image001.png>Smarter Business Communications
> >
> > Hi Michael,
> >
> > I have here an older list of packages, which I disabled (from the defaults) for Alix boards (at that time):
> >
> > Asterisk: dahdi_echocan_oslec, prosody off
> > Asterisk: dahdi_no_card_firmware on
> > Hardware handling: monit, zabbix_proxy, zabbix off
> > Libraries/Networking: pjsip, 'Asterisk PRI Support' off
> > Networking apps: keepalived, netsnmp, openldap-server, strongswan off
> >
> > These were all packages that were not needed on those boxes.
> >
> > Michael
>
> Micheal Keuter brings up a good point.  Though disabling packages in the build system may not be required.
>
> In AstLinux, memory usage has two elements:
>
> 1) Standard application and kernel RAM usage.
> 1a) Reducing memory usage can be achieved by only starting needed services and tuning the /etc/rc.modules to comment-out unused NIC modules.
>
> 2) The Read-Only root filesystem "/" is by default mounted uncompressed using tmpfs.
> 2a) Reducing memory usage can be achieved by adding "noram" to the KCMD of the /oldroot/cdrom/os/*run.conf file(s).  In the "noram" case the root filesystem "/" is mounted compressed using squashfs, via /dev/loop0.
>
> First, a little history.
>
> Back in the day when CompactFlash (CF) storage was the norm and boards contained 256 MB or more RAM, the Read-Only root filesystem "/" was copied to RAM (tmpfs).  This served two purposes,
> 1) no realtime uncompressing of the squashfs run image noticeably reduced file access latency and speed on single core 500 Mhz (or less) CPUs
> 2) root filesystem access was much faster using RAM (tmpfs) rather than a loop mount directly off CF storage.
>
> In later years, while the advantages of the tmpfs root filesystem were less pronounced as SATA storage appeared and multi-core CPUs, there are still advantages today to make a tmpfs root filesystem still the default.  Eliminating the loop mount directly off the storage is probably the main advantage today (KISS), and systems with 1 GB RAM or more there is no practical RAM disadvantage.
>
> As per Michael Knill's use case above, there may be a legitimate case for adding "noram" in the KCMD of the /oldroot/cdrom/os/*run.conf file(s).
>
> Demonstrate via an example.
>
> Using the latest pre-release [1] Guest VM x86-64bit ISO [2] with Proxmox 8, standard build using Asterisk 20.4.0 with pjsip loaded.
>
> The Proxmox VM is configured with 2-cores and 512 MB RAM
>
> Default case:
> Proxmox: Memory usage 38.13% (195.24 MiB of 512.00 MiB)
> --
>       Host Name: pbx.astlinux
>           Linux: 5.10.179-astlinux x86_64
>             CPU: Common KVM processor (2x) @ 2096 MHz
>             RAM: 489 MB, Available 253 MB
>      Board Type: genx86_64-vm
>        Hardware: KVM Hypervisor Guest VM
> --
> none on / type tmpfs (ro,relatime,size=134128k)
> none                    143.4M    143.4M         0 100% /
>
> Note that 143.4 MB of RAM is used for the root filesystem.
>
> Next, add "noram" to the KCMD= line ...
> --
> # mount -o rw,remount /oldroot/cdrom
>
> Edit the KCMD= line by adding noram after astlive
> # vi /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
> After the change, it should show...
> # grep '^KCMD=' /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
> KCMD="root=/dev/ram0 rw init=/linuxrc astlinux=genx86_64-vm astimg=astlinux-1.5-5855-4d7596.run astkd=auto asturw=auto astlive noram libata.dma=3"
>
> # mount -o ro,remount /oldroot/cdrom
> --
>
> Very Important ->  With "noram" in the KCMD, never perform a firmware "Revert to Previous" followed by an "Upgrade with New" without rebooting immediately after the firmware revert.
>
> "noram" case:
> Proxmox: Memory usage 19.81% (101.42 MiB of 512.00 MiB)
> --
>       Host Name: pbx.astlinux
>           Linux: 5.10.179-astlinux x86_64
>             CPU: Common KVM processor (2x) @ 2096 MHz
>             RAM: 489 MB, Available 400 MB
>      Board Type: genx86_64-vm
>        Hardware: KVM Hypervisor Guest VM
> --
> /dev/loop0 on / type squashfs (ro,relatime)
> /dev/loop0               51.1M     51.1M         0 100% /
>
> Note that no extra RAM is used, the 51.1 MB is the compressed squashfs run image directly mounted off storage.
>
> Interestingly, the AstLinux 400 MB vs. 253 MB Available RAM closely matches the tmpfs root filesystem size, but the Proxmox Memory usage only dropped to 101 MB vs. 195 MB.
>
> Additional RAM (albeit minor) RAM reduction
> Disable unused NIC modules (Proxmox example)
> --
> pbx ~ # cat /etc/rc.modules
> ## These modules get modprobe'd when the system starts up.
> ##
> ## Comment out any modules you don't need.
> ##
> ## Ethernet support
> ## Virtual first, then Emulated
> virtio_net
> #vmxnet3
> #pcnet32
> #e1000
> #e1000e
> #8139too
> --
>
> In summary, setting "noram" in the /oldroot/cdrom/os/*run.conf KCMD should be a production-worthy option for special low-RAM VM situations, but since it has not been used by default, testing should be performed.  Don't forget the "Very Important" notice above.
>
> Also note that once "noram" in the /oldroot/cdrom/os/*run.conf KCMD is defined, it is automatically carried forward on future upgrades.
>
> Lonnie
>
> [1] https://www.astlinux-project.org/dev.html
>
> [2] https://astlinux-project.org/beta/iso/astlinux-beta-genx86_64-vm.iso
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel



_______________________________________________
Astlinux-devel mailing list
Ast...@li...
https://lists.sourceforge.net/lists/listinfo/astlinux-devel

[Astlinux-devel] AstLinux Pre-Release: astlinux-1.5-5875-b12fc0

[Lonnie A. <li...@lo...>]

Announcing AstLinux Pre-Release: astlinux-1.5-5875-b12fc0

AstLinux Project -> Development
https://www.astlinux-project.org/dev.html

** IMPORTANT NOTICE

-- Changes to supported firmware builds:
	== Previous 'ast13se' and 'ast16' firmware branches are no longer updated.
	== New 'ast16se' firmware branch, Asterisk 16.x built --without-pjproject and --without-dahdi
	== Previous 'ast18' firmware branch, Asterisk 18.x built --with-pjproject and --with-dahdi
	== New 'ast20' firmware branch, Asterisk 20.x built --with-pjproject and --with-dahdi

** The AstLinux Team is regularly upgrading packages containing security and bug fixes as well as adding new features of our own.

-- Linux Kernel 5.10.191 (version bump), security and bug fixes

-- OpenSSL, version bump to 1.1.1v, security fixes: CVE-2023-3446, CVE-2023-3817

-- chrony, version bump to 4.4

-- libcurl (curl) version bump to 8.2.1, security fixes: CVE-2023-32001

-- libxml2, version bump to 2.11.5

-- msmtp, version bump to 1.8.24

-- netsnmp, version bump to 5.9.4

-- screen, version 4.9.1, security fix: CVE-2023-24626

-- tinyproxy, version 1.11.1, now included in the standard builds, but disabled by default

-- unixodbc, version bump to 2.3.12

-- upgrade-run-image, limit "noram" loop mount from being removed during upgrade.

-- vnStat, version bump to 2.11

-- zlib, version bump to 1.3

-- Asterisk 16.30.0 ('16se' version bump)
	Last Asterisk 16.x "Legacy" version, built --without-pjproject and --without-dahdi

-- Asterisk 18.19.0 (version bump) and 20.4.0 (new version)
	Built --with-pjproject and --with-dahdi

-- DAHDI, dahdi-linux 3.2.0 (no change) and dahdi-tools 3.2.0 (no change)
	Add build fix to include "astribank" utilities

-- Added rc.conf variable DAHDI_DISABLE, disable DAHDI when set to "yes", defaults to "no".

-- pjsip 2.13.1 (version bump)

-- libpri, version bump to 1.6.1

-- Complete Pre-Release ChangeLog:
https://astlinux-project.org/beta/astlinux-changelog/ChangeLog.txt

The "AstLinux Pre-Release ChangeLog" and "Pre-Release Repository URL" entries can be found under the "Development" tab of the AstLinux Project web site ...

AstLinux Project -> Development
https://www.astlinux-project.org/dev.html


AstLinux Team

Re: [Astlinux-devel] Reducing memory usage as much as possible

[Michael K. <mic...@ip...>]

Yes that is amazing. I started playing with Astlinux sometime in 2005/6 I believe. Left it for a bit and came back to it and now have over 100 systems out there and looking to expand to much more!

Thanks for the fix. Pretty sure we have never configured the /etc/inittab file so shouldn’t be an issue.

So I made some changes to our office system (always ends up being the guinea pig) using noram, dropping out ethernet drivers (I assumed I needed vmxnet3 also as I use VMWare) and took out a few Asterisk modules from modules.conf.

Its now running with 281 MB, Available 147 MB so around 135M usage. Looking at my monitoring over the last 3 months I only saw a 50M variation so I think the current setup should be fine. We get notified by Zabbix if it reduces below 50M. Lets see how it goes. Even though its only a single processor, I suspect an Intel Xeon Gold 5118 (1x) @ 2294 MHz will be fine.

PS I have also noticed that it boots up faster. In my build system on Vultr I rebooted the system and could log back in around 45 seconds later. Nice!

Regards
Michael Knill


From: Lonnie Abelbeck <li...@lo...>
Date: Friday, 18 August 2023 at 1:50 am
To: AstLinux Developers Mailing List <ast...@li...>
Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
Cool how after almost 20 years, old AstLinux features become new again :-)

A fix for the "Very Important" warning has been added to master [1].  This would be difficult to fix at the web interface level, other than disabling "Revert to Previous".

I did discover a very minor issue with a "noram" configuration.  If the /etc/inittab file was edited (rarely needed) the edited /etc/inittab is applied to the filesystem after /usr/sbin/chroot is called in the initrd, as such the default base image /etc/inittab is always used.  A custom build for genx86_64-vm could customize the /etc/inittab here [2].  For the tmpfs root filesystem case, the initrd copies-forward the /etc/inittab file to the uncompressed tmpfs image before calling /usr/sbin/chroot.  Again, should not be a big deal for a "noram" configuration.  BTW, before this was fixed for the tmpfs case, AstLinux 1.3.10 through 1.4.2 ignored any user edits to the /etc/inittab file and nobody noticed :-)

Additionally, if you wanted to build a custom genx86_64-vm image with "noram" enabled by default, you would add noram after astlive here [3].

Lonnie

[1] https://github.com/astlinux-project/astlinux/commit/09eed53ddea8f45414195507aeee6d0f8edb9e92

[2] https://github.com/astlinux-project/astlinux/blob/master/project/astlinux/board/genx86_64-vm/target_skeleton/etc/inittab

[3] https://github.com/astlinux-project/astlinux/blob/master/project/astlinux/board/genx86_64-vm/runnix.conf



> On Aug 17, 2023, at 12:04 AM, Michael Knill <mic...@ip...> wrote:
>
> Wow thanks Lonnie that is fantastic and we will test it out. Gosh this will change things considerably. Will be even less once we pull out other modules we don’t use. Having such a lean image will mean that we don’t really need to explore difficult to manage containerisation options such as Docker.
> Will update the System Tab so that you cannot perform the revert and upgrade as you mentioned.
>
> Regards
> Michael Knill
>
>
> From: Lonnie Abelbeck <li...@lo...>
> Date: Thursday, 17 August 2023 at 1:59 am
> To: AstLinux Developers Mailing List <ast...@li...>
> Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
>
>
> > On Aug 16, 2023, at 3:41 AM, Michael Keuter <li...@mk...> wrote:
> >
> >
> >> Am 16.08.2023 um 05:36 schrieb Michael Knill <mic...@ip...>:
> >>
> >> Hi Devs
> >> Im looking to expand considerably and it will be all in the cloud.
> >> Our current hosting provider allows us to overcommit CPU resources which works well for us as Astlinux uses nothing however memory is something that cannot be overcommitted and is actually the most expensive of all the resources. As such, I want to have a build that uses the minimal amount of memory I can possibly get away with safely for my small systems. Note this is only for the small systems which have very low usage and its very easy to add memory if required.
> >> I believe have turned off as much as I can:<image002.png> So is there anything else I can turn off, no matter how small?
> >> Can I get rid of tmpfs somehow since a read only file system is not necessary?
> >> My ultimate goal is to have a VM template that I just push a button and its all built and ready to go and a minimal image will make this far more competitive for the mentioned high end hosting. Of course we could just use Vultr which we love but you get what you pay for ☹
> >> Regards
> >> Michael Knill
> >> Managing Director
> >> D: +61 2 6189 1360
> >> P: +61 2 6140 4656
> >> E: mic...@ip...
> >> W: ipcsolutions.com.au
> >>  <image001.png>Smarter Business Communications
> >
> > Hi Michael,
> >
> > I have here an older list of packages, which I disabled (from the defaults) for Alix boards (at that time):
> >
> > Asterisk: dahdi_echocan_oslec, prosody off
> > Asterisk: dahdi_no_card_firmware on
> > Hardware handling: monit, zabbix_proxy, zabbix off
> > Libraries/Networking: pjsip, 'Asterisk PRI Support' off
> > Networking apps: keepalived, netsnmp, openldap-server, strongswan off
> >
> > These were all packages that were not needed on those boxes.
> >
> > Michael
>
> Micheal Keuter brings up a good point.  Though disabling packages in the build system may not be required.
>
> In AstLinux, memory usage has two elements:
>
> 1) Standard application and kernel RAM usage.
> 1a) Reducing memory usage can be achieved by only starting needed services and tuning the /etc/rc.modules to comment-out unused NIC modules.
>
> 2) The Read-Only root filesystem "/" is by default mounted uncompressed using tmpfs.
> 2a) Reducing memory usage can be achieved by adding "noram" to the KCMD of the /oldroot/cdrom/os/*run.conf file(s).  In the "noram" case the root filesystem "/" is mounted compressed using squashfs, via /dev/loop0.
>
> First, a little history.
>
> Back in the day when CompactFlash (CF) storage was the norm and boards contained 256 MB or more RAM, the Read-Only root filesystem "/" was copied to RAM (tmpfs).  This served two purposes,
> 1) no realtime uncompressing of the squashfs run image noticeably reduced file access latency and speed on single core 500 Mhz (or less) CPUs
> 2) root filesystem access was much faster using RAM (tmpfs) rather than a loop mount directly off CF storage.
>
> In later years, while the advantages of the tmpfs root filesystem were less pronounced as SATA storage appeared and multi-core CPUs, there are still advantages today to make a tmpfs root filesystem still the default.  Eliminating the loop mount directly off the storage is probably the main advantage today (KISS), and systems with 1 GB RAM or more there is no practical RAM disadvantage.
>
> As per Michael Knill's use case above, there may be a legitimate case for adding "noram" in the KCMD of the /oldroot/cdrom/os/*run.conf file(s).
>
> Demonstrate via an example.
>
> Using the latest pre-release [1] Guest VM x86-64bit ISO [2] with Proxmox 8, standard build using Asterisk 20.4.0 with pjsip loaded.
>
> The Proxmox VM is configured with 2-cores and 512 MB RAM
>
> Default case:
> Proxmox: Memory usage 38.13% (195.24 MiB of 512.00 MiB)
> --
>       Host Name: pbx.astlinux
>           Linux: 5.10.179-astlinux x86_64
>             CPU: Common KVM processor (2x) @ 2096 MHz
>             RAM: 489 MB, Available 253 MB
>      Board Type: genx86_64-vm
>        Hardware: KVM Hypervisor Guest VM
> --
> none on / type tmpfs (ro,relatime,size=134128k)
> none                    143.4M    143.4M         0 100% /
>
> Note that 143.4 MB of RAM is used for the root filesystem.
>
> Next, add "noram" to the KCMD= line ...
> --
> # mount -o rw,remount /oldroot/cdrom
>
> Edit the KCMD= line by adding noram after astlive
> # vi /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
> After the change, it should show...
> # grep '^KCMD=' /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
> KCMD="root=/dev/ram0 rw init=/linuxrc astlinux=genx86_64-vm astimg=astlinux-1.5-5855-4d7596.run astkd=auto asturw=auto astlive noram libata.dma=3"
>
> # mount -o ro,remount /oldroot/cdrom
> --
>
> Very Important ->  With "noram" in the KCMD, never perform a firmware "Revert to Previous" followed by an "Upgrade with New" without rebooting immediately after the firmware revert.
>
> "noram" case:
> Proxmox: Memory usage 19.81% (101.42 MiB of 512.00 MiB)
> --
>       Host Name: pbx.astlinux
>           Linux: 5.10.179-astlinux x86_64
>             CPU: Common KVM processor (2x) @ 2096 MHz
>             RAM: 489 MB, Available 400 MB
>      Board Type: genx86_64-vm
>        Hardware: KVM Hypervisor Guest VM
> --
> /dev/loop0 on / type squashfs (ro,relatime)
> /dev/loop0               51.1M     51.1M         0 100% /
>
> Note that no extra RAM is used, the 51.1 MB is the compressed squashfs run image directly mounted off storage.
>
> Interestingly, the AstLinux 400 MB vs. 253 MB Available RAM closely matches the tmpfs root filesystem size, but the Proxmox Memory usage only dropped to 101 MB vs. 195 MB.
>
> Additional RAM (albeit minor) RAM reduction
> Disable unused NIC modules (Proxmox example)
> --
> pbx ~ # cat /etc/rc.modules
> ## These modules get modprobe'd when the system starts up.
> ##
> ## Comment out any modules you don't need.
> ##
> ## Ethernet support
> ## Virtual first, then Emulated
> virtio_net
> #vmxnet3
> #pcnet32
> #e1000
> #e1000e
> #8139too
> --
>
> In summary, setting "noram" in the /oldroot/cdrom/os/*run.conf KCMD should be a production-worthy option for special low-RAM VM situations, but since it has not been used by default, testing should be performed.  Don't forget the "Very Important" notice above.
>
> Also note that once "noram" in the /oldroot/cdrom/os/*run.conf KCMD is defined, it is automatically carried forward on future upgrades.
>
> Lonnie
>
> [1] https://www.astlinux-project.org/dev.html
>
> [2] https://astlinux-project.org/beta/iso/astlinux-beta-genx86_64-vm.iso
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel



_______________________________________________
Astlinux-devel mailing list
Ast...@li...
https://lists.sourceforge.net/lists/listinfo/astlinux-devel

Re: [Astlinux-devel] Reducing memory usage as much as possible

[Lonnie A. <li...@lo...>]

Cool how after almost 20 years, old AstLinux features become new again :-)

A fix for the "Very Important" warning has been added to master [1].  This would be difficult to fix at the web interface level, other than disabling "Revert to Previous".

I did discover a very minor issue with a "noram" configuration.  If the /etc/inittab file was edited (rarely needed) the edited /etc/inittab is applied to the filesystem after /usr/sbin/chroot is called in the initrd, as such the default base image /etc/inittab is always used.  A custom build for genx86_64-vm could customize the /etc/inittab here [2].  For the tmpfs root filesystem case, the initrd copies-forward the /etc/inittab file to the uncompressed tmpfs image before calling /usr/sbin/chroot.  Again, should not be a big deal for a "noram" configuration.  BTW, before this was fixed for the tmpfs case, AstLinux 1.3.10 through 1.4.2 ignored any user edits to the /etc/inittab file and nobody noticed :-)

Additionally, if you wanted to build a custom genx86_64-vm image with "noram" enabled by default, you would add noram after astlive here [3].

Lonnie

[1] https://github.com/astlinux-project/astlinux/commit/09eed53ddea8f45414195507aeee6d0f8edb9e92

[2] https://github.com/astlinux-project/astlinux/blob/master/project/astlinux/board/genx86_64-vm/target_skeleton/etc/inittab

[3] https://github.com/astlinux-project/astlinux/blob/master/project/astlinux/board/genx86_64-vm/runnix.conf



> On Aug 17, 2023, at 12:04 AM, Michael Knill <mic...@ip...> wrote:
> 
> Wow thanks Lonnie that is fantastic and we will test it out. Gosh this will change things considerably. Will be even less once we pull out other modules we don’t use. Having such a lean image will mean that we don’t really need to explore difficult to manage containerisation options such as Docker.
> Will update the System Tab so that you cannot perform the revert and upgrade as you mentioned.
>  
> Regards
> Michael Knill
>  
>  
> From: Lonnie Abelbeck <li...@lo...>
> Date: Thursday, 17 August 2023 at 1:59 am
> To: AstLinux Developers Mailing List <ast...@li...>
> Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible
> 
> 
> > On Aug 16, 2023, at 3:41 AM, Michael Keuter <li...@mk...> wrote:
> > 
> > 
> >> Am 16.08.2023 um 05:36 schrieb Michael Knill <mic...@ip...>:
> >> 
> >> Hi Devs
> >> Im looking to expand considerably and it will be all in the cloud.
> >> Our current hosting provider allows us to overcommit CPU resources which works well for us as Astlinux uses nothing however memory is something that cannot be overcommitted and is actually the most expensive of all the resources. As such, I want to have a build that uses the minimal amount of memory I can possibly get away with safely for my small systems. Note this is only for the small systems which have very low usage and its very easy to add memory if required.
> >> I believe have turned off as much as I can:<image002.png> So is there anything else I can turn off, no matter how small?
> >> Can I get rid of tmpfs somehow since a read only file system is not necessary?
> >> My ultimate goal is to have a VM template that I just push a button and its all built and ready to go and a minimal image will make this far more competitive for the mentioned high end hosting. Of course we could just use Vultr which we love but you get what you pay for ☹
> >> Regards
> >> Michael Knill
> >> Managing Director
> >> D: +61 2 6189 1360
> >> P: +61 2 6140 4656
> >> E: mic...@ip...
> >> W: ipcsolutions.com.au
> >>  <image001.png>Smarter Business Communications
> > 
> > Hi Michael,
> > 
> > I have here an older list of packages, which I disabled (from the defaults) for Alix boards (at that time):
> > 
> > Asterisk: dahdi_echocan_oslec, prosody off
> > Asterisk: dahdi_no_card_firmware on
> > Hardware handling: monit, zabbix_proxy, zabbix off
> > Libraries/Networking: pjsip, 'Asterisk PRI Support' off
> > Networking apps: keepalived, netsnmp, openldap-server, strongswan off
> > 
> > These were all packages that were not needed on those boxes.
> > 
> > Michael
> 
> Micheal Keuter brings up a good point.  Though disabling packages in the build system may not be required.
> 
> In AstLinux, memory usage has two elements:
> 
> 1) Standard application and kernel RAM usage.
> 1a) Reducing memory usage can be achieved by only starting needed services and tuning the /etc/rc.modules to comment-out unused NIC modules.
> 
> 2) The Read-Only root filesystem "/" is by default mounted uncompressed using tmpfs.
> 2a) Reducing memory usage can be achieved by adding "noram" to the KCMD of the /oldroot/cdrom/os/*run.conf file(s).  In the "noram" case the root filesystem "/" is mounted compressed using squashfs, via /dev/loop0.
> 
> First, a little history.
> 
> Back in the day when CompactFlash (CF) storage was the norm and boards contained 256 MB or more RAM, the Read-Only root filesystem "/" was copied to RAM (tmpfs).  This served two purposes,
> 1) no realtime uncompressing of the squashfs run image noticeably reduced file access latency and speed on single core 500 Mhz (or less) CPUs
> 2) root filesystem access was much faster using RAM (tmpfs) rather than a loop mount directly off CF storage.
> 
> In later years, while the advantages of the tmpfs root filesystem were less pronounced as SATA storage appeared and multi-core CPUs, there are still advantages today to make a tmpfs root filesystem still the default.  Eliminating the loop mount directly off the storage is probably the main advantage today (KISS), and systems with 1 GB RAM or more there is no practical RAM disadvantage.
> 
> As per Michael Knill's use case above, there may be a legitimate case for adding "noram" in the KCMD of the /oldroot/cdrom/os/*run.conf file(s).
> 
> Demonstrate via an example.
> 
> Using the latest pre-release [1] Guest VM x86-64bit ISO [2] with Proxmox 8, standard build using Asterisk 20.4.0 with pjsip loaded.
> 
> The Proxmox VM is configured with 2-cores and 512 MB RAM
> 
> Default case:
> Proxmox: Memory usage 38.13% (195.24 MiB of 512.00 MiB)
> --
>       Host Name: pbx.astlinux
>           Linux: 5.10.179-astlinux x86_64
>             CPU: Common KVM processor (2x) @ 2096 MHz
>             RAM: 489 MB, Available 253 MB
>      Board Type: genx86_64-vm
>        Hardware: KVM Hypervisor Guest VM
> --
> none on / type tmpfs (ro,relatime,size=134128k)
> none                    143.4M    143.4M         0 100% /
> 
> Note that 143.4 MB of RAM is used for the root filesystem.
> 
> Next, add "noram" to the KCMD= line ...
> --
> # mount -o rw,remount /oldroot/cdrom
> 
> Edit the KCMD= line by adding noram after astlive
> # vi /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
> After the change, it should show...
> # grep '^KCMD=' /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
> KCMD="root=/dev/ram0 rw init=/linuxrc astlinux=genx86_64-vm astimg=astlinux-1.5-5855-4d7596.run astkd=auto asturw=auto astlive noram libata.dma=3"
> 
> # mount -o ro,remount /oldroot/cdrom
> --
> 
> Very Important ->  With "noram" in the KCMD, never perform a firmware "Revert to Previous" followed by an "Upgrade with New" without rebooting immediately after the firmware revert.
> 
> "noram" case:
> Proxmox: Memory usage 19.81% (101.42 MiB of 512.00 MiB)
> --
>       Host Name: pbx.astlinux
>           Linux: 5.10.179-astlinux x86_64
>             CPU: Common KVM processor (2x) @ 2096 MHz
>             RAM: 489 MB, Available 400 MB
>      Board Type: genx86_64-vm
>        Hardware: KVM Hypervisor Guest VM
> --
> /dev/loop0 on / type squashfs (ro,relatime)
> /dev/loop0               51.1M     51.1M         0 100% /
> 
> Note that no extra RAM is used, the 51.1 MB is the compressed squashfs run image directly mounted off storage.
> 
> Interestingly, the AstLinux 400 MB vs. 253 MB Available RAM closely matches the tmpfs root filesystem size, but the Proxmox Memory usage only dropped to 101 MB vs. 195 MB.
> 
> Additional RAM (albeit minor) RAM reduction
> Disable unused NIC modules (Proxmox example)
> --
> pbx ~ # cat /etc/rc.modules 
> ## These modules get modprobe'd when the system starts up.
> ##
> ## Comment out any modules you don't need.
> ##
> ## Ethernet support
> ## Virtual first, then Emulated
> virtio_net
> #vmxnet3
> #pcnet32
> #e1000
> #e1000e
> #8139too
> --
> 
> In summary, setting "noram" in the /oldroot/cdrom/os/*run.conf KCMD should be a production-worthy option for special low-RAM VM situations, but since it has not been used by default, testing should be performed.  Don't forget the "Very Important" notice above.
> 
> Also note that once "noram" in the /oldroot/cdrom/os/*run.conf KCMD is defined, it is automatically carried forward on future upgrades.
> 
> Lonnie
> 
> [1] https://www.astlinux-project.org/dev.html
> 
> [2] https://astlinux-project.org/beta/iso/astlinux-beta-genx86_64-vm.iso
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel

Re: [Astlinux-devel] Reducing memory usage as much as possible

[Michael K. <mic...@ip...>]

Wow thanks Lonnie that is fantastic and we will test it out. Gosh this will change things considerably. Will be even less once we pull out other modules we don’t use. Having such a lean image will mean that we don’t really need to explore difficult to manage containerisation options such as Docker.
Will update the System Tab so that you cannot perform the revert and upgrade as you mentioned.

Regards
Michael Knill


From: Lonnie Abelbeck <li...@lo...>
Date: Thursday, 17 August 2023 at 1:59 am
To: AstLinux Developers Mailing List <ast...@li...>
Subject: Re: [Astlinux-devel] Reducing memory usage as much as possible

> On Aug 16, 2023, at 3:41 AM, Michael Keuter <li...@mk...> wrote:
>
>
>> Am 16.08.2023 um 05:36 schrieb Michael Knill <mic...@ip...>:
>>
>> Hi Devs
>> Im looking to expand considerably and it will be all in the cloud.
>> Our current hosting provider allows us to overcommit CPU resources which works well for us as Astlinux uses nothing however memory is something that cannot be overcommitted and is actually the most expensive of all the resources. As such, I want to have a build that uses the minimal amount of memory I can possibly get away with safely for my small systems. Note this is only for the small systems which have very low usage and its very easy to add memory if required.
>> I believe have turned off as much as I can:<image002.png> So is there anything else I can turn off, no matter how small?
>> Can I get rid of tmpfs somehow since a read only file system is not necessary?
>> My ultimate goal is to have a VM template that I just push a button and its all built and ready to go and a minimal image will make this far more competitive for the mentioned high end hosting. Of course we could just use Vultr which we love but you get what you pay for ☹
>> Regards
>> Michael Knill
>> Managing Director
>> D: +61 2 6189 1360
>> P: +61 2 6140 4656
>> E: mic...@ip...
>> W: ipcsolutions.com.au
>>  <image001.png>Smarter Business Communications
>
> Hi Michael,
>
> I have here an older list of packages, which I disabled (from the defaults) for Alix boards (at that time):
>
> Asterisk: dahdi_echocan_oslec, prosody off
> Asterisk: dahdi_no_card_firmware on
> Hardware handling: monit, zabbix_proxy, zabbix off
> Libraries/Networking: pjsip, 'Asterisk PRI Support' off
> Networking apps: keepalived, netsnmp, openldap-server, strongswan off
>
> These were all packages that were not needed on those boxes.
>
> Michael

Micheal Keuter brings up a good point.  Though disabling packages in the build system may not be required.

In AstLinux, memory usage has two elements:

1) Standard application and kernel RAM usage.
1a) Reducing memory usage can be achieved by only starting needed services and tuning the /etc/rc.modules to comment-out unused NIC modules.

2) The Read-Only root filesystem "/" is by default mounted uncompressed using tmpfs.
2a) Reducing memory usage can be achieved by adding "noram" to the KCMD of the /oldroot/cdrom/os/*run.conf file(s).  In the "noram" case the root filesystem "/" is mounted compressed using squashfs, via /dev/loop0.

First, a little history.

Back in the day when CompactFlash (CF) storage was the norm and boards contained 256 MB or more RAM, the Read-Only root filesystem "/" was copied to RAM (tmpfs).  This served two purposes,
1) no realtime uncompressing of the squashfs run image noticeably reduced file access latency and speed on single core 500 Mhz (or less) CPUs
2) root filesystem access was much faster using RAM (tmpfs) rather than a loop mount directly off CF storage.

In later years, while the advantages of the tmpfs root filesystem were less pronounced as SATA storage appeared and multi-core CPUs, there are still advantages today to make a tmpfs root filesystem still the default.  Eliminating the loop mount directly off the storage is probably the main advantage today (KISS), and systems with 1 GB RAM or more there is no practical RAM disadvantage.

As per Michael Knill's use case above, there may be a legitimate case for adding "noram" in the KCMD of the /oldroot/cdrom/os/*run.conf file(s).

Demonstrate via an example.

Using the latest pre-release [1] Guest VM x86-64bit ISO [2] with Proxmox 8, standard build using Asterisk 20.4.0 with pjsip loaded.

The Proxmox VM is configured with 2-cores and 512 MB RAM

Default case:
Proxmox: Memory usage 38.13% (195.24 MiB of 512.00 MiB)
--
      Host Name: pbx.astlinux
          Linux: 5.10.179-astlinux x86_64
            CPU: Common KVM processor (2x) @ 2096 MHz
            RAM: 489 MB, Available 253 MB
     Board Type: genx86_64-vm
       Hardware: KVM Hypervisor Guest VM
--
none on / type tmpfs (ro,relatime,size=134128k)
none                    143.4M    143.4M         0 100% /

Note that 143.4 MB of RAM is used for the root filesystem.

Next, add "noram" to the KCMD= line ...
--
# mount -o rw,remount /oldroot/cdrom

Edit the KCMD= line by adding noram after astlive
# vi /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
After the change, it should show...
# grep '^KCMD=' /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
KCMD="root=/dev/ram0 rw init=/linuxrc astlinux=genx86_64-vm astimg=astlinux-1.5-5855-4d7596.run astkd=auto asturw=auto astlive noram libata.dma=3"

# mount -o ro,remount /oldroot/cdrom
--

Very Important ->  With "noram" in the KCMD, never perform a firmware "Revert to Previous" followed by an "Upgrade with New" without rebooting immediately after the firmware revert.

"noram" case:
Proxmox: Memory usage 19.81% (101.42 MiB of 512.00 MiB)
--
      Host Name: pbx.astlinux
          Linux: 5.10.179-astlinux x86_64
            CPU: Common KVM processor (2x) @ 2096 MHz
            RAM: 489 MB, Available 400 MB
     Board Type: genx86_64-vm
       Hardware: KVM Hypervisor Guest VM
--
/dev/loop0 on / type squashfs (ro,relatime)
/dev/loop0               51.1M     51.1M         0 100% /

Note that no extra RAM is used, the 51.1 MB is the compressed squashfs run image directly mounted off storage.

Interestingly, the AstLinux 400 MB vs. 253 MB Available RAM closely matches the tmpfs root filesystem size, but the Proxmox Memory usage only dropped to 101 MB vs. 195 MB.

Additional RAM (albeit minor) RAM reduction
Disable unused NIC modules (Proxmox example)
--
pbx ~ # cat /etc/rc.modules
## These modules get modprobe'd when the system starts up.
##
## Comment out any modules you don't need.
##
## Ethernet support
## Virtual first, then Emulated
virtio_net
#vmxnet3
#pcnet32
#e1000
#e1000e
#8139too
--

In summary, setting "noram" in the /oldroot/cdrom/os/*run.conf KCMD should be a production-worthy option for special low-RAM VM situations, but since it has not been used by default, testing should be performed.  Don't forget the "Very Important" notice above.

Also note that once "noram" in the /oldroot/cdrom/os/*run.conf KCMD is defined, it is automatically carried forward on future upgrades.

Lonnie

[1] https://www.astlinux-project.org/dev.html

[2] https://astlinux-project.org/beta/iso/astlinux-beta-genx86_64-vm.iso
















_______________________________________________
Astlinux-devel mailing list
Ast...@li...
https://lists.sourceforge.net/lists/listinfo/astlinux-devel

Re: [Astlinux-devel] Reducing memory usage as much as possible

[Lonnie A. <li...@lo...>]

> On Aug 16, 2023, at 3:41 AM, Michael Keuter <li...@mk...> wrote:
> 
> 
>> Am 16.08.2023 um 05:36 schrieb Michael Knill <mic...@ip...>:
>> 
>> Hi Devs
>> Im looking to expand considerably and it will be all in the cloud.
>> Our current hosting provider allows us to overcommit CPU resources which works well for us as Astlinux uses nothing however memory is something that cannot be overcommitted and is actually the most expensive of all the resources. As such, I want to have a build that uses the minimal amount of memory I can possibly get away with safely for my small systems. Note this is only for the small systems which have very low usage and its very easy to add memory if required.
>> I believe have turned off as much as I can:<image002.png> So is there anything else I can turn off, no matter how small?
>> Can I get rid of tmpfs somehow since a read only file system is not necessary?
>> My ultimate goal is to have a VM template that I just push a button and its all built and ready to go and a minimal image will make this far more competitive for the mentioned high end hosting. Of course we could just use Vultr which we love but you get what you pay for ☹
>> Regards
>> Michael Knill
>> Managing Director
>> D: +61 2 6189 1360
>> P: +61 2 6140 4656
>> E: mic...@ip...
>> W: ipcsolutions.com.au
>>  <image001.png>Smarter Business Communications
> 
> Hi Michael,
> 
> I have here an older list of packages, which I disabled (from the defaults) for Alix boards (at that time):
> 
> Asterisk: dahdi_echocan_oslec, prosody off
> Asterisk: dahdi_no_card_firmware on
> Hardware handling: monit, zabbix_proxy, zabbix off
> Libraries/Networking: pjsip, 'Asterisk PRI Support' off
> Networking apps: keepalived, netsnmp, openldap-server, strongswan off
> 
> These were all packages that were not needed on those boxes.
> 
> Michael

Micheal Keuter brings up a good point.  Though disabling packages in the build system may not be required.

In AstLinux, memory usage has two elements:

1) Standard application and kernel RAM usage.
1a) Reducing memory usage can be achieved by only starting needed services and tuning the /etc/rc.modules to comment-out unused NIC modules.

2) The Read-Only root filesystem "/" is by default mounted uncompressed using tmpfs.
2a) Reducing memory usage can be achieved by adding "noram" to the KCMD of the /oldroot/cdrom/os/*run.conf file(s).  In the "noram" case the root filesystem "/" is mounted compressed using squashfs, via /dev/loop0.

First, a little history.

Back in the day when CompactFlash (CF) storage was the norm and boards contained 256 MB or more RAM, the Read-Only root filesystem "/" was copied to RAM (tmpfs).  This served two purposes,
1) no realtime uncompressing of the squashfs run image noticeably reduced file access latency and speed on single core 500 Mhz (or less) CPUs
2) root filesystem access was much faster using RAM (tmpfs) rather than a loop mount directly off CF storage.

In later years, while the advantages of the tmpfs root filesystem were less pronounced as SATA storage appeared and multi-core CPUs, there are still advantages today to make a tmpfs root filesystem still the default.  Eliminating the loop mount directly off the storage is probably the main advantage today (KISS), and systems with 1 GB RAM or more there is no practical RAM disadvantage.

As per Michael Knill's use case above, there may be a legitimate case for adding "noram" in the KCMD of the /oldroot/cdrom/os/*run.conf file(s).

Demonstrate via an example.

Using the latest pre-release [1] Guest VM x86-64bit ISO [2] with Proxmox 8, standard build using Asterisk 20.4.0 with pjsip loaded.

The Proxmox VM is configured with 2-cores and 512 MB RAM

Default case:
Proxmox: Memory usage 38.13% (195.24 MiB of 512.00 MiB)
--
      Host Name: pbx.astlinux
          Linux: 5.10.179-astlinux x86_64
            CPU: Common KVM processor (2x) @ 2096 MHz
            RAM: 489 MB, Available 253 MB
     Board Type: genx86_64-vm
       Hardware: KVM Hypervisor Guest VM
--
none on / type tmpfs (ro,relatime,size=134128k)
none                    143.4M    143.4M         0 100% /

Note that 143.4 MB of RAM is used for the root filesystem.

Next, add "noram" to the KCMD= line ...
--
# mount -o rw,remount /oldroot/cdrom

Edit the KCMD= line by adding noram after astlive
# vi /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
After the change, it should show...
# grep '^KCMD=' /oldroot/cdrom/os/astlinux-1.5-5855-4d7596.run.conf
KCMD="root=/dev/ram0 rw init=/linuxrc astlinux=genx86_64-vm astimg=astlinux-1.5-5855-4d7596.run astkd=auto asturw=auto astlive noram libata.dma=3"

# mount -o ro,remount /oldroot/cdrom
--

Very Important ->  With "noram" in the KCMD, never perform a firmware "Revert to Previous" followed by an "Upgrade with New" without rebooting immediately after the firmware revert.

"noram" case:
Proxmox: Memory usage 19.81% (101.42 MiB of 512.00 MiB)
--
      Host Name: pbx.astlinux
          Linux: 5.10.179-astlinux x86_64
            CPU: Common KVM processor (2x) @ 2096 MHz
            RAM: 489 MB, Available 400 MB
     Board Type: genx86_64-vm
       Hardware: KVM Hypervisor Guest VM
--
/dev/loop0 on / type squashfs (ro,relatime)
/dev/loop0               51.1M     51.1M         0 100% /

Note that no extra RAM is used, the 51.1 MB is the compressed squashfs run image directly mounted off storage.

Interestingly, the AstLinux 400 MB vs. 253 MB Available RAM closely matches the tmpfs root filesystem size, but the Proxmox Memory usage only dropped to 101 MB vs. 195 MB.

Additional RAM (albeit minor) RAM reduction
Disable unused NIC modules (Proxmox example)
--
pbx ~ # cat /etc/rc.modules 
## These modules get modprobe'd when the system starts up.
##
## Comment out any modules you don't need.
##
## Ethernet support
## Virtual first, then Emulated
virtio_net
#vmxnet3
#pcnet32
#e1000
#e1000e
#8139too
--

In summary, setting "noram" in the /oldroot/cdrom/os/*run.conf KCMD should be a production-worthy option for special low-RAM VM situations, but since it has not been used by default, testing should be performed.  Don't forget the "Very Important" notice above.

Also note that once "noram" in the /oldroot/cdrom/os/*run.conf KCMD is defined, it is automatically carried forward on future upgrades.

Lonnie

[1] https://www.astlinux-project.org/dev.html

[2] https://astlinux-project.org/beta/iso/astlinux-beta-genx86_64-vm.iso

Re: [Astlinux-devel] Reducing memory usage as much as possible

[Michael K. <li...@mk...>]

> Am 16.08.2023 um 05:36 schrieb Michael Knill <mic...@ip...>:
> 
> Hi Devs
>  Im looking to expand considerably and it will be all in the cloud.
> Our current hosting provider allows us to overcommit CPU resources which works well for us as Astlinux uses nothing however memory is something that cannot be overcommitted and is actually the most expensive of all the resources. As such, I want to have a build that uses the minimal amount of memory I can possibly get away with safely for my small systems. Note this is only for the small systems which have very low usage and its very easy to add memory if required.
>  I believe have turned off as much as I can:<image002.png> So is there anything else I can turn off, no matter how small?
> Can I get rid of tmpfs somehow since a read only file system is not necessary?
>  My ultimate goal is to have a VM template that I just push a button and its all built and ready to go and a minimal image will make this far more competitive for the mentioned high end hosting. Of course we could just use Vultr which we love but you get what you pay for ☹
>  Regards
>  Michael Knill
> Managing Director
>  D: +61 2 6189 1360
> P: +61 2 6140 4656
> E: mic...@ip...
> W: ipcsolutions.com.au
>   <image001.png>Smarter Business Communications

Hi Michael,

I have here an older list of packages, which I disabled (from the defaults) for Alix boards (at that time):

Asterisk: dahdi_echocan_oslec, prosody off
Asterisk: dahdi_no_card_firmware on
Hardware handling: monit, zabbix_proxy, zabbix off
Libraries/Networking: pjsip, 'Asterisk PRI Support' off
Networking apps: keepalived, netsnmp, openldap-server, strongswan off

These were all packages that were not needed on those boxes.

Michael

http://www.mksolutions.info

[Astlinux-devel] Reducing memory usage as much as possible

[Michael K. <mic...@ip...>]

Hi Devs

Im looking to expand considerably and it will be all in the cloud.
Our current hosting provider allows us to overcommit CPU resources which works well for us as Astlinux uses nothing however memory is something that cannot be overcommitted and is actually the most expensive of all the resources. As such, I want to have a build that uses the minimal amount of memory I can possibly get away with safely for my small systems. Note this is only for the small systems which have very low usage and its very easy to add memory if required.

I believe have turned off as much as I can:
[cid:image002.png@01D9D044.A7B7BB90]

So is there anything else I can turn off, no matter how small?
Can I get rid of tmpfs somehow since a read only file system is not necessary?

My ultimate goal is to have a VM template that I just push a button and its all built and ready to go and a minimal image will make this far more competitive for the mentioned high end hosting. Of course we could just use Vultr which we love but you get what you pay for ☹

Regards

Michael Knill
Managing Director

D: +61 2 6189 1360<tel:+61261891360>
P: +61 2 6140 4656<tel:+61261404656>
E: mic...@ip...<mailto:mic...@ip...>
W: ipcsolutions.com.au<https://ipcsolutions.com.au/>

 [Icon  Description automatically generated]
Smarter Business Communications

[Astlinux-devel] Announcing AstLinux Release: 1.5.1

[Lonnie A. <li...@lo...>]

Announcing AstLinux Release: 1.5.1

More Info: AstLinux Project
https://www.astlinux-project.org/

AstLinux 1.5.1 Highlights:
* Asterisk Versions: 13.38.3, 16.30.0, 18.18.0

* Linux Kernel 5.10.179, security and bug fixes
* RUNNIX, version bump to runnix-0.6.15
* OpenSSL, version bump to 1.1.1u, security fixes: CVE-2023-0464, CVE-2023-0465, CVE-2023-0466, CVE-2023-2650
* libcurl (curl) version bump to 8.1.2, security fixes: CVE-2023-27537, CVE-2023-27536, CVE-2023-27535, CVE-2023-27534, CVE-2023-27533, CVE-2023-28319, CVE-2023-28320, * CVE-2023-28321, CVE-2023-28322
* libcap, version bump to 2.69, security fixes: CVE-2023-2602, CVE-2023-2603
* libpcap, version bump to 1.10.4
* libxml2, version bump to 2.10.4, security fixes: CVE-2023-29469, CVE-2023-28484
* dnsmasq, version 2.84, security fix: CVE-2023-28450
* Fossil, (major) version bump to 2.22
* ncurses, version bump to 6.4, security fix: CVE-2023-29491
* pjsip version bump to 2.13
* sngrep, version bump to 1.7.0
* sqlite, version bump to 3.42.0
* tiff, version bump to 4.5.1
* tcpdump, version bump to 4.99.4
* udev (eudev), version bump to 3.2.12
* zabbix, version bump to 4.0.47, security fix: CVE-2023-29456
* Asterisk '13se' (stable edition) version 13.38.3 is the last Asterisk 13.x "Legacy" version, built --without-pjproject
* Package upgrades providing important security and bug fixes

Full ChangeLog:
https://raw.githubusercontent.com/astlinux-project/astlinux/1.5.1/docs/ChangeLog.txt

All users are encouraged to upgrade, read the ChangeLog for the details.

AstLinux Team

Re: [Astlinux-devel] FOP2 sha1 checksum error when building Astlinux from source

[eucalyptus <euc...@pr...>]

Hello Lonnie,

I restarted the build just after your reply and I can confirm that it completed successfully today.

Thanks for the quick reply and the helpful information.

Regards,
eucalyptus



Sent with Proton Mail secure email.

------- Original Message -------
On Monday, July 3rd, 2023 at 8:54 PM, Lonnie Abelbeck <li...@lo...> wrote:


> Hi eucalyptus,
> 
> Thanks for the note, the FOP2 project has the bad habit of updating download files without changing the version number.
> 
> I have updated the FOP2 .sha1 and the current 2.31.34 version download.
> 
> BTW, the FOP2 binary is a special case package, and it is not a part of the AstLinux image you build. The FOP2 package download is only used by the AstLinux Team.
> 
> Restart your build and you should be good now.
> 
> Lonnie
> 
> 
> 
> > On Jul 3, 2023, at 7:37 PM, eucalyptus via Astlinux-devel ast...@li... wrote:
> > 
> > Greetings,
> > 
> > While building Astlinux 18 (cloned from GitHub last week) from source, the build is halting at the FOP2 step, with an error that indicates the FOP2 file (fop2-2.31.34-debian-x86_64.tgz) is failing its checksum.
> > 
> > I've downloaded both the FOP2 file and the checksum from the same endpoints that the build script does and verified that the two checksums do not match.
> > 
> > Is this something that I could modify the build script to temporarily bypass?
> > 
> > Thanks,
> > eucalyptus
> > 
> > Sent with Proton Mail secure email.
> > 
> > _______________________________________________
> > Astlinux-devel mailing list
> > Ast...@li...
> > https://lists.sourceforge.net/lists/listinfo/astlinux-devel
> 
> 
> 
> 
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel

Re: [Astlinux-devel] FOP2 sha1 checksum error when building Astlinux from source

[Lonnie A. <li...@lo...>]

Hi eucalyptus,

Thanks for the note, the FOP2 project has the bad habit of updating download files without changing the version number.

I have updated the FOP2 .sha1 and the current 2.31.34 version download.

BTW, the FOP2 binary is a special case package, and it is not a part of the AstLinux image you build.  The FOP2 package download is only used by the AstLinux Team.

Restart your build and you should be good now.

Lonnie




> On Jul 3, 2023, at 7:37 PM, eucalyptus via Astlinux-devel <ast...@li...> wrote:
> 
> Greetings,
> 
> While building Astlinux 18 (cloned from GitHub last week) from source, the build is halting at the FOP2 step, with an error that indicates the FOP2 file (fop2-2.31.34-debian-x86_64.tgz) is failing its checksum.
> 
> I've downloaded both the FOP2 file and the checksum from the same endpoints that the build script does and verified that the two checksums do not match.
> 
> Is this something that I could modify the build script to temporarily bypass?
> 
> Thanks,
> eucalyptus
> 
> 
> 
> Sent with Proton Mail secure email.
> _______________________________________________
> Astlinux-devel mailing list
> Ast...@li...
> https://lists.sourceforge.net/lists/listinfo/astlinux-devel

[Astlinux-devel] FOP2 sha1 checksum error when building Astlinux from source

[eucalyptus <euc...@pr...>]

Greetings,

While building Astlinux 18 (cloned from GitHub last week) from source, the build is halting at the FOP2 step, with an error that indicates the FOP2 file (fop2-2.31.34-debian-x86_64.tgz) is failing its checksum.

I've downloaded both the FOP2 file and the checksum from the same endpoints that the build script does and verified that the two checksums do not match.

Is this something that I could modify the build script to temporarily bypass?

Thanks,
eucalyptus

Sent with [Proton Mail](https://proton.me/) secure email.

[Astlinux-devel] AstLinux Pre-Release: astlinux-1.5-5809-61d23a

[Lonnie A. <li...@lo...>]

Announcing AstLinux Pre-Release: astlinux-1.5-5809-61d23a

AstLinux Project -> Development
https://www.astlinux-project.org/dev.html

** The AstLinux Team is regularly upgrading packages containing security and bug fixes as well as adding new features of our own.

-- Linux Kernel 5.10.178 (version bump), security and bug fixes

-- OpenSSL, version bump to 1.1.1u, security fixes: CVE-2023-0464, CVE-2023-0465, CVE-2023-0466, CVE-2023-2650

-- libcurl (curl) version bump to 8.1.2, security fixes: CVE-2023-27535, CVE-2023-28319, etc.

-- dnsmasq, version 2.84, security fix: CVE-2023-28450

-- Fossil, (major) version bump to 2.22

-- keepalived, version bump to 2.2.8

-- libpcap, version bump to 1.10.4

-- libxml2, version bump to 2.10.4, security fixes: CVE-2023-29469, CVE-2023-28484

-- ncurses, version bump to 6.4, security fix: CVE-2023-29491 using --disable-root-environ build

-- pciutils, version bump to 3.10.0

-- sqlite, version bump to 3.42.0

-- sngrep, version bump to 1.7.0

-- tcpdump, version 4.99.4

-- udev (eudev), version bump to 3.2.12

-- zabbix, version bump to 4.0.46

-- ca-certificates, update trusted root certificates 2023-05-30

-- mac2vendor, oui.txt database snapshot 2023-06-10

-- Time Zone Database update, tzdata2023c and php-timezonedb-2023.3

-- Asterisk 13.38.3 ('13se' no change)
	Last Asterisk 13.x "Legacy" version, built --without-pjproject

-- Asterisk 16.30.0 (no change) and 18.18.0 (version bump)
	Add Asterisk 16.x pjsip_pubsub-fixes-for-pjsip-2.13 patch

-- DAHDI, dahdi-linux 3.2.0 (no change) and dahdi-tools 3.2.0 (no change)

-- pjsip 2.13 (version bump)

-- Complete Pre-Release ChangeLog:
https://astlinux-project.org/beta/astlinux-changelog/ChangeLog.txt

The "AstLinux Pre-Release ChangeLog" and "Pre-Release Repository URL" entries can be found under the "Development" tab of the AstLinux Project web site ...

AstLinux Project -> Development
https://www.astlinux-project.org/dev.html


AstLinux Team

[Astlinux-devel] Cisco SPA112 2-Port Phone Adapters Remote Command Execution Vulnerability

[Michael K. <li...@mk...>]

Hi list,

if you use Cisco SPA112 ATAs this is important:

https://sec.cloudapps.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-spa-unauth-upgrade-UqhyTWW

Michael

http://www.mksolutions.info